Abstract: A technique for user interaction with an autonomous unmanned aerial vehicle (UAV) is described. In an example embodiment, perception inputs from one or more sensor devices are processed to build a shared virtual environment that is representative of a physical environment. The sensor devices used to generate perception inputs can include image capture devices onboard an autonomous aerial vehicle that is in flight through the physical environment. The shared virtual environment can provide a continually updated representation of the physical environment which is accessible to multiple network-connected devices, including multiple UAVs and multiple mobile computing devices. The shared virtual environment can be used, for example, to display visual augmentations at network-connected user devices and guide autonomous navigation by the UAV.

Abstract: The disclosed techniques may include receiving a request for a first step of a development workflow for deploying a standardized computer function within an organization. In addition, the techniques may include identifying a first file corresponding to the first step of the workflow. The techniques may include providing the first file to the application of the client device to cause the application of the client device to display a first text element at a first portion of a graphical interface and a first interactive element at a second portion of the graphical interface. The techniques may include receiving first information from the first interactive element. The techniques may include performing a first action in accordance with the first step of the development workflow. Further, the techniques may include generating a second file to implement a second step of the workflow based at least in part on the received first information.

Abstract: Systems and methods are disclosed for tracking objects in a physical environment using visual sensors onboard an autonomous unmanned aerial vehicle (UAV). In certain embodiments, images of the physical environment captured by the onboard visual sensors are processed to extract semantic information about detected objects. Processing of the captured images may involve applying machine learning techniques such as a deep convolutional neural network to extract semantic cues regarding objects detected in the images. The object tracking can be utilized, for example, to facilitate autonomous navigation by the UAV or to generate and display augmentative information regarding tracked objects to users.

Abstract: An introduced autonomous aerial vehicle can include multiple cameras for capturing images of a surrounding physical environment that are utilized for motion planning by an autonomous navigation system. In some embodiments, the cameras can be integrated into one or more rotor assemblies that house powered rotors to free up space within the body of the aerial vehicle. In an example embodiment, an aerial vehicle includes multiple upward-facing cameras and multiple downward-facing cameras with overlapping fields of view to enable stereoscopic computer vision in a plurality of directions around the aerial vehicle. Similar camera arrangements can also be implemented in fixed-wing aerial vehicles.

Abstract: Autonomous aerial navigation in low-light and no-light conditions includes using night mode obstacle avoidance intelligence and mechanisms for vision-based unmanned aerial vehicle (UAV) navigation to enable autonomous flight operations of a UAV in low-light and no-light environments using infrared data.

Abstract: The technology described herein relates to autonomous aerial vehicle rotor configurations. In some embodiments, the aerial vehicle includes a central body that extends along a longitudinal axis from a forward end to an aft end including a port side opposite a starboard side. Multiple rotor arms each have a proximal end coupled to the central body and a rotor assembly arranged at a distal end to provide propulsion for the aerial vehicle. The rotor assemblies include a first set of rotor assemblies and a second set of rotor assemblies. The first set of rotor assemblies are arranged in a non-inverted configuration on a top side of the aerial vehicle such that each rotor assembly includes an upward-facing rotor. The second set of rotor assemblies are arranged in an inverted configuration on a bottom side of the aerial vehicle such that each rotor assembly includes a downward-facing rotor.

Abstract: An autonomous vehicle that is equipped with image capture devices can use information gathered from the image capture devices to plan a future three-dimensional (3D) trajectory through a physical environment. To this end, a technique is described for image-space based motion planning. In an embodiment, a planned 3D trajectory is projected into an image-space of an image captured by the autonomous vehicle. The planned 3D trajectory is then optimized according to a cost function derived from information (e.g., depth estimates) in the captured image. The cost function associates higher cost values with identified regions of the captured image that are associated with areas of the physical environment into which travel is risky or otherwise undesirable. The autonomous vehicle is thereby encouraged to avoid these areas while satisfying other motion planning objectives.

Abstract: An autonomous vehicle that is equipped with image capture devices can use information gathered from the image capture devices to plan a future three-dimensional (3D) trajectory through a physical environment. To this end, a technique is described for image-space based motion planning. In an embodiment, a planned 3D trajectory is projected into an image-space of an image captured by the autonomous vehicle. The planned 3D trajectory is then optimized according to a cost function derived from information (e.g., depth estimates) in the captured image. The cost function associates higher cost values with identified regions of the captured image that are associated with areas of the physical environment into which travel is risky or otherwise undesirable. The autonomous vehicle is thereby encouraged to avoid these areas while satisfying other motion planning objectives.

Abstract: Techniques are described for developing and using applications and skills with autonomous vehicles. In some embodiments, a development platform is provided that enables access to a developer console for developing software modules for use with autonomous vehicles. For example, a developer can specify instructions for causing an autonomous vehicle to perform one or more operations. To control the behavior of an autonomous vehicle, the instructions can cause an executing computer system at the autonomous vehicle to generate calls to an application programming interface (API) associated with an autonomous navigation system of autonomous vehicle. Such calls to the API can be configured to adjust parameters of a behavioral objective associated with a trajectory generation process performed by the autonomous navigation system that controls the behavior of the autonomous vehicle. The instructions specified by the developer can be packaged as a software module that can be deployed for use at autonomous vehicle.

Abstract: Methods and systems are described for new paradigms for user interaction with an unmanned aerial vehicle (referred to as a flying digital assistant or FDA) using a portable multifunction device (PMD) such as smart phone. In some embodiments, a method for synchronizing video and audio is described. The method includes capturing video of a physical environment, receiving first audio of the physical environment captured by a first microphone of a first distributed electronic device, and synchronizing the video of the physical environment with the first audio of the physical environment.

Abstract: In one general aspect, a method can include displaying, on a display device included in a computing device, content in an application executing on the computing device. The method can further include displaying, in a user interface on the display device, at least one identifier, receiving a selection of the at least one identifier, and initiating casting in response to receiving the selection of the at least one identifier.

Abstract: Technology for generating and displaying a graphical user interface for operating an unmanned aerial vehicle (UAV) is disclosed herein that generates and updates a representation of a spline flight path. In various implementations, a graphical user interface detects user interactions with a remote control device directing the flight control subsystem of the UAV to record keyframes and to compute a spline based on the keyframes during flight. The graphical user interface displays a real-time perspective of the UAV with a representation of the spline and the keyframes overlaying the view. The graphical user interface continually updates the representation as the UAV flies and when the spline is updated as the keyframes are updated.

Abstract: Autonomous aerial navigation in low-light and no-light conditions includes using night mode obstacle avoidance intelligence and mechanisms for vision-based unmanned aerial vehicle (UAV) navigation to enable autonomous flight operations of a UAV in low-light and no-light environments using infrared data.

Abstract: An actuator is introduced that utilizes the forces that result from placing a current carrying coil in a magnetic field to rotate a connected object about at least one axis. In some embodiments, the introduced coil actuator includes a coil of conductor coupled to an arm or other type of structural element that extends radially from an axis of rotation. The introduced coil actuator can be utilized to provide motion control in a variety of different applications such as gimbal mechanisms. In some embodiments, the introduced coil actuator can be implemented in a gimbal mechanism for adjusting an orientation of a device such as a camera relative to a connected platform such as the body of an aerial vehicle.

Abstract: Technology for operating an unmanned aerial vehicle (UAV) is disclosed herein that allows a drone to be flown along a computed spline, while also accommodating in-flight modifications. In various implementations, a UAV includes a flight control subsystem and an electromechanical subsystem. The flight control subsystem records keyframes during flight and computes a spline based on the keyframes. The flight control subsystem then saves the computed spline for playback, at which time the UAV automatically flies in accordance with the computed spline.

Abstract: An unmanned aerial vehicle (UAV) performs operations to semantically understand components of a structure under inspection. During an exploration inspection of the structure, a camera of the UAV captures images of the structure. Components of the structure are determined based on the images and a taxonomy associated with the structure, for example, using a computer vision process and a machine learning model. A visual representation of the components (e.g., a semantic scene graph, such as a three-dimensional graphical representation of a hierarchical text representation) is generated and output to a user device in communication with the UAV to enable selections, via a graphical user interface output for display at the user device, of ones of the components for further inspection using the UAV.

Abstract: Semantic segmentation rendering is performed to encode structural data, including precise and relative component locations, in pixel values of an image depicting a structure. Images captured during a UAV-based exploration inspection of a structure are obtained. In each pixel value that corresponds to the structure within the images, identifiers of the structure, a component of the structure depicted using the pixel value, and a location of the component are encoded. The pixel values of the images are segmented into polygons according to the encoded identifiers, and data indicative of the polygons is stored for use in a further inspection of the structure. In connection with the semantic segmentation rendering, a three-dimensional graphical representation of the structure is obtained and rendered, according to the encoded identifiers, using shaders that visually distinguish each component of the structure, in which the data indicative of the polygons identifies respective ones of the shaders.

Abstract: Semantic three-dimensional scan is performed for the multi-phase inspection of a structure using an unmanned aerial vehicle (UAV). The multi-phase inspection includes a first inspection phase and a second inspection phase. A UAV performs the first inspection phase of the structure to determine a semantic understanding of components associated with the structure and pose information of the components. Based on the semantic understanding of the components and the pose information, a flight path indicating capture points and camera poses associated with the capture points is determined. The UAV then performs the second inspection phase of the structure according to the flight path, in which all or some of the components are inspected.

Abstract: Autonomous aerial navigation in low-light and no-light conditions includes using night mode obstacle avoidance intelligence, training, and mechanisms for vision-based unmanned aerial vehicle (UAV) navigation to enable autonomous flight operations of a UAV in low-light and no-light environments using infrared data.

Abstract: An introduced autonomous aerial vehicle can include multiple cameras for capturing images of a surrounding physical environment that are utilized for motion planning by an autonomous navigation system. In some embodiments, the cameras can be integrated into one or more rotor assemblies that house powered rotors to free up space within the body of the aerial vehicle. In an example embodiment, an aerial vehicle includes multiple upward-facing cameras and multiple downward-facing cameras with overlapping fields of view to enable stereoscopic computer vision in a plurality of directions around the aerial vehicle. Similar camera arrangements can also be implemented in fixed-wing aerial vehicles.