Abstract: Certain aspects of the technology disclosed herein include generating a virtual sound field based on data from an ambisonic recording device. The ambisonic device records sound of a surrounding environment using at least four microphones having a tetrahedral orientation. An omnidirectional microphone having an audio-isolated portion can be used to isolate sound from a particular direction. Sound received from the plurality of microphones can be used to generate a virtual sound field. The virtual sound field include a dataset indicating a pressure signal and a plurality of velocity vectors. The ambisonic recording device can include a wide angle camera and generate wide angle video corresponding to the virtual sound field.

Abstract: Various embodiments concern fingerprint sensors for mobile devices. By modifying one or more layers of a display assembly, a fingerprint sensor can be positioned directly beneath the display. For example, the air gap within the backlight structure could be replaced by an optically clear adhesive that allows waves emitted by a fingerprint sensor to properly propagate through the display assembly. In some embodiments, a fingerprint sensor is set within a recess in the display assembly. For example, a depression could be created in the protective substrate, and the fingerprint sensor could be set within the depression using an optically clear adhesive. The functionality of a fingerprint sensor could also be replicated by increasing the density of touch-sensing elements in certain areas of a display assembly that utilizes in-cell technology.

Abstract: A home assistant device assisting with the setup of devices is described. An assistant device can determine setup instructions for devices. The setup instructions for one device can be determined to include a step requesting information related to the setup of another device. The setup of the devices can be ordered based on that determination. The setup instructions can then be provided.

Abstract: A contextual user interface based on video game playback is described. An assistant device can receive video game state information indicating a state of a playback of a video game on a video game console. Using that information, the assistant device can determine characteristics of the playback and generate a button on a graphical user interface (GUI) based on the characteristics. A user can then select the button and cause the assistant device to instruct the video game console to perform a corresponding action.

Abstract: The disclosed embodiments include a housing of a handheld mobile device. The housing includes a ceramic layer forming a continuous outermost surface of the handheld mobile device, and an antenna layer adjacent to the ceramic layer. The antenna layer including conductive elements formed from a metal injection molded substrate, and an antenna break formed of non-conductive material electrically separating the conductive elements to collectively form an antenna of the handheld mobile device that is hidden by the ceramic layer from an exterior view of the handheld mobile device.

Abstract: Presented here is technology to efficiently process camera images to generate artistic images and videos using an artificial intelligence module receiving inputs from multiple sensors. Multiple sensors can include a depth sensor, a conventional camera, and a motion tracker providing inputs to the artificial intelligence module. Based on the inputs, the artificial intelligence module can segment the received image and/or video into a foreground image and a background image to produce portrait imagery by blurring the background image and/or video. The artificial intelligence module can select the most aesthetically pleasing image from a video. In addition, the artificial intelligence module can adjust lighting in an image or video to create artistic lighting effects. All the processing can be done in real time due to efficient combination of artificial intelligence modules, traditional image processing techniques, and use of specialized hardware.

Abstract: Switching the providing of a conversation among communication links is described. An electronic device can determine that a mobile device is being used to provide a conversation (e.g., a phone call) using a first communication link. The mobile device can enter an environment that includes a second communication link. The conversation can be switched to the second communication link that provides more content than the first communication link (e.g., audio and visual content rather than just audio content).

Abstract: A handheld device can include an image capture subsystem and a depth sensing subsystem. The image capture subsystem includes an image capture light source operable to emit visible light and an image capture camera operable to capture an image of a scene illuminated by the visible light emitted by the visible light source. The depth sensing subsystem includes a depth light source operable to emit infrared light and a depth camera operable to capture reflected infrared light, including at least some of the infrared light emitted from the depth light source, after reflecting off objects in the scene. The image capture light source, the depth light source, and the depth camera are housed in a single integrated structure including a single transparent panel through which the visible and infrared light are emitted onto the scene, and through which the depth camera receives the reflected infrared light from the scene.

Abstract: The disclosed embodiments include a method performed by a handheld mobile device (HMD). The method can include enabling the HMD to capture one or more views spanning 360-degrees including a view facing a user of the HMD, receiving an input by the HMD to view an image of a view of the views spanning 360-degrees, and responsive to another input, toggling from any image of any view of the views spanning 360-degrees to an image of the view facing the user of the HMD.

Abstract: Ambient assistance is described. An assistant device can operate in an active experience mode in which a response to a user's speech is provided if it includes a hardware activation phrase. Based on characteristics of the environment, the mode can be adjusted to a passive experience mode in which a response to speech is provided even if it does not include the hardware activation phrase.

Abstract: Some embodiments include a mobile device with a light sensor overlaid under a display (e.g., a touchscreen). The mobile device can identify a command to capture an image. The mobile device can adjust at least a target portion of an opaqueness adjustable region of the display directly over the light sensor. The opaqueness adjustable region is capable of transforming from substantially opaque to substantially transparent. The mobile device can capture the image utilizing at least the light sensor while the target portion of the opaqueness adjustable region is substantially transparent.

Abstract: Presented here are devices and methods to correct ambient light measurements made in the presence of optical elements, such as the curved edge of the cover glass associated with the mobile device. In one embodiment, a film with optical properties is placed within the ambient light sensor to diffuse the high-intensity light beam coming from the optical element. In another embodiment, an aperture associated with the ambient light sensor is disposed to prevent the high-intensity light beam from entering the ambient light sensor. In another embodiment, a processor coupled to the ambient light sensor smooths the peak associated with the high intensity light beam produced by the optical element.

Abstract: The present disclosure contemplates a variety of improved methods and systems for enabling developers to program and control a variety of disparate devices coupled to the framework and/or ambient operating system. The Home API can expose operation of the devices in a natural and intuitive manner, facilitating control of unknown devices by high-level commands.

Abstract: Presented here are manufacturing techniques to create an irregularly shaped electronic display, including a hollow within which a sensor, such as a camera, can be placed. The manufacturing techniques enable the creation of the hollow anytime during the manufacturing process. The resulting electronic display occupies the full side of the mobile device, with the sensors placed within and surrounded by the display.

Abstract: The technology disclosed here maximizes the size of the display area associated with the mobile device by various camera placement. In one embodiment, the camera is placed inside the mobile device, and can pop outside the mobile device when the camera is activated. When the camera is inactive the camera retracts inside the mobile device, and becomes unnoticeable to the user. In another embodiment, the camera is integrated into the mobile device display as a camera icon. The integrated camera serves two purposes: to record pictures, and to act as a camera icon, that when selected activates the camera. By removing the camera from the front side of the mobile device, or by integrating the camera into the display screen of the mobile device, the size of the mobile device display screen can be increased.

Abstract: The technology disclosed here integrates a light sensor with a dual-mode display, thus increasing the size of the dual-mode display. The light sensor can be a camera, an ambient sensor, a proximity sensor, etc. The light sensor is placed beneath the dual-mode display and can detect incoming light while the dual-mode display is displaying a display image. The dual-mode display and the light sensor can operate at the same time. For example, a camera placed beneath the dual-mode display can record an image of the environment, while at the same time the dual-mode display is showing the display image. Further, multiple light sensors can be placed at various locations beneath the dual-mode display.

Abstract: The present disclosure contemplates a variety of improved methods and systems for enabling users to set up and manage a variety of disparate IoT devices coupled to the framework and/or ambient operating system of an assistant device. The described solution includes a voice-driven assistant device setup process. An assistant device can determine the IoT devices in its physical environment and provide setup instructions to the user. The assistant device can further capture user responses and use the captured responses to connect to and manage/troubleshoot/control the IoT devices.

Abstract: Management of devices on a wireless network is described. An assistant device can determine that a device is missing configuration settings to access resources within a connected environment. Characteristics of that device can be determined and configuration settings to a selection of the resources can be provided based on the characteristics.

Abstract: A data construct called a semcard is a semantic (meaning-based) software object including semantic meta-tags and meta-data that describes a target object or thing. A target object can be any type of digital or physical entity or identifier, or it can be tacit knowledge, such as ideas, concepts, processes or other data existing in a user's mind, provided that the user represents this knowledge in the semcard. A semcard embodies information about its own structure-rules, history, state, policies and goals regarding automation, display, access permissions, sharing and other operations of the semcard and any optional target object. It can also represent a semantic link between two semcards, or a semantically typed link or a standard Web hyperlink between a semcard and its referent target. A collection of semcards represents a knowledge network; single semcards, and knowledge networks, can be browsed, shared, searched, disseminated, manipulated, displayed, organized, and stored.

Abstract: The present disclosure contemplates a variety of improved methods and systems for enabling developers to program and control a variety of disparate devices coupled to the framework and/or ambient operating system. The Home API can expose operation of the devices in a natural and intuitive manner, facilitating control of unknown devices by high-level commands.