Abstract: Privacy control in a connected environment is described. An assistant device can detect speech spoken by a user. The speech can include a hardware activation phrase that indicates whether the user intends for at least a portion of the speech to be provided to local resources or cloud resources. The speech can then be provided to the appropriate resource based on the hardware activation phrase.

Abstract: Certain aspects of the technology disclosed herein include generating a virtual sound field based on data from an ambisonic recording device. The ambisonic device records sound of a surrounding environment using at least four microphones having a tetrahedral orientation. An omnidirectional microphone having an audio-isolated portion can be used to isolate sound from a particular direction. Sound received from the plurality of microphones can be used to generate a virtual sound field. The virtual sound field include a dataset indicating a pressure signal and a plurality of velocity vectors. The ambisonic recording device can include a wide angle camera and generate wide angle video corresponding to the virtual sound field.

Abstract: Various embodiments concern fingerprint sensors for mobile devices. By modifying one or more layers of a display assembly, a fingerprint sensor can be positioned directly beneath the display. For example, the air gap within the backlight structure could be replaced by an optically clear adhesive that allows waves emitted by a fingerprint sensor to properly propagate through the display assembly. In some embodiments, a fingerprint sensor is set within a recess in the display assembly. For example, a depression could be created in the protective substrate, and the fingerprint sensor could be set within the depression using an optically clear adhesive. The functionality of a fingerprint sensor could also be replicated by increasing the density of touch-sensing elements in certain areas of a display assembly that utilizes in-cell technology.

Abstract: A home assistant device assisting with the setup of devices is described. An assistant device can determine setup instructions for devices. The setup instructions for one device can be determined to include a step requesting information related to the setup of another device. The setup of the devices can be ordered based on that determination. The setup instructions can then be provided.

Abstract: A contextual user interface based on video game playback is described. An assistant device can receive video game state information indicating a state of a playback of a video game on a video game console. Using that information, the assistant device can determine characteristics of the playback and generate a button on a graphical user interface (GUI) based on the characteristics. A user can then select the button and cause the assistant device to instruct the video game console to perform a corresponding action.

Abstract: In one embodiment, a light sensor, such as a camera, records an image through the surface with the residue to produce a stained image. A processor associated with the camera identifies object outlines within the image using a machine learning model, and smooth the colors within the object outlines. In another embodiment, the light sensor is placed beneath a dual-mode region of a display containing the residue. The dual-mode region can be opaque and function as part of the display, or can be transparent and allow environment light to reach the light sensor. Initially, the processor determines the position of the residue by causing the dual-mode region to display a predetermined pattern, while the light sensor records the predetermined pattern. Using the determined position of the residue, the processor corrects the pixels within the residue in the recorded image, by interpolating the values of the pixels outside of the residue.

Abstract: Presented here is technology to efficiently process camera images to generate artistic images and videos using an artificial intelligence module receiving inputs from multiple sensors. Multiple sensors can include a depth sensor, a conventional camera, and a motion tracker providing inputs to the artificial intelligence module. Based on the inputs, the artificial intelligence module can segment the received image and/or video into a foreground image and a background image to produce portrait imagery by blurring the background image and/or video. The artificial intelligence module can select the most aesthetically pleasing image from a video. In addition, the artificial intelligence module can adjust lighting in an image or video to create artistic lighting effects. All the processing can be done in real time due to efficient combination of artificial intelligence modules, traditional image processing techniques, and use of specialized hardware.

Abstract: Switching the providing of a conversation among communication links is described. An electronic device can determine that a mobile device is being used to provide a conversation (e.g., a phone call) using a first communication link. The mobile device can enter an environment that includes a second communication link. The conversation can be switched to the second communication link that provides more content than the first communication link (e.g., audio and visual content rather than just audio content).

Abstract: The present disclosure contemplates a variety of improved methods and systems for initializing a curated scene. The described solution includes a method comprising receiving a user instruction including a user intention to initialize a scene controlling the functionality of one or more devices within an environment, determining the scene associated with the user instruction, and performing the functionalities of the scene using an assistant device and the one or more devices within the environment that are capable of performing the activities.

Abstract: The present disclosure contemplates a variety of improved methods and systems for initializing a curated scene. The described solution includes a method comprising receiving an instruction including a user intention to initialize a scene controlling the functionality of one or more devices within an environment, determining the scene associated with the instruction, and performing the functionalities of the scene using an assistant device and the one or more devices within the environment that are capable of performing the activities.

Abstract: A mobile docking station (MDS) includes a surface through which the MDS can electrically and communicatively couple to a handheld mobile device (HMD) while remaining physically unattached from the MDS. The MDS power circuitry configured to electrically couple the MDS and the HMD while a surface of the HMD physically contacts the surface of the MDS, wherein the electrical coupling enables transferring of power between the MDS and the HMD. The MDS also includes communications circuitry configured to communicatively couple the MDS and the HMD while the surface of the HMD physically contacts the surface of the MDS, wherein the communicative coupling enables unidirectional or bidirectional communications between the MDS and HMD.

Abstract: A home assistant device assisting with the setup and troubleshooting of devices is described. An assistant device can determine setup instructions for devices. The setup instructions for one device can be determined the technical aptitude of the user and assistant the user with setting up devices within their own technical abilities. The setup instructions can then be provided.

Abstract: Disclosed are structures, methods and systems for wireless devices that are configured to allow the transmission of light through an enhanced antenna break for a wide variety of purposes. In an illustrative embodiment, an antenna break is configured to allow the passage of both wireless communication signals and light, such as to notify the user of one or more conditions. The light transmission can be integrated with other functions of the wireless device, such as to provide a flash for operation of a camera, to provide a light source for scanning or 3D sensing, to provide light in conjunction to acoustic or vibration output, or to sense light for other integrated functions.

Abstract: A home assistant device assisting with the setup of devices is described. An assistant device can determine setup instructions for devices. The setup instructions for one device can be determined to include a step requesting information related to the setup of another device. The setup of the devices can be ordered based on that determination. The setup instructions can then be provided.

Abstract: A home assistant device assisting with the setup and troubleshooting of devices is described. An assistant device can determine setup instructions for devices. The setup instructions for one device can be determined the technical aptitude of the user and assistant the user with setting up devices within their own technical abilities. The setup instructions can then be provided.

Abstract: The present disclosure contemplates a variety of improved methods and systems for creating a unique user experience using an ambient operating system connected to a variety of disparate IoT devices. The described solution includes a curated scene or set of actions initiated by an assistant device. For example, the assistant device detects it is a weekday morning and initiates actions associated with the wakeup routine such as opening the blinds, making coffee and notifying the user of the current traffic report.

Abstract: The present disclosure contemplates a variety of improved methods and systems for enabling set up of a variety of disparate IoT devices coupled to the framework and/or ambient operating system of an assistant device. The described solution includes a voice-driven assistant device setup process. An assistant device can determine the IoT devices in its physical environment and provide setup instructions to the user. The setup including determining voice activatable commands, device information and the adapter to allow the assistant device to operate the one or more IoT devices in response to user instructions.

Abstract: Some embodiments include a mobile device with a light sensor overlaid under a display (e.g., a touchscreen). The mobile device can identify a command to capture an image. The mobile device can adjust at least a target portion of an opaqueness adjustable region of the display directly over the light sensor. The opaqueness adjustable region is capable of transforming from substantially opaque to substantially transparent. The mobile device can capture the image utilizing at least the light sensor while the target portion of the opaqueness adjustable region is substantially transparent.

Abstract: Ambient assistance is described. An assistant device can operate in an active experience mode in which a response to a user's speech is provided if it includes a hardware activation phrase. Based on characteristics of the environment, the mode can be adjusted to a passive experience mode in which a response to speech is provided even if it does not include the hardware activation phrase.