Abstract: The present disclosure contemplates a variety of improved methods and systems for controlling a variety of disparate home devices coupled to the framework and/or ambient operating system (AmbientOS). This home application programming interface (API) exposes the operation of the home devices in a natural and intuitive manner, facilitating control of unknown devices by high level commands. The home devices are contextually responsive to API commands. Additionally, the home application programming interface can customizing an application developed to control devices in an environment to control the devices in the environment via an assistant device with artificial intelligence (AI) capabilities.

Abstract: A contextual user interface based on environment is described. An assistant device can determine that speech has been spoken and determine the context of an environment of that speech. A user interface can then be generated based on the context of the environment and the content of the speech. Different context can result in different user interfaces being generated.

Abstract: In one embodiment, a light sensor, such as a camera, records an image through the surface with the residue to produce a stained image. A processor associated with the camera identifies object outlines within the image using a machine learning model, and smooth the colors within the object outlines. In another embodiment, the light sensor is placed beneath a dual-mode region of a display containing the residue. The dual-mode region can be opaque and function as part of the display, or can be transparent and allow environment light to reach the light sensor. Initially, the processor determines the position of the residue by causing the dual-mode region to display a predetermined pattern, while the light sensor records the predetermined pattern. Using the determined position of the residue, the processor corrects the pixels within the residue in the recorded image, by interpolating the values of the pixels outside of the residue.

Abstract: The present disclosure contemplates a variety of improved methods and systems for enabling set up of a variety of disparate IoT devices coupled to the framework and/or ambient operating system of an assistant device. The described solution includes a voice-driven assistant device setup process. An assistant device can determine the IoT devices in its physical environment and provide setup instructions to the user. The setup including determining voice activatable commands, device information and the adapter to allow the assistant device to operate the one or more IoT devices in response to user instructions.

Abstract: A home assistant device assisting with the setup and troubleshooting of devices is described. An assistant device can determine setup instructions for devices. The setup instructions for one device can be determined the technical aptitude of the user and assistant the user with setting up devices within their own technical abilities. The setup instructions can then be provided.

Abstract: Various embodiments concern techniques for intelligently switching between multiple sources of USB signals. More specifically, user devices are described that include a physical USB port for receiving a USB connector and one or more wireless transceivers that communicate with an accessory. The wireless transceiver(s) may communicate with the accessory using a USB-based protocol (e.g., Wireless USB). The user devices described herein can intelligently switch between these different sources of USB signals so that USB signals can be simultaneously or sequentially received from a peripheral (via the USB port) and an accessory (via the wireless transceiver(s)). In some embodiments, a switching routine is executed (e.g., by a processor or signal switch) that determines which peripheral and/or accessory is connected to a user device at a given point in time.

Abstract: Providing audio and/or video content from one user to a second user within an environment are described. An electronic device can receive conversational content from one user and determine that it is directed to a second user. The second user can be determined to be within another location of the environment (e.g., a different room of a house) and identify the capabilities of devices within the other location. Play back of the conversational content can then be performed using the identified devices within the other location.

Abstract: Presented here is technology to efficiently process camera images to generate artistic images and videos using an artificial intelligence module receiving inputs from multiple sensors. Multiple sensors can include a depth sensor, a conventional camera, and a motion tracker providing inputs to the artificial intelligence module. Based on the inputs, the artificial intelligence module can segment the received image and/or video into a foreground image and a background image to produce portrait imagery by blurring the background image and/or video. The artificial intelligence module can select the most aesthetically pleasing image from a video. In addition, the artificial intelligence module can adjust lighting in an image or video to create artistic lighting effects. All the processing can be done in real time due to efficient combination of artificial intelligence modules, traditional image processing techniques, and use of specialized hardware.

Abstract: Various embodiments concern sensors and other components that can be disposed beneath a variable transparency layer of a mobile device. By modifying how much voltage is applied to the variable transparency layer, a component, such as a camera, can be readily hidden when not in use. More specifically, the variable transparency layer may be substantially opaque when the camera is not in use and at least partially transparent when the camera is in use and ready to capture an image. The opacity level of the variable transparency layer can be modified by a voltage source that is electrically coupled to the variable transparency layer. The various levels of opacity could also enable the variable transparency layer to act as an electronic aperture for the camera.

Abstract: Presented here are manufacturing techniques to create an irregularly shaped electronic display, including a hollow within which a sensor, such as a camera, can be placed. The manufacturing techniques enable the creation of the hollow anytime during the manufacturing process. The resulting electronic display occupies the full side of the mobile device, with the sensors placed within and surrounded by the display.

Abstract: Various embodiments concern a wireless accessory bus for a user device that enables accessories to be easily and securely attached to the user device. More specifically, the wireless accessory bus enables data to be wirelessly transmitted between the accessory and the user device when the accessory and the user device are located within close proximity to one another (e.g., when the accessory is securely attached to the wireless accessory bus). Power could also be wirelessly transferred from the user device to the accessory (or vice versa). In some embodiments, the user device includes fastening component(s) (e.g., magnets) that allow the user device and the accessory to be magnetically secured to one another. The accessory typically enables the user device to readily utilize a new functionality or an improvement to an existing functionality.

Abstract: Presented here is technology to efficiently process camera images to generate lead part artistic images and videos using an artificial intelligence module receiving inputs from multiple sensors. Multiple sensors can include a depth sensor, a conventional camera, and a motion tracker providing inputs to the artificial intelligence module. Based on the inputs, the artificial intelligence module can segment the received image and/or video into a foreground image and a background image to produce portrait imagery by blurring the background image and/or video. The artificial intelligence module can select the most aesthetically pleasing image from a video. In addition, the artificial intelligence module can adjust lighting in an image or video to create artistic lighting effects. All the processing can be done in real time due to efficient combination of artificial intelligence modules, traditional image processing techniques, and use of specialized hardware.

Abstract: Presented here is technology to efficiently process camera images to generate artistic images and videos using an artificial intelligence module receiving inputs from multiple sensors. Multiple sensors can include a depth sensor, a conventional camera, and a motion tracker providing inputs to the artificial intelligence module. Based on the inputs, the artificial intelligence module can segment the received image and/or video into a foreground image and a background image to produce portrait imagery by blurring the background image and/or video. The artificial intelligence module can select the most aesthetically pleasing image from a video. In addition, the artificial intelligence module can adjust lighting in an image or video to create artistic lighting effects. All the processing can be done in real time due to efficient combination of artificial intelligence modules, traditional image processing techniques, and use of specialized hardware.

Abstract: Various embodiments include a device that implements a high bandwidth wireless data interface for communication between the device and a device accessory. The device can include an external shell and a display (e.g., a touchscreen). In some embodiments, the external shell and the display together hermetically seal various other components of the device. In various embodiments, the device includes an attachment mechanism, an alignment mechanism (e.g., for magnetic alignment between a device and a device accessory), a wireless data transceiver, and a wired or wireless energy source (e.g., for sharing power between the device and the device accessory). The wireless data transceiver can transmit data at extremely high electromagnetic frequencies between the device in the device accessory.

Abstract: A mobile device can include electronic circuitry and a chassis shell adapted to protect the electronic circuitry. The chassis shell includes an opening exposing a tray inside the mobile device. The tray can include an external card slot and a patterned wall. The tray can be disposed within a cavity of a support structure of the mobile device. An input/output (I/O) component, such as a speaker, and a card reader can also be disposed within the cavity. The I/O component can communicate via at least a hole in the patterned wall (e.g., a speaker grill).

Abstract: A mobile device can include electronic circuitry and a chassis shell adapted to protect the electronic circuitry. The chassis shell includes an opening exposing a tray inside the mobile device. The tray can include an external card slot and a patterned wall. The tray can be disposed within a cavity of a support structure of the mobile device. An input/output (I/O) component, such as a speaker, and a card reader can also be disposed within the cavity. The I/O component can communicate via at least a hole in the patterned wall (e.g., a speaker grill).

Abstract: Various embodiments concern fingerprint sensors for mobile devices. By modifying one or more layers of a display assembly, a fingerprint sensor can be positioned directly beneath the display. For example, the air gap within the backlight structure could be replaced by an optically clear adhesive that allows waves emitted by a fingerprint sensor to properly propagate through the display assembly. In some embodiments, a fingerprint sensor is set within a recess in the display assembly. For example, a depression could be created in the protective substrate, and the fingerprint sensor could be set within the depression using an optically clear adhesive. The functionality of a fingerprint sensor could also be replicated by increasing the density of touch-sensing elements in certain areas of a display assembly that utilizes in-cell technology.

Abstract: Disclosed herein are various embodiments of an imaging device having a wide field of view configured to connect to another device. According to one embodiment, the wide field of view camera can include an array of cameras for recording a wide view (e.g., 360° view) of the surrounding environment and generate a 360° image of the surrounding environment. The wide field of view camera can be configured to connect and/or attach to another device, including a mobile device. Embodiments include a wired or wireless connection mechanism to facilitate communication between the wide field of view camera and another device. The connection mechanism may enable transmission of data associated with the wide field of view camera to another device. Embodiments include an attachment mechanism to fasten the wide field of view camera to another device.

Abstract: Ambient assistance is described. An assistant device can detect speech in its environment and determine that the speech includes words or phrases of a local dictionary of the assistant device. The assistant device can then generate an interaction opportunity based on the words or phrases of the speech.

Abstract: A home assistant device assisting with the setup and troubleshooting of devices is described. An assistant device can determine setup instructions for devices. The setup instructions for one device can be determined the technical aptitude of the user and assistant the user with setting up devices within their own technical abilities. The setup instructions can then be provided.