Abstract: Certain aspects of the technology disclosed herein integrate a camera with an electronic display. An electronic display includes several layers, such as a cover layer, a color filter layer, a display layer including light emitting diodes or organic light emitting diodes, a thin film transistor layer, etc. The camera can include a plurality of noncontiguous pixel regions disposed beneath the display layer and be configured to capture a plurality of images corresponding to the plurality of noncontiguous pixel regions. In some embodiments, the plurality of images can form a composite image with depth.

Abstract: Presented here is a system and method to record images having varying zoom levels using a lens having a substantially fixed zoom level. The resulting image can have varying zoom levels and can be of high quality. The light sensor recording the image can have a resolution higher than the desired resolution of the resulting image. The high resolution of the light sensor allows digital zooming, i.e., cropping of the image. The light sensor can operate in two modes, where each mode produces varying resolutions of the resulting image. Operating the light sensor in the mode producing a high resolution of the resulting image, however, does not produce images of satisfactory quality in low light conditions, and the light sensor may need to be switched between the first and the second mode depending on the zoom level and the lighting conditions.

Abstract: Certain aspects of the technology disclosed herein integrate a camera with an electronic display. An electronic display includes several layers, such as a cover layer, a color filter layer, a display layer including light emitting diodes or organic light emitting diodes, a thin film transistor layer, etc. In one embodiment, the layers include a substantially transparent region disposed above the camera. The substantially transparent region allows light from outside to reach the camera, enabling the camera to record an image. In another embodiment, the color filter layer does not include a substantially transparent region, and the camera records the light from the outside colored by the color filter layer. According to another embodiment, while none of the layers include a substantially transparent region, the layers are all substantially transparent, and the camera disposed beneath the layers records light reaching the camera from outside.

Abstract: Various embodiments concern sensors and other components that can be disposed beneath a variable transparency layer of a mobile device. By modifying how much voltage is applied to the variable transparency layer, a component, such as a camera, can be readily hidden when not in use. More specifically, the variable transparency layer may be substantially opaque when the camera is not in use and at least partially transparent when the camera is in use and ready to capture an image. The opacity level of the variable transparency layer can be modified by a voltage source that is electrically coupled to the variable transparency layer. The various levels of opacity could also enable the variable transparency layer to act as an electronic aperture for the camera.

Abstract: Presented here is technology to efficiently process camera images to generate lead part artistic images and videos using an artificial intelligence module receiving inputs from multiple sensors. Multiple sensors can include a depth sensor, a conventional camera, and a motion tracker providing inputs to the artificial intelligence module. Based on the inputs, the artificial intelligence module can segment the received image and/or video into a foreground image and a background image to produce portrait imagery by blurring the background image and/or video. The artificial intelligence module can select the most aesthetically pleasing image from a video. In addition, the artificial intelligence module can adjust lighting in an image or video to create artistic lighting effects. All the processing can be done in real time due to efficient combination of artificial intelligence modules, traditional image processing techniques, and use of specialized hardware.

Abstract: The disclosed embodiments include a light detection and ranging (LIDAR) system. The LIDAR system includes a modular LIDAR device, which includes a scanner component and a structurally separate base component. The scanner component is mountable on an external surface of a vehicle, and includes a light source and detector to capture LIDAR data including a measure of a distance to an object relative to the vehicle. The scanner component also includes a communications transmitter to transmit LIDAR data indicative of the distance, and a power receiver to wirelessly power the scanner component. The base component is at least partially mountable on an interior surface of the vehicle, and includes a power transmitter to wirelessly power the scanner component, a communications receiver to wirelessly receive the LIDAR data, and a processor to enable autonomous or semi-autonomous navigation of the vehicle based on processed LIDAR data.

Abstract: The embodiments include a handheld mobile device (HMD) operable to cause a subsystem of a vehicle to perform a vehicular operation associated with an existing capability of the subsystem. The HMD includes a processor, a port operable to communicatively couple the HMD to an existing on-board computer of a vehicle via an existing data link connector of the vehicle, and a wireless interface operable to communicatively couple the HMD to at least one of the vehicle or a remote service. The HMD further includes a memory storing a software application that, when executed by the processor, causes the HMD to communicate a command to the on-board computer via the data link connector to perform the vehicular operation associated with the existing capability of the subsystem.

Abstract: The disclosed teachings include a 360-degree camera accessory mountable on a smartphone. The 360-degree camera accessory includes a camera module disposed partially within the camera accessory, where the camera module including an optical lens configured to capture an image of a scene. The 360-degree camera accessory also includes a rigid structure disposed within the accessory to support the camera module, and a deformable structure disposed between a camera barrel that surrounds the optical lens of the camera module and the rigid structure. The deformable structure can compensate for movement of the camera module to improve image capturing of the scene.

Abstract: Presented here are systems and methods to enable software applications to display images on irregular screen shapes containing the shape without display functionality. In one embodiment, the shape without display functionality is confined within a bounding shape, which, when subtracted from the irregular screen shape leaves a rectangular area of the screen where the image can be displayed. The area within the bounding shape can be occupied by images provided by the operating system associated with the irregular screen, such as a status bar and/or navigation bar. In another embodiment, the image can be displayed on the full irregular screen, such that the shape without display functionality covers a part of the image. When the occluded part of the image contains a critical object, such as a menu item, a button, etc., the image can be modified to move the critical object out of the area occluded by the shape without display functionality.

Abstract: Various embodiments concern motion sensor modules able to monitor movement. A motion sensor module can be detachably connected to an electronic device to which the motion sensor module transfers motion data. In some embodiments, the motion sensor module is connected to an electronic device associated with an individual. In other embodiments, the motion sensor module is embedded within an article of clothing worn by the individual (e.g., a shirt, swimsuit, or shoe). The motion sensor module includes at least one inertial measurement unit (IMU) that generates motion data indicative of the motion of the motion sensor module as a whole.

Abstract: Some embodiments include a mobile device with a light sensor overlaid under a display (e.g., a touchscreen). The mobile device can identify a command to capture an image. The mobile device can adjust at least a target portion of an opaqueness adjustable region of the display directly over the light sensor. The opaqueness adjustable region is capable of transforming from substantially opaque to substantially transparent. The mobile device can capture the image utilizing at least the light sensor while the target portion of the opaqueness adjustable region is substantially transparent.

Abstract: Presented here is a system to record high-resolution infrared images without the need to include additional stand-alone sensors into the mobile device. According to one embodiment, an organic light emitting diode (OLED) display is modified to emit IR and near-IR light in a large field. The modified display allows for depth sensing and infrared imaging without a stand-alone emitter. Additionally, an IR shutter filter can be applied to the existing front facing red, green, blue (RGB) camera that would only be in place when the display is in IR emission mode. The combination of these two technologies allows a facial recognition system using existing hardware, and not require additional sensors or emitters to achieve face recognition.

Abstract: The present disclosure contemplates a variety of improved methods and systems for creating a unique user experience using an ambient operating system connected to a variety of disparate IoT devices. The described solution includes a curated scene or set of actions initiated by an assistant device. For example, the assistant device detects it is a weekday morning and initiates actions associated with the wakeup routine such as opening the blinds, making coffee and notifying the user of the current traffic report.

Abstract: The present disclosure contemplates a variety of improved methods and systems for controlling a variety of disparate home devices coupled to the framework and/or ambient operating system (AmbientOS). This home application programming interface (API) exposes the operation of the home devices in a natural and intuitive manner, facilitating control of unknown devices by high level commands. The home devices are contextually responsive to API commands. Additionally, the home application programming interface can customizing an application developed to control devices in an environment to control the devices in the environment via an assistant device with artificial intelligence (AI) capabilities.

Abstract: In one embodiment, a light sensor, such as a camera, records an image through the surface with the residue to produce a stained image. A processor associated with the camera identifies object outlines within the image using a machine learning model, and smooth the colors within the object outlines. In another embodiment, the light sensor is placed beneath a dual-mode region of a display containing the residue. The dual-mode region can be opaque and function as part of the display, or can be transparent and allow environment light to reach the light sensor. Initially, the processor determines the position of the residue by causing the dual-mode region to display a predetermined pattern, while the light sensor records the predetermined pattern. Using the determined position of the residue, the processor corrects the pixels within the residue in the recorded image, by interpolating the values of the pixels outside of the residue.

Abstract: A contextual user interface based on environment is described. An assistant device can determine that speech has been spoken and determine the context of an environment of that speech. A user interface can then be generated based on the context of the environment and the content of the speech. Different context can result in different user interfaces being generated.

Abstract: A home assistant device assisting with the setup and troubleshooting of devices is described. An assistant device can determine setup instructions for devices. The setup instructions for one device can be determined the technical aptitude of the user and assistant the user with setting up devices within their own technical abilities. The setup instructions can then be provided.

Abstract: Various embodiments concern techniques for intelligently switching between multiple sources of USB signals. More specifically, user devices are described that include a physical USB port for receiving a USB connector and one or more wireless transceivers that communicate with an accessory. The wireless transceiver(s) may communicate with the accessory using a USB-based protocol (e.g., Wireless USB). The user devices described herein can intelligently switch between these different sources of USB signals so that USB signals can be simultaneously or sequentially received from a peripheral (via the USB port) and an accessory (via the wireless transceiver(s)). In some embodiments, a switching routine is executed (e.g., by a processor or signal switch) that determines which peripheral and/or accessory is connected to a user device at a given point in time.