Abstract: Aspects of the subject technology provide improved techniques for human-computer interactions including disambiguation of a human user's linguistic command. The improved techniques may include integrating linguistic input from a user with additional input from sensors relating to the user and the user's environment. In an aspect, imagery of a user's environment may be first analyzed to identify objects in the environment. Input regarding the user, such as a user's gaze location, may trigger a second analysis of a subset of the identified objects in the environment. The results of these analyses may then be used to resolve an ambiguity in linguistic user input from the user.

Abstract: A user can emulate touch screen events with motions and gestures that the user performs at a distance from a computing device. A user can utilize specific gestures, such as a pinch gesture, to designate portions of motion that are to be interpreted as input, to differentiate from other portions of the motion. A user can then perform actions such as text input by performing motions with the pinch gesture that correspond to words or other selections recognized by a text input program. A camera-based detection approach can be used to recognize the location of features performing the motions and gestures, such as a hand, finger, and/or thumb of the user.

Abstract: Instances of content, such as search results or browse items, can be displayed using a plurality of three-dimensional elements, with selected pieces of information for each instance placed upon faces, sides, or other portions of those elements. A user can view similar information for each of the instances of content by rotating the elements, such as by interacting with an input element or rotating a portable computing device rendering the elements. The user can apply various filtering criteria or value ranges, whereby the relative position of the elements in three-dimensional space can be adjusted based at least in part upon the applied values. By rotating the elements, applying criteria, and changing the camera view of the elements, a user can quickly compare a large number of instances of context according to a number of different criteria, and can quickly locate items of interest from a large selection of items.

Abstract: A user interacting with an electronic device can receive suggestions for applications or services that can help the user with a specific task. The user might enter information that can be processed semantically to determine one or more actions, objects, or other types of information useful in discovering related applications or services, which might be local or remote to that device. One or more of these discovered applications or services is selected to suggest to the user based on any number of criteria, such as user behavior, preferences, location, time of day, etc. Using such an approach, a device can utilize a semantic process to attempt to infer an action or intent relating to a simple or complex task, and can discover and suggest applications or services that can assist with that task even where the user is unaware of, or not looking for, such an application or service.

Abstract: Various interfaces and control schemes are provided that enable a user to enter text or select specific elements of an interface based at least in part upon relative motion. In some embodiments, a user can enter text by tilting a device to locate a specific character, and press on a button or specified area of a device to select that character. In other embodiments, a user can select a character by looking at a character and performing a selection action, a user device able to determine the character by tracking a gaze direction of the user. Any relative movement of a device and/or a user can be used as input, allowing for one-handed or even hands-free entrance of text or other selection of elements of a graphical user interface.

Abstract: Computing devices can interact and share data and/or resources in response to detecting a contact event. The devices can include force and/or pressure sensing components that enable the devices to determine when force or pressure is being applied. By comparing information (e.g., time and location) for contact events detected by each device, it can be determined that the devices were brought into contact with each other. In response, the devices can enable data to be transferred between the devices, and can allow resources to be shared for a variety of purposes. In at least some embodiments, users can obtain a larger display screen or enhanced audio capabilities by bringing the devices into contact such that display and/or audio elements of the devices can be used together to present different aspects of the same content.

Abstract: A user attempting to obtain information about an object can capture image information including a view of that object, and the image information can be used with a matching or identification process to provide information about that type of object to the user. Information about the orientation of the camera and/or device used to capture the image can be provided in order to limit an initial search space for the matching or identification process. In some embodiments, images can be selected for matching based at least in part upon having a view matching the orientation of the camera or device. In other embodiments, images of objects corresponding to the orientation can be selected. Such a process can increase the average speed and efficiency in locating matching images. If a match cannot be found in the initial space, images of other views and categories can be analyzed as well.

Abstract: A computing device can continuously monitor application usage. The data monitored can include an amount (absolute or relative) of processing, memory, storage, network, and/or power resources used by each application of the computing device over time. The computing device can simultaneously display the application usage data for multiple applications over a specified period of time. A user may interact with how the computing device presents the application usage data, such as by scrolling a time axis to review application usage data over a different period of time, scrolling an application axis to review application usage data of one or more different applications over the specified period of time, zoom in/zoom out to view the application usage data at a more/less granular unit of time, tilt the computing device to lengthen or shorten the time axis (and shorten/lengthen the application axis), or sort the application usage data, among other possibilities.

Abstract: An electronic device can attempt to determine a gaze direction of a user, which can be used to determine an object of interest to the user. Determining the gaze direction helps to reduce the search space and reduce processing requirements for identifying the object. Image information can be captured that includes the object of interest, which then can be analyzed to recognize the type of object. Upon recognizing the object, the user can be provided with information about the object, which in some cases can depend at least in part upon a current context or location of the object. If the object is a networked device, the user can potentially be provided with input options for controlling the device.

Abstract: An electronic device can attempt to determine a gaze direction of a user, which can be used to determine an object of interest to the user. Determining the gaze direction helps to reduce the search space and reduce processing requirements for identifying the object. Image information can be captured that includes the object of interest, which then can be analyzed to recognize the type of object. Upon recognizing the object, the user can be provided with information about the object, which in some cases can depend at least in part upon a current context or location of the object. If the object is a networked device, the user can potentially be provided with input options for controlling the device.

Abstract: An electronic device can utilize one or more sensors and/or imaging elements to determine the relative position of at least one light source relative to the device. In various embodiments, the intensity of light can be determined in various directions around the device. By determining the relative intensity around the device, an approximate direction of one or more light sources can be determined. Utilizing the relative position of each light source, the electronic device can properly light or shade a graphical object to be rendered by the device, or otherwise process image information captured by the device.

Abstract: A user can use a finger, or other such object, to provide input to a computing device. The finger does not have to contact the device, but can be positioned and/or oriented in such a way that the device can determine an input that the user is attempting to provide, such as an element or icon that the user is intended to select. One or more cameras can capture image information, which can be analyzed to attempt to determine the location and/or orientation of the finger. If the finger is at least partially outside a field of view of the camera(s), the device can use a sensor (e.g., EMF) to attempt to determine a location of at least a portion of the finger, which can be used with the image information to determine the location and/or orientation of the finger. Other estimation processes can be used as well.

Abstract: The computational resources needed to perform processes such as image recognition can be reduced by determining appropriate frames of image information to use for the processing. In some embodiments, infrared imaging can be used to determine when a person is looking substantially towards a device, such that an image frame captured at that time will likely be adequate for facial recognition. In other embodiments, sound triangulation or motion sensing can be used to assist in determining which captured image frames to discard and which to select for processing based on any of a number of factors indicative of a proper frame for processing.

Abstract: An electronic device can attempt to determine a gaze direction of a user, which can be used to determine an object of interest to the user. Determining the gaze direction helps to reduce the search space and reduce processing requirements for identifying the object. Image information can be captured that includes the object of interest, which then can be analyzed to recognize the type of object. Upon recognizing the object, the user can be provided with information about the object, which in some cases can depend at least in part upon a current context or location of the object. If the object is a networked device, the user can potentially be provided with input options for controlling the device.

Abstract: An electronic device can utilize one or more image capture elements on the sides or edges of the device to capture image information in a way that is more discrete than for conventional devices with a single camera on a back of the device. Further, the ability to utilize cameras on multiple sides of the device provides enhanced capability. Devices can utilize information such as the current orientation of a device to select one or more appropriate image capture elements to activate at a given time. The image information can be used to provide information about the user's surroundings, such as the names of people in a meeting, without making it obvious that the user is trying to determine that information.

Abstract: Content displayed on a display element of a computing device can be rendered from a viewpoint that updates with changes in the relative position of a user's head, or other such feature, with respect to the device. At least one camera of the device can attempt to track the relative position in order to determine the proper viewpoint. When the user's head can no longer be seen, or when confidence in the determined relative position drops below an acceptable value, for example, the device can utilize information from another source, such as a motion or rotation sensor, to attempt to estimate the current position of the user's head. Physical limitations of the user with respect to the device also can be used to improve the accuracy of the determination.

Abstract: An electronic device can utilize image capture technology to detect the presence and location of another device. Using this information, the electronic device can display, in a user interface, a graphical element representing a detected device, along with identity information and the location of the detected device relative to the electronic device. The location of each detected device relative to the electronic device can be tracked and thus the graphical element can be updated in the user interface.

Abstract: The use of resources on a computing device can be optimized for current conditions to reduce or minimize the amount of resources needed to provide a sufficient level of performance for various types of tasks. In some embodiments, one or more optimization algorithms can analyze information such as a type of task to be performed and the state of one or more environmental conditions to attempt to select a number and configuration of components, such as cameras and illumination elements, to use in performing the tasks. The performance of the tasks can be monitored, and the selection or states updated in order to maintain a sufficient level of performance without using more resources than are necessary.

Abstract: A user can use a finger, or other such object, to provide input to a computing device. The finger does not have to contact the device, but can be positioned and/or oriented in such a way that the device can determine an input that the user is attempting to provide, such as an element or icon that the user is intended to select. One or more cameras can capture image information, which can be analyzed to attempt to determine the location and/or orientation of the finger. If the finger is at least partially outside a field of view of the camera(s), the device can use a sensor (e.g., EMF) to attempt to determine a location of at least a portion of the finger, which can be used with the image information to determine the location and/or orientation of the finger. Other estimation processes can be used as well.

Abstract: Elements of a graphical user interface, such as a virtual keypad, can have display characteristics adjusted based upon aspects such as a relative hand or finger position of a user. In one example, the transparency of a keypad used to enter text on a portable computing device can be adjusted based on the distance of a user's hand, such that the user can still view content displayed through the interface. A user can also select elements, such as text fields for text entry, by adjusting a relative position of the user's hand and performing a selection gesture or other such action. Various types of functionality can be activated in response to a selection gesture, such as to display a keypad of an appropriate transparency.