Abstract: A user can make a symbol with their hand, or other such gesture, at a distance from a computing device that can be captured by at least one imaging element of the device. The captured information can be analyzed to attempt to determine the location of distinguishing features of the symbol in the image information. The image information is then compared to hand gesture information stored in, for example, a library of hand gestures for the user. Upon identifying a match, an input to an application executing on the computing device is provided when the image information contains information matching at least one hand gesture with at least a minimum level of certainty. The hand gesture could include a single “static” gesture, such as a specific letter in sign language, for example, or include two or more “static” gestures. The gesture could also include motion, such as hand movement.

Abstract: The accuracy of an image matching and/or object identification process can be improved by utilizing a BCM network-based process that maintains higher order relationships between features in an image. A dataset of images can be converted to floating point vectors and then processed using a BCM-based approach. The resulting vectors can be stored as an image library for purposes of matching subsequently received images. When a match is located for a query image, for example, information associated with the matching image can be provided in order to help identify one or more objects in the received query image.

Abstract: A user interacting with an electronic device can receive suggestions for applications or services that can help the user with a specific task. The user might enter information that can be processed semantically to determine one or more actions, objects, or other types of information useful in discovering related applications or services, which might be local or remote to that device. One or more of these discovered applications or services is selected to suggest to the user based on any number of criteria, such as user behavior, preferences, location, time of day, etc. Using such an approach, a device can utilize a semantic process to attempt to infer an action or intent relating to a simple or complex task, and can discover and suggest applications or services that can assist with that task even where the user is unaware of, or not looking for, such an application or service.

Abstract: A user of a computing device can be authenticated using image information captured by at least one camera of the computing device. In addition to analyzing the image information using a facial recognition algorithm, for example, variations in color of a portion of the captured image information corresponding to a user's face can be monitored over a period of time. The variations can be analyzed to determine whether the captured image information likely corresponds to an actual human user instead of a representation (e.g., photo) of a human user, such as where the chroma variations in at least a red channel occur with an oscillation frequency and amplitude consistent with changes due to a pulse or heartbeat.

Abstract: A user can use a finger, or other such object, to provide input to a computing device. The finger can be positioned and/or oriented at a distance from the device, and the device can determine an input that the user is attempting to provide based on a position, motion, or gesture associated with the finger. One or more cameras can capture image information, which can be analyzed to attempt to determine the location and/or orientation of the finger. If the finger is at least partially outside a field of view of the camera(s), the device can use another sensor (e.g., EMF) to determine one or more potential locations of the fingertip. The image information then can be used to determine which of the potential locations actually corresponds to the fingertip by analyzing that portion of the finger or hand represented in the image information.

Abstract: A user attempting to obtain information about an object can capture image information including a view of that object, and the image information can be used with a matching or identification process to provide information about that type of object to the user. Information about the orientation of the camera and/or device used to capture the image can be provided in order to limit an initial search space for the matching or identification process. In some embodiments, images can be selected for matching based at least in part upon having a view matching the orientation of the camera or device. In other embodiments, images of objects corresponding to the orientation can be selected. Such a process can increase the average speed and efficiency in locating matching images. If a match cannot be found in the initial space, images of other views and categories can be analyzed as well.

Abstract: A low distraction interface can be provided for an electronic device by monitoring information such as a gaze direction of a user. A device can be configured to only display information to a user when the user is looking substantially toward the device, and can be configured to present that information in a way that is of minimal distraction to others nearby. The user can control aspects of the display by looking away from the device, continuing to look at the device for a period of time, or otherwise providing input without physically interacting with the device. Such an approach can be beneficial for settings such as business meetings where a user might want to obtain information from the device but does not want to appear inconsiderate by picking up and checking information on the device.

Abstract: An electronic device can attempt to determine a gaze direction of a user, which can be used to determine an object of interest to the user. Determining the gaze direction helps to reduce the search space and reduce processing requirements for identifying the object. Image information can be captured that includes the object of interest, which then can be analyzed to recognize the type of object. Upon recognizing the object, the user can be provided with information about the object, which in some cases can depend at least in part upon a current context or location of the object. If the object is a networked device, the user can potentially be provided with input options for controlling the device.

Abstract: The amount of power and processing needed to process gesture input for a computing device can be reduced by utilizing a separate gesture sensor. The gesture sensor can have a form factor similar to that of conventional cameras to reduce costs by being able to utilize readily available low cost parts, but can have a lower resolution and adjustable virtual shutter such that fast motions can be captured and/or recognized by the device. In some devices, a subset of the pixels of the gesture sensor can be used as a motion detector, enabling the gesture sensor to run in a low power state unless there is likely gesture input to process. Further, at least some of the processing and circuitry can be included with the gesture sensor such that various functionality can be performed without accessing a central processor or system bus, thus further reducing power consumption.

Abstract: An electronic device can attempt to determine a gaze direction of a user, which can be used to determine an object of interest to the user. Determining the gaze direction helps to reduce the search space and reduce processing requirements for identifying the object. Image information can be captured that includes the object of interest, which then can be analyzed to recognize the type of object. Upon recognizing the object, the user can be provided with information about the object, which in some cases can depend at least in part upon a current context or location of the object. If the object is a networked device, the user can potentially be provided with input options for controlling the device.

Abstract: An electronic device can utilize one or more sensors and/or imaging elements to determine the relative position of at least one light source relative to the device. In various embodiments, the intensity of light can be determined in various directions around the device. By determining the relative intensity around the device, an approximate direction of one or more light sources can be determined. Utilizing the relative position of each light source, the electronic device can properly light or shade a graphical object to be rendered by the device, or otherwise process image information captured by the device.

Abstract: Approaches are described for enabling a user to utilize fingers or other features of both hands to interact with an interface on an electronic device. A user can hold an device such as a mobile phone in a first hand, and can use a finger of the other hand to navigate through an interface or other display, such as to move relative to the device to cause the interface to scroll, zoom, or otherwise adjust. The user can use a thumb or finger of the hand holding the device to select or interact with one or more user-selectable elements, objects, or portions of the interface displayed on a display element of the device, such as by touching the screen over one of the user-selectable elements for at least a determined period of time to activate or change a function associated with a gesture used to interact with the interface.

Abstract: A user can use a finger, or other such object, to provide input to a computing device. The finger does not have to contact the device, but can be positioned and/or oriented in such a way that the device can determine an input that the user is attempting to provide, such as an element or icon that the user is intended to select. One or more cameras can capture image information, which can be analyzed to attempt to determine the location and/or orientation of the finger. If the finger is at least partially outside a field of view of the camera(s), the device can use a sensor (e.g., EMF) to attempt to determine a location of at least a portion of the finger, which can be used with the image information to determine the location and/or orientation of the finger. Other estimation processes can be used as well.

Abstract: An electronic device can attempt to determine a gaze direction of a user, which can be used to determine an object of interest to the user. Determining the gaze direction helps to reduce the search space and reduce processing requirements for identifying the object. Image information can be captured that includes the object of interest, which then can be analyzed to recognize the type of object. Upon recognizing the object, the user can be provided with information about the object, which in some cases can depend at least in part upon a current context or location of the object. If the object is a networked device, the user can potentially be provided with input options for controlling the device.

Abstract: Content displayed on a display element of a computing device can be rendered from a viewpoint that updates with changes in the relative position of a user's head, or other such feature, with respect to the device. At least one camera of the device can attempt to track the relative position in order to determine the proper viewpoint. When the user's head can no longer be seen, or when confidence in the determined relative position drops below an acceptable value, for example, the device can utilize information from another source, such as a motion or rotation sensor, to attempt to estimate the current position of the user's head. Physical limitations of the user with respect to the device also can be used to improve the accuracy of the determination.

Abstract: Approaches are described for enabling a user to utilize fingers or other features of both hands to interact with an interface on an electronic device. A user can hold an device such as a mobile phone in a first hand, and can use a finger of the other hand to navigate through an interface or other display, such as to move relative to the device to cause the interface to scroll, zoom, or otherwise adjust. The user can use a thumb or finger of the hand holding the device to select or interact with one or more user-selectable elements, objects, or portions of the interface displayed on a display element of the device, such as by touching the screen over one of the user-selectable elements for at least a determined period of time to activate or change a function associated with a gesture used to interact with the interface.

Abstract: A computing device can capture video data of at least a portion of a mouth area (e.g., mouth, lips, tongue, chin, jaw) of a user of the device. The computing device can also capture sound data including a voice of the user as well as noise (e.g. background noise). The video data can be processed to detect a movement of the portion of the mouth area. The movement of the portion of the mouth area can be analyzed and compared with mouth area movement models characteristic of oral communication (e.g., speech, song). If the movement of the portion of the mouth area corresponds to at least one model characteristic of oral communication, then the movement indicates that the user is likely engaging in oral communication. Noise reduction can be applied and/or increased on the captured sound data to reduce noise and in turn enhance the user's voice.

Abstract: The use of resources on a computing device can be optimized for current conditions to reduce or minimize the amount of resources needed to provide a sufficient level of performance for various types of tasks. In some embodiments, one or more optimization algorithms can analyze information such as a type of task to be performed and the state of one or more environmental conditions to attempt to select a number and configuration of components, such as cameras and illumination elements, to use in performing the tasks. The performance of the tasks can be monitored, and the selection or states updated in order to maintain a sufficient level of performance without using more resources than are necessary.

Abstract: A user can use a finger, or other such object, to provide input to a computing device. The finger does not have to contact the device, but can be positioned and/or oriented in such a way that the device can determine an input that the user is attempting to provide, such as an element or icon that the user is intended to select. One or more cameras can capture image information, which can be analyzed to attempt to determine the location and/or orientation of the finger. If the finger is at least partially outside a field of view of the camera(s), the device can use a sensor (e.g., EMF) to attempt to determine a location of at least a portion of the finger, which can be used with the image information to determine the location and/or orientation of the finger. Other estimation processes can be used as well.

Abstract: The amount of power and processing needed to process gesture input for a computing device can be reduced by utilizing a separate gesture sensor. The gesture sensor can have a form factor similar to that of conventional camera elements, in order to reduce costs by being able to utilize readily available low cost parts, but can have a lower resolution and adjustable virtual shutter such that fast motions can be captured and/or recognized by the device. In some devices, a subset of the pixels of the gesture sensor can be used as a motion detector, enabling the gesture sensor to run in a low power state unless there is likely gesture input to process. Further, at least some of the processing and circuitry can be included with the gesture sensor such that various functionality can be performed without accessing a central processor or system bus, thus further reducing power consumption.