Abstract: Aspects of the present disclosure relate to systems and methods for controlling a function of a computing system using gaze detection. In examples, one or more images of a user are received and gaze information may be determined from the received one or more images. Non-gaze information may be received when the gaze information is determined to satisfy a condition. Accordingly, a function may be enabled based on the received non-gaze information. In examples, the gaze information may be determined by extracting a plurality of features from the received one or more images, providing the plurality of features to a neural network, and determining, utilizing the neural network, a location at a display device at which a gaze of the user is directed.

Abstract: A technique uses a machine-trained model to generate a response based on a prompt which expresses current input information and abstract token information. The abstract token information summarizes a full dialogue history of a dialogue, and is generated by the model itself. The technique reduces the size of the prompt by incorporating the abstract summary information in lieu of the full dialogue history. A training system trains the machine-trained model by successively improving the predictive accuracy of the machine-trained model, while rewarding the machine-trained model based on an extent to which the machine-trained model compresses instances of abstract token information.

Abstract: A method of training a neural network for detecting target features in images is described. The neural network is trained using a first data set that includes labeled images, where at least some of the labeled images having subjects with labeled features, including: dividing each of the labeled images of the first data set into a respective plurality of tiles, and generating, for each of the plurality of tiles, a plurality of feature anchors that indicate target features within the corresponding tile. Target features that correspond to the plurality of feature anchors are detected in a second data set of unlabeled images. Images of the second data set having target features that were not detected are labeled. A third data set that includes the first data set and the labeled images of the second data set is generated. The neural network is trained using the third data set.

Abstract: Machine learning techniques are leveraged to provide personalized assistance on a computing device. In some configurations a timeline of a user's interactions with the computing device is generated. For example, screenshots and audio streams may be saved as entries in the timeline. Context—the state of the computing device when the entry is created, such as which documents and websites are open—is also stored. Entries in the timeline are processed by a model to generate embedding vectors. The timeline may be searched by finding the embedding vector that is closest to an embedding vector derived from a search query. The user may select a query result, causing the associated context to be restored. For example, if the query is “show me all documents related to my upcoming trip to Japan”, the query result may open documents and websites that were open when booking a flight to Japan.

Abstract: Aspects of the present disclosure relate to systems and methods for controlling a function of a computing system using gaze detection. In examples, one or more images of a user are received and gaze information may be determined from the received one or more images. Non-gaze information may be received when the gaze information is determined to satisfy a condition. Accordingly, a function may be enabled based on the received non-gaze information. In examples, the gaze information may be determined by extracting a plurality of features from the received one or more images, providing the plurality of features to a neural network, and determining, utilizing the neural network, a location at a display device at which a gaze of the user is directed.

Abstract: Machine learning techniques are leveraged to provide personalized assistance on a computing device. In some configurations a timeline of a user's interactions with the computing device is generated. For example, screenshots and audio streams may be saved as entries in the timeline. Context—the state of the computing device when the entry is created, such as which documents and websites are open—is also stored. Entries in the timeline are processed by a model to generate embedding vectors. The timeline may be searched by finding the embedding vector that is closest to an embedding vector derived from a search query. The user may select a query result, causing the associated context to be restored. For example, if the query is “show me all documents related to my upcoming trip to Japan”, the query result may open documents and websites that were open when booking a flight to Japan.

Abstract: A technique uses a machine-trained model to generate a response based on a prompt which expresses current input information and abstract token information. The abstract token information summarizes a full dialogue history of a dialogue, and is generated by the model itself. The technique reduces the size of the prompt by incorporating the abstract summary information in lieu of the full dialogue history. A training system trains the machine-trained model by successively improving the predictive accuracy of the machine-trained model, while rewarding the machine-trained model based on an extent to which the machine-trained model compresses instances of abstract token information.

Abstract: Machine learning techniques are leveraged to provide personalized assistance on a computing device. In some configurations a timeline of a user's interactions with the computing device is generated. For example, screenshots and audio streams may be saved as entries in the timeline. Context—the state of the computing device when the entry is created, such as which documents and websites are open—is also stored. Entries in the timeline are processed by a model to generate embedding vectors. The timeline may be searched by finding the embedding vector that is closest to an embedding vector derived from a search query. The user may select a query result, causing the associated context to be restored. For example, if the query is “show me all documents related to my upcoming trip to Japan”, the query result may open documents and websites that were open when booking a flight to Japan.

Abstract: Methods and systems for generating and using a semantic index are provided. In some examples, content data is received. The content data includes a plurality of subsets of content data. Each of the plurality of subsets of content data are labelled, based on a semantic context corresponding to the content data. The plurality of subsets of content data and their corresponding labels are stored. The plurality of subsets of content data are grouped, based on their labels, thereby generating one or more groups of subsets of content data. Further, a computing device is adapted to perform an action, based on the one or more groups of subsets of content data.

Abstract: Aspects of the present disclosure relate to systems and methods for controlling a function of a computing system using gaze detection. In examples, one or more images of a user are received and gaze information may be determined from the received one or more images. Non-gaze information may be received when the gaze information is determined to satisfy a condition. Accordingly, a function may be enabled based on the received non-gaze information. In examples, the gaze information may be determined by extracting a plurality of features from the received one or more images, providing the plurality of features to a neural network, and determining, utilizing the neural network, a location at a display device at which a gaze of the user is directed.

Abstract: A method of training a neural network for detecting target features in images is described. The neural network is trained using a first data set that includes labeled images, where at least some of the labeled images having subjects with labeled features, including: dividing each of the labeled images of the first data set into a respective plurality of tiles, and generating, for each of the plurality of tiles, a plurality of feature anchors that indicate target features within the corresponding tile. Target features that correspond to the plurality of feature anchors are detected in a second data set of unlabeled images. Images of the second data set having target features that were not detected are labeled. A third data set that includes the first data set and the labeled images of the second data set is generated. The neural network is trained using the third data set.

Abstract: Methods and systems for generating and using a semantic index are provided. In some examples, content data is received. The content data includes a plurality of subsets of content data. Each of the plurality of subsets of content data are labelled, based on a semantic context corresponding to the content data. The plurality of subsets of content data and their corresponding labels are stored. The plurality of subsets of content data are grouped, based on their labels, thereby generating one or more groups of subsets of content data. Further, a computing device is adapted to perform an action, based on the one or more groups of subsets of content data.

Abstract: Methods and systems for generating and using a semantic index are provided. In some examples, content data is received. The content data includes a plurality of subsets of content data. Each of the plurality of subsets of content data are labelled, based on a semantic context corresponding to the content data. The plurality of subsets of content data and their corresponding labels are stored. The plurality of subsets of content data are grouped, based on their labels, thereby generating one or more groups of subsets of content data. Further, a computing device is adapted to perform an action, based on the one or more groups of subsets of content data.

Abstract: A computational photography system is described herein including a guidance system and a detail enhancement system. The guidance system uses a first neural network that maps an original image provided by an image sensor to a guidance image, which represents a color-corrected and lighting-corrected version of the original image. A combination unit combines the original image and the guidance image to produce a combined image. A detail-enhancement system then uses a second neural network to map the combined image to a predicted image. The predicted image supplements the guidance provided by the first neural network by sharpening details in the original image. A training system is also described herein for training the first and second neural networks. The training system alternates in the data it feeds the second neural network, first using a guidance image as input to the second neural network, and then using a corresponding ground-truth image.

Abstract: A method of training a neural network for detecting target features in images is described. The neural network is trained using a first data set that includes labeled images, where at least some of the labeled images having subjects with labeled features, including: dividing each of the labeled images of the first data set into a respective plurality of tiles, and generating, for each of the plurality of tiles, a plurality of feature anchors that indicate target features within the corresponding tile. Target features that correspond to the plurality of feature anchors are detected in a second data set of unlabeled images. Images of the second data set having target features that were not detected are labeled. A third data set that includes the first data set and the labeled images of the second data set is generated. The neural network is trained using the third data set.

Abstract: Methods and systems for automatically generating training data for use in machine learning are disclosed. The methods can involve the use of environmental data derived from first and second environmental sensors for a single event. The environmental data types derived from each environmental sensor are different. The event is detected based on first environmental data derived from the first environmental sensor, and a portion of second environmental data derived from the second environmental sensor is selected to generate training data for the detected event. The resulting training data can be employed to train machine learning models.

Abstract: Aspects of the present disclosure relate to systems and methods for controlling a function of a computing system using gaze detection. In examples, one or more images of a user are received and gaze information may be determined from the received one or more images. Non-gaze information may be received when the gaze information is determined to satisfy a condition. Accordingly, a function may be enabled based on the received non-gaze information. In examples, the gaze information may be determined by extracting a plurality of features from the received one or more images, providing the plurality of features to a neural network, and determining, utilizing the neural network, a location at a display device at which a gaze of the user is directed.

Abstract: A computational photography system is described herein including a guidance system and a detail enhancement system. The guidance system uses a first neural network that maps an original image provided by an image sensor to a guidance image, which represents a color-corrected and lighting-corrected version of the original image. A combination unit combines the original image and the guidance image to produce a combined image. A detail-enhancement system then uses a second neural network to map the combined image to a predicted image. The predicted image supplements the guidance provided by the first neural network by sharpening details in the original image. A training system is also described herein for training the first and second neural networks. The training system alternates in the data it feeds the second neural network, first using a guidance image as input to the second neural network, and then using a corresponding ground-truth image.

Abstract: An optical system, comprising a multi-spectral optical element, a switchable filter, a dual bandpass filter, and a sensor. The multi-spectral optical element receives light in at least a first spectral band and a second spectral band. The dual bandpass filter filters out wavelengths of light in a transition region of the switchable filter between the first spectral band and the second spectral band. The switchable filter filters light received from the dual bandpass filter in the first spectral band in a first mode where the switchable filter transmits light in the first spectral band and in a second mode where the switchable filter does not transmit light in the first spectral band. The sensor is disposed at an image plane, and the multi-spectral optical element is configured to produce a modulation transfer function value that is a above a predetermined threshold for each of the spectral bands.

Abstract: An intelligent assistant device is configured to communicate non-verbal cues. Image data indicating presence of a human is received from one or more cameras of the device. In response, one or more components of the device are actuated to non-verbally communicate the presence of the human. Data indicating context information of the human is received from one or more of the sensors. Using at least this data one or more contexts of the human are determined, and one or more components of the device are actuated to non-verbally communicate the one or more contexts of the human.