Abstract: Systems and methods for image understanding can include one or more object recognition systems and one or more vision language models to generate an augmented language output that can be both scene-aware and object-aware. The systems and methods can process an input image with an object recognition model to generate an object recognition output descriptive of identification details for an object depicted in the input image. The systems and methods can include processing the input image with a vision language model to generate a language output descriptive of a predicted scene description. The object recognition output can then be utilized to augment the language output to generate an augmented language output that includes the scene understanding of the language output with the specificity of the object recognition output.

Abstract: The present disclosure relates generally to the processing of machine-readable visual encodings in view of contextual information. One embodiment of aspects of the present disclosure comprises obtaining image data descriptive of a scene that includes a machine-readable visual encoding; processing the image data with a first recognition system configured to recognize the machine-readable visual encoding; processing the image data with a second, different recognition system configured to recognize a surrounding portion of the scene that surrounds the machine-readable visual encoding; identifying a stored reference associated with the machine-readable visual encoding based at least in part on one or more first outputs generated by the first recognition system based on the image data and based at least in part on one or more second outputs generated by the second recognition system based on the image data; and performing one or more actions responsive to identification of the stored reference.

Abstract: The technology provides for a system for determining wireless charging alignment. In this regard, one or more processors may receive motion data from one or more sensors of a computing device indicating a motion of the computing device. The one or more processors may also receive charging data related to a state of an energy storage of the computing device or a state of energy transfer between a wireless charger and the computing device. Based on the motion data and the charging data, a reference vector associated with at least two charging rates may be determined. An alignment vector between the computing device and the wireless charger may then be determined based on the reference vector and the associated charging rates. Based on the alignment vector, an output may be generated guiding movement of the computing device to align with the wireless charger.

Abstract: Implementations relate to proactively aggregating client device data to append to client assistant data that is communicated to a server device in response to a user request to a client automated assistant. When a user request that is associated with, for example, a peripheral client device, is received at a client device, the client device can communicate, to a server device, data that embodies the user request (e.g., audio data and/or local speech recognition data), along with peripheral device data that was received before the client device received the user request. In this way, the client automated assistant can bypass expressly soliciting peripheral device data each time a user request is received at another client device. Instead, a peripheral device can proactively communicate device data to a client device so that the device data can be appended to request data communicated to the server device from a particular client device.

Abstract: This application is directed to presenting a unified user interface on a network-connected television device. The unified user interface displays media content recommendations selected and organized based on knowledge of a user (e.g., search queries, search results, watch history, purchase history, physical activities). The unified user interface also includes a focus area for displaying a series of media content items (e.g., an advertisement) sequentially according to a temporal order for the purposes of promoting a media content item, product, event or service. In some implementations, playback of a media content item presented on the unified user interface relies on a collaborative implementation of a corresponding media player application and a media content casting application, particularly when the play involves a restricted mode. The content casting application is enabled to play the media content item at the restrict mode when the media player application does not operate in the restricted mode.

Abstract: A system and method are provided for sizing and fitting a head mounted wearable computing device for a user based on image data of the head of the user, including a known reference device having a known scale. The system and method may include capturing image data including a face of the user to be fitted for the head mounted wearable computing device. The known reference device having the known scale is compared to features detected in the image data to determine a scaling factor. The scaling factor is used to size, or assign measures to facial features detected in the image data. A three-dimensional model of the head of the user may be generated from the captured image data.

Abstract: Implementations are described herein for leveraging digital media files retrieved and/or created by users to predict/determine topics of potential relevance to the users. In various implementations, digital media file(s) created and/or retrieved by a user with a client device may be applied as input across trained machine learning model(s), which in some cases are local to the client device, to generate output that indicates object(s) detected in the digital media file(s). Data indicative of the indicated object(s) may be provided to a remote computing system without providing the digital media file(s) themselves. In some implementations, information associated with the indicated object(s) may be retrieved and proactively output to the user. In some implementations, a frequency at which objects occur across a corpus of digital media files may be considered when determining a likelihood that a detected object is potentially relevant to a user.

Abstract: Implementations are described herein for analyzing existing interactive web sites to facilitate automatic engagement with those web sites, e.g., by automated assistants or via other user interfaces, with minimal effort from the hosts of those websites. For example, in various implementations, techniques described herein may be used to extract, validate, maintain, generalize, extend and/or distribute individual actions and “traces” of actions that are useable to navigate through various interactive websites. Additionally, techniques are described herein for leveraging these actions and/or traces to automate aspects of interaction with a third party website.

Abstract: Methods, apparatus, and computer readable media related to determining that no resources responsive to a query of a user at a first time satisfy one or more criteria (e.g., one or more quality criteria) and, in response to such a determination: providing for presentation to the user at a later time, content that is based on a given resource that is responsive to the query at the later time and that satisfies the criteria. The given resource that is responsive to the query at the later time may be a resource that is in addition to any resources responsive to the query at the first time or may be a refined version of a resource that was responsive to the query at the first time.

Abstract: Optical systems may include MEMS mirrors having elliptical mirror plates. A laser scanning system may include a MEMS mirror that scans an incident light beam along a single scanning axis. The MEMS mirror may include an elliptical mirror plate having a semi-major axis that is aligned parallel or perpendicular to the rotational axis of the elliptical mirror plate. The incident light beam may have an elliptical cross-section, such that the incident light beam completely or substantially overlaps the reflecting surface of the elliptical mirror plate. After being reflected by the elliptical mirror plate, the light beam may be circularized via one or more shaping lenses disposed in the optical path of the reflected light beam, prior to projection of the light beam.

Abstract: Implementations relate to using deep reinforcement learning to train a model that can be utilized, at each of a plurality of time steps, to determine a corresponding robotic action for completing a robotic task. Implementations additionally or alternatively relate to utilization of such a model in controlling a robot. The robotic action determined at a given time step utilizing such a model can be based on: current sensor data associated with the robot for the given time step, and free-form natural language input provided by a user. The free-form natural language input can direct the robot to accomplish a particular task, optionally with reference to one or more intermediary steps for accomplishing the particular task. For example, the free-form natural language input can direct the robot to navigate to a particular landmark, with reference to one or more intermediary landmarks to be encountered in navigating to the particular landmark.

Abstract: Implementations set forth herein relate to an automated assistant that can be invoked while a user is interfacing with a foreground application in order to retrieve data from one or more different applications, and then provide the retrieved data to the foreground application. A user can invoke the automated assistant while operating the foreground application by providing a spoken utterance, and the automated assistant can select one or more other applications to query based on content of the spoken utterance. Application data collected by the automated assistant from the one or more other applications can then be used to provide an input to the foreground application. In this way, the user can bypass switching between applications in the foreground in order to retrieve data that has been generated by other applications.

Abstract: Systems, methods and computer program products are provided for managing contactless transactions. A first tap is performed when a system is placed within a predetermined proximity to a payment terminal. A first select command including an AID corresponding to a first application is received from the payment terminal. A first response based on the first select command is transmitted to the payment terminal. A data request including information indicating supported data types is received from the payment terminal. A second response based on the data request and including transaction data is transmitted to the payment terminal. The transaction data includes at least a portion of commerce data stored in the at least one memory.

Abstract: Systems and methods of the present disclosure are directed to a method for machine-learned code segment prediction for optimizing software development. The method includes obtaining an incomplete segment of code. The method includes processing the incomplete segment of code with a machine-learned code prediction model to obtain a sampled set of segment completion predictions that include code that completes the incomplete segment of code. The method includes determining an aggregated segment completion prediction from the sampled set of segment completion predictions. The method includes replacing a portion of the aggregated segment completion prediction with an input field, wherein the portion of the aggregated segment completion prediction is associated with a degree of certainty less than a threshold degree of certainty.

Abstract: Systems and methods for providing audio data, from an initially invoked automated assistant to a subsequently invoked automated assistant. An initially invoked automated assistant may be invoked by a user utterance, followed by audio data that includes a query. The query is provided to a secondary automated assistant for processing. Subsequently, the user can submit a query that is related to the first query. In response, the initially invoked automated assistant provides the query to the secondary automated assistant in lieu of providing the query to other secondary automated assistants based on similarity between the first query and the subsequent query.

Abstract: A computer-implemented method for generating results for a client-requested query involves receiving a query produced by a client communication device, generating a result for the query in response to reception of the query, determining one or more predictive follow-up requests before receiving an actual follow-up request from the client device, and initiating retrieval of information associated with the one or more predictive follow-up requests, and transmitting at least part of the result to the client device, and then transmitting to the client device at least part of the information associated with the one or more predictive follow-up requests.

Abstract: Systems and methods of predicting documentation associated with an encounter between attendees are provided. For instance, attendee data indicative of one or more previous visit notes associated with a first attendee can be obtained. The attendee data can be inputted into a machine-learned note prediction model that includes a neural network. The neural network can generate one or more context vectors descriptive of the attendee data. Data indicative of a predicted visit note can be received as output of the machine-learned note prediction model based at least in part on the context vectors. The predicted visit note can include a set of predicted information expected to be included in a subsequently generated visit note associated with the first attendee.

Abstract: In one example, a method includes receiving, by a computing system, context information associated with a computing device; inferring, by the computing system and based on the context information, an action of a user of the computing device, the action associated with at least one entity; determining, by the computing system and based on stored attribute information associated with the at least one entity, and based on a stored set of rules associated with the inferred action, that the inferred action is not advisable; and responsive to determining that the inferred action is not advisable, outputting, by the computing system and for display on the computing device, notification data indicating that the inferred action is not advisable.