Abstract: Systems, methods, and apparatus for using a multimodal response in the dynamic generation of client device output that is tailored to a current modality of a client device is disclosed herein. Multimodal client devices can engage in a variety of interactions across the multimodal spectrum including voice only interactions, voice forward interactions, multimodal interactions, visual forward interactions, visual only interactions etc. A multimodal response can include a core message to be rendered for all interaction types as well as one or more modality dependent components to provide a user with additional information.

Abstract: Implementations described herein utilize an independent server for facilitating secure exchange of data between multiple disparate parties. The independent server receives client data, via an automated assistant application executing at least in part at a client device, that is to be transmitted to a given third-party application. The independent server processes the client data, using a first encoder-decoder model, to generate opaque client data, and transmits the opaque client data to the given third-party application and without transmitting any of the client data. Further, the independent server receives response data, via the given third-party application, that is generated based on the opaque client data and that is to be transmitted back to the client device. The independent server processes the response data, using a second encoder-decoder model, to generate opaque response data, and transmits the opaque response data to the client device and without transmitting any of the response data.

Abstract: Implementations described herein are directed to enabling collaborative ranking of interpretations of spoken utterances based on data that is available to an automated assistant and third-party agent(s), respectively. The automated assistant can determine first-party interpretation(s) of a spoken utterance provided by a user, and can cause the third-party agent(s) to determine third-party interpretation(s) of the spoken utterance provided by the user. In some implementations, the automated assistant can select a given interpretation, from the first-party interpretation(s) and the third-party interpretation(s), of the spoken utterance, and can cause a given third-party agent to satisfy the spoken utterance based on the given interpretation.

Abstract: Implementations described herein utilize an independent server for facilitating secure exchange of data between multiple disparate parties. The independent server receives client data, via an automated assistant application executing at least in part at a client device, that is to be transmitted to a given third-party application. The independent server processes the client data, using a first encoder-decoder model, to generate opaque client data, and transmits the opaque client data to the given third-party application and without transmitting any of the client data. Further, the independent server receives response data, via the given third-party application, that is generated based on the opaque client data and that is to be transmitted back to the client device. The independent server processes the response data, using a second encoder-decoder model, to generate opaque response data, and transmits the opaque response data to the client device and without transmitting any of the response data.

Abstract: A system and methodology provide for annotating videos with entities and associated probabilities of existence of the entities within video frames. A computer-implemented method identifies an entity from a plurality of entities identifying characteristics of video items. The computer-implemented method selects a set of features correlated with the entity based on a value of a feature of a plurality of features, determines a classifier for the entity using the set of features, and determines an aggregation calibration function for the entity based on the set of features. The computer-implemented method selects a video frame from a video item, where the video frame having associated features, and determines a probability of existence of the entity based on the associated features using the classifier and the aggregation calibration function.

Abstract: Implementations described herein are directed to enabling collaborative ranking of interpretations of spoken utterances based on data that is available to an automated assistant and third-party agent(s), respectively. The automated assistant can determine first-party interpretation(s) of a spoken utterance provided by a user, and can cause the third-party agent(s) to determine third-party interpretation(s) of the spoken utterance provided by the user. In some implementations, the automated assistant can select a given interpretation, from the first-party interpretation(s) and the third-party interpretation(s), of the spoken utterance, and can cause a given third-party agent to satisfy the spoken utterance based on the given interpretation.

Abstract: A computer-implemented method for selecting representative frames for videos is provided. The method includes receiving a video and identifying a set of features for each of the frames of the video. The features including frame-based features and semantic features. The semantic features identifying likelihoods of semantic concepts being present as content in the frames of the video. A set of video segments for the video is subsequently generated. Each video segment includes a chronological subset of frames from the video and each frame is associated with at least one of the semantic features. The method generates a score for each frame of the subset of frames for each video segment based at least on the semantic features, and selecting a representative frame for each video segment based on the scores of the frames in the video segment. The representative frame represents and summarizes the video segment.

Abstract: Systems, methods, and apparatus for using a multimodal response in the dynamic generation of client device output that is tailored to a current modality of a client device is disclosed herein. Multimodal client devices can engage in a variety of interactions across the multimodal spectrum including voice only interactions, voice forward interactions, multimodal interactions, visual forward interactions, visual only interactions etc. A multimodal response can include a core message to be rendered for all interaction types as well as one or more modality dependent components to provide a user with additional information.

Abstract: Implementations described herein are directed to enabling collaborative ranking of interpretations of spoken utterances based on data that is available to an automated assistant and third-party agent(s), respectively. The automated assistant can determine first-party interpretation(s) of a spoken utterance provided by a user, and can cause the third-party agent(s) to determine third-party interpretation(s) of the spoken utterance provided by the user. In some implementations, the automated assistant can select a given interpretation, from the first-party interpretation(s) and the third-party interpretation(s), of the spoken utterance, and can cause a given third-party agent to satisfy the spoken utterance based on the given interpretation.

Abstract: Systems, methods, and apparatus for using a multimodal response in the dynamic generation of client device output that is tailored to a current modality of a client device is disclosed herein. Multimodal client devices can engage in a variety of interactions across the multimodal spectrum including voice only interactions, voice forward interactions, multimodal interactions, visual forward interactions, visual only interactions etc. A multimodal response can include a core message to be rendered for all interaction types as well as one or more modality dependent components to provide a user with additional information.

Abstract: Implementations described herein utilize an independent server for facilitating secure exchange of data between multiple disparate parties. The independent server receives client data, via an automated assistant application executing at least in part at a client device, that is to be transmitted to a given third-party application. The independent server processes the client data, using a first encoder-decoder model, to generate opaque client data, and transmits the opaque client data to the given third-party application and without transmitting any of the client data. Further, the independent server receives response data, via the given third-party application, that is generated based on the opaque client data and that is to be transmitted back to the client device. The independent server processes the response data, using a second encoder-decoder model, to generate opaque response data, and transmits the opaque response data to the client device and without transmitting any of the response data.

Abstract: Implementations described herein are directed to enabling collaborative ranking of interpretations of spoken utterances based on data that is available to an automated assistant and third-party agent(s), respectively. The automated assistant can determine first-party interpretation(s) of a spoken utterance provided by a user, and can cause the third-party agent(s) to determine third-party interpretation(s) of the spoken utterance provided by the user. In some implementations, the automated assistant can select a given interpretation, from the first-party interpretation(s) and the third-party interpretation(s), of the spoken utterance, and can cause a given third-party agent to satisfy the spoken utterance based on the given interpretation.

Abstract: Implementations include identifying, from a database of entries reflecting past automated assistant commands submitted within a threshold amount of time relative to a current time, particular entries that each reflect corresponding features of a corresponding user submission of a particular command. Further, those implementations include determining that the particular command is a golden command, for a particular automated assistant function, responsive to determining that: at least a threshold percentage of the user submissions of the particular command triggered the particular automated assistant function, and a quantity of the user submission of the particular command satisfies a threshold quantity.

Abstract: A system and methodology provide for annotating videos with entities and associated probabilities of existence of the entities within video frames. A computer-implemented method identifies an entity from a plurality of entities identifying characteristics of video items. The computer-implemented method selects a set of features correlated with the entity based on a value of a feature of a plurality of features, determines a classifier for the entity using the set of features, and determines an aggregation calibration function for the entity based on the set of features. The computer-implemented method selects a video frame from a video item, where the video frame having associated features, and determines a probability of existence of the entity based on the associated features using the classifier and the aggregation calibration function.

Abstract: Implementations include identifying, from a database of entries reflecting past automated assistant commands submitted within a threshold amount of time relative to a current time, particular entries that each reflect corresponding features of a corresponding user submission of a particular command. Further, those implementations include determining that the particular command is a golden command, for a particular automated assistant function, responsive to determining that: at least a threshold percentage of the user submissions of the particular command triggered the particular automated assistant function, and a quantity of the user submission of the particular command satisfies a threshold quantity.

Abstract: Systems, methods, and apparatus for using a multimodal response in the dynamic generation of client device output that is tailored to a current modality of a client device is disclosed herein. Multimodal client devices can engage in a variety of interactions across the multimodal spectrum including voice only interactions, voice forward interactions, multimodal interactions, visual forward interactions, visual only interactions etc. A multimodal response can include a core message to be rendered for all interaction types as well as one or more modality dependent components to provide a user with additional information.

Abstract: A system and methodology provide for annotating videos with entities and associated probabilities of existence of the entities within video frames. A computer-implemented method identifies an entity from a plurality of entities identifying characteristics of video items. The computer-implemented method selects a set of features correlated with the entity based on a value of a feature of a plurality of features, determines a classifier for the entity using the set of features, and determines an aggregation calibration function for the entity based on the set of features. The computer-implemented method selects a video frame from a video item, where the video frame having associated features, and determines a probability of existence of the entity based on the associated features using the classifier and the aggregation calibration function.

Abstract: Systems, methods, and apparatus for using a multimodal response in the dynamic generation of client device output that is tailored to a current modality of a client device is disclosed herein. Multimodal client devices can engage in a variety of interactions across the multimodal spectrum including voice only interactions, voice forward interactions, multimodal interactions, visual forward interactions, visual only interactions etc. A multimodal response can include a core message to be rendered for all interaction types as well as one or more modality dependent components to provide a user with additional information.

Abstract: A computer-implemented method for selecting representative frames for videos is provided. The method includes receiving a video and identifying a set of features for each of the frames of the video. The features including frame-based features and semantic features. The semantic features identifying likelihoods of semantic concepts being present as content in the frames of the video. A set of video segments for the video is subsequently generated. Each video segment includes a chronological subset of frames from the video and each frame is associated with at least one of the semantic features. The method generates a score for each frame of the subset of frames for each video segment based at least on the semantic features, and selecting a representative frame for each video segment based on the scores of the frames in the video segment. The representative frame represents and summarizes the video segment.

Abstract: A computer-implemented method for selecting representative frames for videos is provided. The method includes receiving a video and identifying a set of features for each of the frames of the video. The features including frame-based features and semantic features. The semantic features identifying likelihoods of semantic concepts being present as content in the frames of the video. A set of video segments for the video is subsequently generated. Each video segment includes a chronological subset of frames from the video and each frame is associated with at least one of the semantic features. The method generates a score for each frame of the subset of frames for each video segment based at least on the semantic features, and selecting a representative frame for each video segment based on the scores of the frames in the video segment. The representative frame represents and summarizes the video segment.