Abstract: Provided are systems and methods for general text-driven image editing, example implementations of which may be referred to as “UniTune”. UniTune can receive as input an arbitrary image and a textual edit description, and can carry out the edit while maintaining high semantic and visual fidelity to the input image. UniTune does not require any additional inputs, like masks or sketches. According to an aspect of the present disclosure, with the right choice of parameters, example systems described herein can fine-tune a large diffusion model (e.g., Imagen) on a single image, encouraging the model to maintain fidelity to the input image, both visually and semantically, while still allowing expressive manipulations.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating an output video. One of the methods include: obtaining an input video; obtaining input text that includes a description of an output video; generating, based at least on applying downsampling to the input video, a degraded version of the input video; and generating the output video based on the description in the input text by updating the degraded version of the input video by using a video diffusion model across a plurality of reverse diffusion steps.

Abstract: Implementations are directed to receiving unstructured free-form natural language input, generating a chatbot based on the unstructured free-form natural language input and in response to receiving the unstructured free-form natural language input, and causing the chatbot to perform task(s) associated with an entity and on behalf of the user. In various implementations, the unstructured free-form natural language input conveys details of the task(s) to be performed, but does not define any corresponding dialog state map (e.g., does not define any dialog states or any dialog state transitions). Nonetheless, the unstructured free-form natural language input may be utilized to fine-tune and/or prime a machine learning model that is already capable of being utilized in conducting generalized conversations. As a result, the chatbot can be generated and deployed in a quick and efficient manner for performance of the task(s) on behalf of the user.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, relating to synthetic call status updates. In some implementations, a method includes determining, by a task manager module, that a triggering event has occurred to provide a current status of a user call request. The method may then determine, by the task manager module, the current status of the user call request. A representation of the current status of the user call request is generated. Then, the generated representation of the current status of the user call request is provided to the user.

Abstract: Implementations are directed to receiving unstructured free-form natural language input, generating a chatbot based on the unstructured free-form natural language input and in response to receiving the unstructured free-form natural language input, and causing the chatbot to perform task(s) associated with an entity and on behalf of the user. In various implementations, the unstructured free-form natural language input conveys details of the task(s) to be performed, but does not define any corresponding dialog state map (e.g., does not define any dialog states or any dialog state transitions). Nonetheless, the unstructured free-form natural language input may be utilized to fine-tune and/or prime a machine learning model that is already capable of being utilized in conducting generalized conversations. As a result, the chatbot can be generated and deployed in a quick and efficient manner for performance of the task(s) on behalf of the user.

Abstract: Implementations are directed to updating a trained voice bot that is deployed for conducting conversations on behalf of a third-party. A third-party developer can interact with a voice bot development system that enables the third-party developer to train, update, validate, and monitor performance of the trained voice bot. In various implementations, the trained voice bot can be updated by updating a corpus of training instances that was initially utilized to train the voice bot, and updating the trained voice bot based on the updated corpus. In some implementations, the corpus of training instances may be updated in response to identifying occurrence(s) of behavioral error(s) of the trained voice bot while the conversations are being conducted on behalf of the third-party. In additional or alternative implementations, the corpus of training instances may be updated in response to determining the trained voice bot does not include a desired behavior.

Abstract: Processor(s) of a client device of a user can receive a telephone call that is initiated by an additional user, and, in response to receiving the telephone call, identify an entity that is associated with the additional user, and determine, based on the entity that is associated with the additional user, whether to (1) fully automate the telephone call, or (2) partially automate the telephone call. In fully automating the telephone call, the processor(s) can cause a chatbot to engage in a corresponding conversation with the additional user and without prompting the user for any input. In partially automating the telephone call, the processor(s) can cause the chatbot to engage in a corresponding conversation with the additional user but with prompting the user for input(s) via suggestion chip(s). In some implementations, the processor(s) can further determine whether to (3) refrain from automating the telephone call entirely.

Abstract: Implementations are directed to providing a voice bot development platform that enables a third-party developer to train a voice bot based on training instance(s). The training instance(s) can each include training input and training output. The training input can include a portion of a corresponding conversation and a prior context of the corresponding conversation. The training output can include a corresponding ground truth response to the portion of the corresponding conversation. Subsequent to training, the voice bot can be deployed for conducting conversations on behalf of a third-party. In some implementations, the voice bot is further trained based on a corresponding feature emphasis input that attentions the voice bot to a particular feature of the portion of the corresponding conversation. In some additional or alternative implementations, the voice bot is further trained to interact with third-party system(s) via remote procedure calls (RPCs).

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for an automated calling system are disclosed. In one aspect, a method includes the actions of receiving audio data of an utterance spoken by a user who is having a telephone conversation with a bot. The actions further include determining a context of the telephone conversation. The actions further include determining a user intent of a first previous portion of the telephone conversation spoken by the user and a bot intent of a second previous portion of the telephone conversation outputted by a speech synthesizer of the bot. The actions further include, based on the audio data of the utterance, the context of the telephone conversation, the user intent, and the bot intent, generating synthesized speech of a reply by the bot to the utterance. The actions further include, providing, for output, the synthesized speech.

Abstract: Implementations are directed to updating a trained voice bot that is deployed for conducting conversations on behalf of a third-party. A third-party developer can interact with a voice bot development system that enables the third-party developer to train, update, validate, and monitor performance of the trained voice bot. In various implementations, the trained voice bot can be updated by updating a corpus of training instances that was initially utilized to train the voice bot, and updating the trained voice bot based on the updated corpus. In some implementations, the corpus of training instances may be updated in response to identifying occurrence(s) of behavioral error(s) of the trained voice bot while the conversations are being conducted on behalf of the third-party. In additional or alternative implementations, the corpus of training instances may be updated in response to determining the trained voice bot does not include a desired behavior.

Abstract: Implementations are directed to providing a voice bot development platform that enables a third-party developer to train a voice bot based on training instance(s). The training instance(s) can each include training input and training output. The training input can include a portion of a corresponding conversation and a prior context of the corresponding conversation. The training output can include a corresponding ground truth response to the portion of the corresponding conversation. Subsequent to training, the voice bot can be deployed for conducting conversations on behalf of a third-party. In some implementations, the voice bot is further trained based on a corresponding feature emphasis input that attentions the voice bot to a particular feature of the portion of the corresponding conversation. In some additional or alternative implementations, the voice bot is further trained to interact with third-party system(s) via remote procedure calls (RPCs).

Abstract: Processor(s) of a client device of a user can receive a telephone call that is initiated by an additional user, and, in response to receiving the telephone call, identify an entity that is associated with the additional user, and determine, based on the entity that is associated with the additional user, whether to (1) fully automate the telephone call, or (2) partially automate the telephone call. In fully automating the telephone call, the processor(s) can cause a chatbot to engage in a corresponding conversation with the additional user and without prompting the user for any input. In partially automating the telephone call, the processor(s) can cause the chatbot to engage in a corresponding conversation with the additional user but with prompting the user for input(s) via suggestion chip(s). In some implementations, the processor(s) can further determine whether to (3) refrain from automating the telephone call entirely.

Abstract: Methods, systems, and apparatus for an automated calling system are disclosed. Some implementations are directed to using a bot to initiate telephone calls and conduct telephone conversations with a user. The bot may be interrupted while providing synthesized speech during the telephone call. The interruption can be classified into one of multiple disparate interruption types, and the bot can react to the interruption based on the interruption type. Some implementations are directed to determining that a first user is placed on hold by a second user during a telephone conversation, and maintaining the telephone call in an active state in response to determining the first user hung up the telephone call. The first user can be notified when the second user rejoins the call, and a bot associated with the first user can notify the first user that the second user has rejoined the telephone call.

Abstract: Implementations are directed to using an automated assistant to initiate an assisted call on behalf of a given user. The assistant can, during the assisted call, receiving a request, from an additional user on the assisted call, for information that is not known to the assistant. In response, the assistant can render a prompt for the information and, while awaiting responsive input from the given user, continue the assisted call using already resolved value(s) for the assisted call. If responsive input is received within a threshold duration of time, synthesized speech, corresponding to the responsive input, is rendered as part of the assisted call. Implementations are additionally or alternatively directed to using the automated assistant to provide, during an ongoing call between a given user and an additional user, output that is based on a value requested by the additional user during the ongoing call.

Abstract: Methods, systems, and apparatus for an automated calling system are disclosed. Some implementations are directed to using a bot to initiate telephone calls and conduct telephone conversations with a user. The bot may be interrupted while providing synthesized speech during the telephone call. The interruption can be classified into one of multiple disparate interruption types, and the bot can react to the interruption based on the interruption type. Some implementations are directed to determining that a first user is placed on hold by a second user during a telephone conversation, and maintaining the telephone call in an active state in response to determining the first user hung up the telephone call. The first user can be notified when the second user rejoins the call, and a bot associated with the first user can notify the first user that the second user has rejoined the telephone call.

Abstract: Implementations are directed to using an automated assistant to initiate an assisted call on behalf of a given user. The assistant can, during the assisted call, receiving a request, from an additional user on the assisted call, for information that is not known to the assistant. In response, the assistant can render a prompt for the information and, while awaiting responsive input from the given user, continue the assisted call using already resolved value(s) for the assisted call. If responsive input is received within a threshold duration of time, synthesized speech, corresponding to the responsive input, is rendered as part of the assisted call. Implementations are additionally or alternatively directed to using the automated assistant to provide, during an ongoing call between a given user and an additional user, output that is based on a value requested by the additional user during the ongoing call.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for an automated calling system are disclosed. In one aspect, a method includes the actions of receiving audio data of an utterance spoken by a user who is having a telephone conversation with a bot. The actions further include determining a context of the telephone conversation. The actions further include determining a user intent of a first previous portion of the telephone conversation spoken by the user and a bot intent of a second previous portion of the telephone conversation outputted by a speech synthesizer of the bot. The actions further include, based on the audio data of the utterance, the context of the telephone conversation, the user intent, and the bot intent, generating synthesized speech of a reply by the bot to the utterance. The actions further include, providing, for output, the synthesized speech.

Abstract: Processor(s) of a client device of a user can receive a telephone call that is initiated by an additional user, and, in response to receiving the telephone call, identify an entity that is associated with the additional user, and determine, based on the entity that is associated with the additional user, whether to (1) fully automate the telephone call, or (2) partially automate the telephone call. In fully automating the telephone call, the processor(s) can cause a chatbot to engage in a corresponding conversation with the additional user and without prompting the user for any input. In partially automating the telephone call, the processor(s) can cause the chatbot to engage in a corresponding conversation with the additional user but with prompting the user for input(s) via suggestion chip(s). In some implementations, the processor(s) can further determine whether to (3) refrain from automating the telephone call entirely.

Abstract: Implementations receive, via a client device, user input to initiate a telephone call with an entity, and, in response to receiving the user input to initiate the telephone call with the entity and prior to initiating the telephone call with the entity: obtain pre-call information that is stored in association with the entity, and cause the pre-call information that is stored in association with the entity to be provided for presentation to the user via the client device. The pre-call information may include any information that would be provided for presentation to a user subsequent to initiation of the telephone call with the entity. Further, implementations determine, based on user consumption of the pre-call information, whether to (1) proceed with initiating the telephone call with the entity, or (2) refrain from initiating the telephone call with the entity, and cause the client device to implement the appropriate action.

Abstract: Implementations are directed to receiving unstructured free-form natural language input, generating a chatbot based on the unstructured free-form natural language input and in response to receiving the unstructured free-form natural language input, and causing the chatbot to perform task(s) associated with an entity and on behalf of the user. In various implementations, the unstructured free-form natural language input conveys details of the task(s) to be performed, but does not define any corresponding dialog state map (e.g., does not define any dialog states or any dialog state transitions). Nonetheless, the unstructured free-form natural language input may be utilized to fine-tune and/or prime a machine learning model that is already capable of being utilized in conducting generalized conversations. As a result, the chatbot can be generated and deployed in a quick and efficient manner for performance of the task(s) on behalf of the user.