Abstract: In a method for speech generation, text data and speech data corresponding to the text data are received, the speech data comprising a speech segment and a speech unit that follows the speech segment, the speech segment comprising at least one speech unit. A causal autoregressive module in a machine learning model determines a latent feature associated with the text data and the speech segment. A diffusion module in the machine learning model determines a prediction of the speech unit based on the latent feature. The machine learning model is updated based on the speech unit and the prediction of the speech unit. With these implementations, the latent feature may comprise various aspects of both of the text data and the speech data, and then the latent feature may work as the condition feature to control the diffusion module to output the speech data related to the next patch.

Abstract: Embodiments of the disclosure relate to speech synthesis. A method provided herein includes: constructing, based on target text and prompt speech content, an input sequence corresponding to a sequence template, wherein the sequence template includes a placeholder, and a sequence segment, in the input sequence, corresponding to the placeholder is: preset content independent of the prompt speech content, or a speech feature representation generated based on the prompt speech content; and processing the input sequence with a target model to generate target speech content corresponding to the target text, wherein the target model is trained with a set of training sequences constructed based on the sequence template, the set of training sequences corresponds to a set of training speech content, and the set of training sequences is constructed by replacing the placeholder with the preset content or a training speech feature representation corresponding to respective training speech content.

Abstract: Embodiments in the disclosure relate to joint training. A method provided herein includes: obtaining a first sequence and a second sequence, wherein the first sequence is generated based on text content and the second sequence is generated based on speech content matching the text content, wherein the first sequence includes a plurality of text tokens and the second sequence includes a plurality of speech tokens; constructing a mixed sequence based on an alignment relationship between the plurality of text tokens and the plurality of speech tokens, the mixed sequence including at least one of the plurality of text tokens and at least one of the plurality of speech tokens; and training a target model with the mixed sequence.