Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for jointly training an encoder neural network and a decoder neural network. In one aspect, a method comprises, for each latent dimension in a proper subset of a plurality of latent dimensions of a latent space: processing a predefined embedding that represents the latent dimension using the decoder neural network to generate multi-modal data, having a plurality of feature dimensions, that defines a predicted multi-modal data archetype corresponding to the latent dimension; and updating the values of the set of decoder parameters using gradients of an archetype loss function that measures an error between: (i) a predicted multi-modal data archetype corresponding to the latent dimension, and (ii) a target multi-modal data archetype corresponding to the latent dimension.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a clinical recommendation for medical treatment of a patient.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for jointly training an encoder neural network and a decoder neural network. In one aspect, a method comprises: updating current values of a set of encoder parameters and current values of a set of decoder parameters using gradients of a reconstruction loss function that measures an error in a reconstruction of multi-modal data from a training example, wherein: the reconstruction loss function comprises a plurality of scaling factors that each scale a respective term in the reconstruction loss function that measures an error in the reconstruction of a corresponding proper subset of feature dimensions of the multi-modal data from the training example.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for jointly training an encoder neural network and a decoder neural network. In one aspect, a method comprises: updating current values of a set of encoder parameters and current values of a set of decoder parameters using gradients of a reconstruction loss function that measures an error in a reconstruction of multi-modal data from a training example, wherein: the reconstruction loss function comprises a plurality of scaling factors that each scale a respective term in the reconstruction loss function that measures an error in the reconstruction of a corresponding proper subset of feature dimensions of the multi-modal data from the training example.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating multi-modal data archetypes. In one aspect, a method comprises obtaining a plurality of training examples, wherein each training example corresponds to a respective patient and includes multi-modal data, having a plurality of feature dimensions, that characterizes the patient; jointly training an encoder neural network and a decoder neural network on the plurality of training examples; and generating a plurality of multi-modal data archetypes that each correspond to a respective dimension of a latent space, comprising, for each multi-modal data archetype: processing a predefined embedding that represents the corresponding dimension of the latent space using the decoder neural network to generate multi-modal data, having the plurality of feature dimensions, that defines the multi-modal data archetype.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for classifying a patient. In one aspect, a method comprises: receiving multi-modal data characterizing a patient, wherein the multi-modal data comprises a respective feature representation for each of a plurality of modalities; processing the multi-modal data characterizing the patient using an encoder neural network to generate an embedding of the multi-modal data characterizing the patient; determining a respective classification score for each patient category in a set of patient categories based on the embedding of the multi-modal data characterizing the patient; and classifying the patient as being included in a corresponding patient category from the set of patient categories based on the classification scores.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for classifying a patient. In one aspect, a method comprises: receiving multi-modal data characterizing a patient, wherein the multi-modal data comprises a respective feature representation for each of a plurality of modalities; processing the multi-modal data characterizing the patient using an encoder neural network to generate an embedding of the multi-modal data characterizing the patient; determining a respective classification score for each patient category in a set of patient categories based on the embedding of the multi-modal data characterizing the patient; and classifying the patient as being included in a corresponding patient category from the set of patient categories based on the classification scores.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for jointly training an encoder neural network and a decoder neural network. In one aspect, a method comprises: updating current values of a set of encoder parameters and current values of a set of decoder parameters using gradients of a reconstruction loss function that measures an error in a reconstruction of multi-modal data from a training example, wherein: the reconstruction loss function comprises a plurality of scaling factors that each scale a respective term in the reconstruction loss function that measures an error in the reconstruction of a corresponding proper subset of feature dimensions of the multi-modal data from the training example.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating multi-modal data archetypes. In one aspect, a method comprises obtaining a plurality of training examples, wherein each training example corresponds to a respective patient and includes multi-modal data, having a plurality of feature dimensions, that characterizes the patient; jointly training an encoder neural network and a decoder neural network on the plurality of training examples; and generating a plurality of multi-modal data archetypes that each correspond to a respective dimension of a latent space, comprising, for each multi-modal data archetype: processing a predefined embedding that represents the corresponding dimension of the latent space using the decoder neural network to generate multi-modal data, having the plurality of feature dimensions, that defines the multi-modal data archetype.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for classifying a patient. In one aspect, a method comprises: receiving multi-modal data characterizing a patient, wherein the multi-modal data comprises a respective feature representation for each of a plurality of modalities; processing the multi-modal data characterizing the patient using an encoder neural network to generate an embedding of the multi-modal data characterizing the patient; determining a respective classification score for each patient category in a set of patient categories based on the embedding of the multi-modal data characterizing the patient; and classifying the patient as being included in a corresponding patient category from the set of patient categories based on the classification scores.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a clinical recommendation for medical treatment of a patient.