Abstract: Multi-modal data autoencoding is provided. The method comprises receiving a multimodal dataset comprising number of different modalities of data related to a physical phenomenon common to the different modalities of data and encoding each of the different modalities of data into an individual latent representation. The individual latent representations are combined into a single Gaussian mixture distribution in a shared latent space. A number of parallel decoders and physics simulators decode the Gaussian mixture, wherein the decoders and physics simulators respectively reconstruct the multimodal dataset. When a unimodal dataset comprising a single modality of data related to the physical phenomenon is received a value of the physical phenomenon is predicted according to cross-modal inference learning from encoding and decoding of the multimodal dataset.