Abstract: Aspects of the disclosure are directed to a multimodal processing system for processing both structured and un-structured data. Real-world data is not always consistent in form or content. The multimodal processing system includes model that can be trained to account for this characteristic of real-world data, by selectively masking data of different modalities during pretraining to learn outputs that are the same or comparable between the masked and un-masked inputs. The model is trained according to modality-specific masking objectives computed for each modality of data and joint modality similarity-based masking objectives for a joint representation of the data across all modalities. The system provides consistent and accurate input, even when input data may have substantial portions of data from different modalities missing. Cross-modal relationships in data are reinforced by the model as different portions of data are masked, contributing to an overall increase in model accuracy versus other approaches.

Abstract: Aspects of the disclosure provide a deep sequence model, referred to as Koopman Neural Forecaster (KNF), for time series forecasting. KNF leverages deep neural networks (DNNs) to learn the linear Koopman space and the coefficients of chosen measurement functions. KNF imposes appropriate inductive biases for improved robustness against distributional shifts, employing both a global operator to learn shared characteristics, and a local operator to capture changing dynamics, as well as a specially-designed feedback loop to continuously update the learnt operators over time for rapidly varying behaviors. KNF achieves superior performance on multiple time series datasets that are shown to suffer from distribution shifts.

Abstract: Aspects of the disclosure are directed to a canonical approach for feature selection referred to as sparse learnable masks (SLM). SLM integrates learnable sparse masks into end-to-end training. For the fundamental non-differentiability challenge of selecting a desired number of features, SLM includes dual mechanisms for automatic mask scaling by achieving a desired feature sparsity and gradually tempering this sparsity for effective learning. SLM further employs an objective that increases mutual information (MI) between selected features and labels in an efficient and scalable manner. Empirically, SLM can achieve or improve upon state-of-the-art results on several benchmark datasets, often by a significant margin, while reducing computational complexity and cost.