Abstract: Systems and methods for executing queries on tensor datasets are disclosed. A system can identify a query for a multi-dimensional sample dataset. Each sample of the multi-dimensional sample dataset can include one or more tensors. Each tensor of the one or more tensors can be associated with a respective identifier that is common to each sample of the multi-dimensional sample dataset. The query specifying a first identifier of a first tensor of the multi-dimensional sample dataset and a first range of a first dimension of the first tensor, or one or more operations such as sampling, grouping, ungrouping, or transformation operations, to perform on the first tensor of the multi-dimensional sample dataset. The system can parse the query, and execute the query to generate query results. The system can provide the query results as output.

Abstract: Provided is a method and system for storing different types of datasets in a unified storage format such as in a tensorial form, and streaming to machine learning frameworks, as if the data is local to the machine. Data elements of large-scale datasets are transformed into a tensorial representation for each data type. The data elements have an arbitrary shape and length and a set of data elements ordered in a multi-dimensional space is treated as a dynamic tensor. Multiple transformation functions are concatenated together as a dependency directed acyclic graph to transform the plurality of large-scale datasets from one form into another. These transformation functions are user-defined, serverless lambda functions. The datasets thus transformed are compressed and stored by selecting a suitable compression and storage strategy or a compression kernel using machine learning, that is personalized for each large-scale dataset.