Abstract: Systems and methods for implementing trained embedding mappings for improved retrieval augmented generation are disclosed. A system can maintain a dataset comprising a first set of embeddings corresponding to a first embeddings space and stored in association with a set of query results for the first set of embeddings. The set of query results can correspond to a second embeddings space. The system can train a transformation data structure using the first set of embeddings and the set of query results. The transformation data structure can be used to transform the first set of embeddings to the second embeddings space. The system can execute a search operation for the second embeddings space by applying the transformation data structure to a second set of embeddings corresponding to the first embeddings space.

Abstract: Provided is a method and system for storing different types of datasets in a unified storage format such as in a tensorial form, and streaming to machine learning frameworks, as if the data is local to the machine. Data elements of large-scale datasets are transformed into a tensorial representation for each data type. Multiple transformation functions are concatenated together as a dependency directed acyclic graph to transform the plurality of large-scale datasets from one form into another. The datasets thus transformed are stored.

Abstract: A method can include identifying, by one or more processors, filtering criteria for displaying a multi-dimensional sample dataset within a display region. The method can include selecting, by the one or more processors, based on the filtering criteria, a subset of samples from the multi-dimensional sample dataset, each sample of the subset of samples associated with a respective set of tensors that are to be displayed within the display region. The method can include mapping, by the one or more processors, display data associated with the respective set of tensors of each sample of the subset of samples to respective display locations for display in the display region. The method can include presenting, by the one or more processors, the display data associated with the respective set of tensors in the display region according to the mapping.

Abstract: Systems and methods for executing queries on tensor datasets are disclosed. A system can identify a query for a multi-dimensional sample dataset. Each sample of the multi-dimensional sample dataset can include one or more tensors. Each tensor of the one or more tensors can be associated with a respective identifier that is common to each sample of the multi-dimensional sample dataset. The query specifying a first identifier of a first tensor of the multi-dimensional sample dataset and a first range of a first dimension of the first tensor, or one or more operations such as sampling, grouping, ungrouping, or transformation operations, to perform on the first tensor of the multi-dimensional sample dataset. The system can parse the query, and execute the query to generate query results. The system can provide the query results as output.

Abstract: Systems and methods for implementing tensor query-based vector search operations for multi-dimensional sample datasets of tensors are disclosed. The solution can utilize one or more processors coupled to memory to identify a query for a multi-dimensional sample dataset. The query can indicate an operation to search embeddings in the plurality of tensors of a plurality of samples of the dataset. Each sample can have a respective tensor of the plurality of tensors comprising one or more embeddings of the respective sample. The one or more processors can execute the query to generate an output dataset comprising a subset of samples of the plurality of samples. The subset of samples can be identified based on the operation and the respective one or more embeddings of each tensor of the subset of samples. The one or more processors can provide the output dataset.

Abstract: Provided is a method and system for storing different types of datasets in a unified storage format such as in a tensorial form, and streaming to machine learning frameworks, as if the data is local to the machine. Data elements of large-scale datasets are transformed into a tensorial representation for each data type. The data elements have an arbitrary shape and length and a set of data elements ordered in a multi-dimensional space is treated as a dynamic tensor. Multiple transformation functions are concatenated together as a dependency directed acyclic graph to transform the plurality of large-scale datasets from one form into another. These transformation functions are user-defined, serverless lambda functions. The datasets thus transformed are compressed and stored by selecting a suitable compression and storage strategy or a compression kernel using machine learning, that is personalized for each large-scale dataset.

Abstract: Systems and methods for executing queries on tensor datasets are disclosed. A system can identify a query for a multi-dimensional sample dataset. Each sample of the multi-dimensional sample dataset can include one or more tensors. Each tensor of the one or more tensors can be associated with a respective identifier that is common to each sample of the multi-dimensional sample dataset. The query specifying a first identifier of a first tensor of the multi-dimensional sample dataset and a first range of a first dimension of the first tensor, or one or more operations such as sampling, grouping, ungrouping, or transformation operations, to perform on the first tensor of the multi-dimensional sample dataset. The system can parse the query, and execute the query to generate query results. The system can provide the query results as output.

Abstract: Provided is a method and system for storing different types of datasets in a unified storage format such as in a tensorial form, and streaming to machine learning frameworks, as if the data is local to the machine. Data elements of large-scale datasets are transformed into a tensorial representation for each data type. The data elements have an arbitrary shape and length and a set of data elements ordered in a multi-dimensional space is treated as a dynamic tensor. Multiple transformation functions are concatenated together as a dependency directed acyclic graph to transform the plurality of large-scale datasets from one form into another. These transformation functions are user-defined, serverless lambda functions. The datasets thus transformed are compressed and stored by selecting a suitable compression and storage strategy or a compression kernel using machine learning, that is personalized for each large-scale dataset.