Abstract: A data processing service receives a request from a user. The request specifies a task for generating embedding vectors to represent a target dataset using a target embedding model. The service applies an automatic pipeline to the target dataset, which pre-processes the target dataset to generate batches of the target dataset. Each batch includes a plurality of data chunks in the target dataset. The automatic pipeline assigns computing resources to each batch of the target dataset; and applies, by the assigned computing resources, the target embedding model to each batch to generate the embedding vectors. Each embedding vector represents a data chunk of the target dataset in a latent space and is incrementally updated based on an update to a corresponding data chunk represented by the embedding vector. The configuration stores the generated one or more embedding vectors in a vector database.

Abstract: A data processing system accesses a dataset from a data source and generates a set of embedding vectors representing the dataset in a latent space. The system splits the dataset into a set of data chunks and generates the embedding vectors. Each embedding vector represents a data chunk. The system may store the generated set of embedding vectors in a vector database that includes a plurality of embedding vectors. The system updates the embedding vectors by detecting a change to a first dataset that is represented by a first set of embedding vectors in the vector database, determining that the change to the first dataset is related to a first data chunk of the first set of data chunks included in the first dataset, updating a first embedding vector representing the first data chunk with the detected change; and storing the updated first embedding vector in the vector database.