Abstract: This disclosure provides methods, devices, and systems for generating vector embeddings. The present implementations more specifically relate to detecting changes in a data asset for targeted embeddings generation. For example, a data processing pipeline may receive a data asset to be converted to a set of vector embeddings. In some aspects, the data processing pipeline may map the data asset to one or more hash values and create a user table for the data asset based at least in part on the one or more hash values, where the user table includes one or more pointers that point to one or more records stored in a vector repository, respectively, where each record of the one or more records includes a vector embedding and a hash value associated therewith that matches a respective hash value of the one or more hash values.

Abstract: This disclosure provides methods, devices, and systems for generating vector embeddings. The present implementations more specifically relate to detecting changes in a data asset for targeted embeddings generation. For example, a data processing pipeline may receive a data asset to be converted to a set of vector embeddings. In some aspects, the data processing pipeline may map the data asset to one or more hash values and compare the hash values to a lookup table. The lookup table stores known hash values associated with previously generated vector embeddings stored in a vector repository. The data processing pipeline selectively maps the data asset to one or more vector embeddings based on whether the hash values match any of the known hash values in the lookup table. More specifically, the data processing pipeline may refrain from generating any new vector embeddings if each of the hash values matches a known hash value.

Abstract: This disclosure provides methods, devices, and systems for data management. The present implementations more specifically relate to a data orchestration system that can dynamically or programmatically produce a data processing pipeline. In some aspects, the data orchestration system of the present implementations may infer or otherwise determine the steps to be included in a data processing pipeline with little or no input from a user. In some implementations, the data orchestration system may select the steps based, at least in part, on a set of rules and policies defined by a user. The data orchestration system may further enable the user to modify the recommended data flows in the preconfigured data processing pipeline. In some aspects, the data orchestration system may aggregate data regarding usage, flow, and/or steps across multiple users to detect usage patterns, define repositories, and/or recommend data flows (such as by training a machine learning model).

Abstract: This disclosure provides methods, devices, and systems for data management. The present implementations more specifically relate to a data orchestration system that can dynamically reconfigure a data processing pipeline based on telemetry received from various steps or data operations in the pipeline. For example, the telemetry may indicate a success, failure, time of entry, time of exit, or total duration of a given step or data flow in the processing pipeline. In some aspects, the data orchestration system may dynamically invoke new data flows based on the received telemetry. In some implementations, the new data flows may allocate additional memory and/or processing resources for the data processing pipeline. In some other implementations, the new data flows may deallocate memory and/or processing resources for the data processing pipeline. Still further, in some implementations, the new data flows may trigger an alert to a user or manager of the data processing pipeline.

Abstract: This disclosure provides methods, devices, and systems for metadata extraction. The present implementations more specifically relate to a universal data representation (UDR) for heterogeneous data. As used herein, the term “UDR” refers to a metadata format that can be used to represent source data from various source data repositories and/or source content types. More specifically, metadata can be extracted from various content items and stored in respective UDR documents that describe heterogenous data in a common format. In other words, UDR documents share a common schema regardless of the schema or format of the source content. For example, a UDR data structure for a text document can have the same (or substantially similar) format as a UDR data structure for a relational database. Accordingly, UDR can significantly reduce data processing complexity by reducing the number of disparate data representations that must be understood by a data processing pipeline and/or application.