Abstract: Systems, computer program products, and methods are described herein for shard-based data indexing and retrieval in multi-key environments. The present disclosure is configured to streamline data management by utilizing a shard directory to assign unique shard key hashes to database records. This configuration allows for the efficient creation, addition, updating, and deletion of keys with minimal computational overhead. By leveraging a domain, key type, and key value, the system computes a compact shard key hash, facilitating targeted data retrieval within a specific shard. This eliminates the need to search across all shards, thereby enhancing performance. The system's instructions, when executed by a processing device, ensure that these operations preserve data integrity and optimize database interactions, leading to a more responsive and scalable data infrastructure.

Abstract: This disclosure describes techniques for performing batch processing using a custom batch processing routine (e.g., a custom filtering routine) that calls a native batch processing routine (e.g., a native filtering routine) associated with a batch processing system as well as a rule database for one or more additional filtering rules. In some cases, a custom filtering routine may be configured to: (i) receive a batch of data, (ii) call a native filtering routine to process the received batch data, and (iii) apply one or more additional filtering operations on the output of the native filtering routine. For example, the custom filtering routine may be configured to: (i) make a query to the rule database to extract one or more filtering rules, and (ii) apply the extracted filtering rules to the output of the native filtering routine to perform the additional filtering operations.

Abstract: Examples may be related to entity-aware multi-task machine learning. An example may involve: obtaining a search query; retrieving, using an entity retrieval model, at least one entity based on the search query; generating, using a neural network, query embedding data based on the search query and the at least one entity; generating, using a plurality of task-specific networks, task prediction data for a plurality of tasks based on the query embedding data, wherein each task of the plurality of tasks captures a different aspect of a user intent associated with the search query; and generating at least one search result for the search query based on the task prediction data for the plurality of tasks.

Abstract: A computer-implemented method for retrieving named entities using semantic similarity is disclosed. A query in natural language is received from a user to identify one or more named entities associated with domain-specific data of a domain. A transformer-based deep neural network decomposes the query into one or more semantic components expressed in natural language. Each component is converted into a query embedding and compared against a semantic index storing embeddings of named entities previously analyzed within the domain. The semantic index represents extracted semantic understandings of entities based on their characteristics from structured and unstructured domain data. Semantic embeddings most similar to the query embeddings are identified, and corresponding named entities are retrieved. A response including ranked matching entities is generated and returned to the user.

Abstract: Methods, systems, and devices for automating compliance control mapping are described. In some implementations, the system may ingest compliance artifacts, including text and images extracted from uploaded files. The ingested compliance artifacts may be analyzed by generating vector embeddings for the extracted text and images. These vector embeddings may be mapped to compliance controls based on predefined criteria, with the compliance controls stored in a knowledge graph database. The mappings between the vector embeddings and the compliance controls may be stored in the knowledge graph database, facilitating efficient retrieval and association of compliance information. Some implementations may enhance the accuracy and efficiency of compliance documentation by leveraging advanced computational techniques and structured data storage.

Abstract: According to examples, a cloud platform implementing a tiered storage that decouples Request Units from storage is disclosed. Data received from partitions that are characterized by corresponding partition keys is stored to the local storage medium of a virtual machine (VM) as hot data. The received data is formatted as immutable segments and stored to a disaggregated storage medium shared by a plurality of VMs. The immutable segments are grouped into sorted runs within the disaggregated storage medium and are further assigned to different levels. The hot data is deleted from a local storage medium when time-to-cold (TTC) criteria are met. The cold data can be accessed for document retrieval via a search key that includes a partition key and a documentID. Additional Request Units can be allocated or de-allocated for a portion of the cold data without moving the cold data.

Abstract: A computer-implemented method for partition table data writing includes obtaining a first data row to be written into a first partition table and a current partitioning rule indicating at least that the first partition table includes a partitioning key that is a basis for dividing data rows to be written into logical partitions and a first quantity of partitions. A target logical partition corresponding to the first data row is determined. A target physical partition is determined, where the target physical partition has a globally unique partition identifier. The globally unique partition identifier of the target physical partition and an auto-increment identifier of the first data row to the first data row are added as a composite primary key value. The first data row with the composite primary key value is stored to the first partition table.

Abstract: Methods, apparatuses, and computer-program products are disclosed. The method may include storing, on a local machine, a first vector database and a second vector database that include first and second vector embeddings of first and second context information. The method may include receiving a query requesting generation of code. The method may include comparing a third vector embedding that is based on the query with the plurality of first vector embeddings and the plurality of second vector embeddings to identify one or more first vector embeddings and one or more second vector embeddings. The method may include generating a prompt that may include the query, first information and second information corresponding with the one or more first and second vector embeddings. The method may include generating, with a generative artificial intelligence (AI) model, a response to the query based on the query, the first information, and the second information.

Abstract: An improved system and an improved method are provided for organizing and managing a dataset using an ordinal system based on an optimized indexing tree, such as a Stern-Brocot tree. The ordinal system employs unique fractional numbers in the Stern-Brocot tree as indices, which are assigned to data entries of the dataset to locate and access these data entries quickly. Thus, the present disclosure provides an improved data management strategy with improved space complexity, improved comparison operation performance, simplified implementation, and reduced storage growth size/rate, compared to its predecessors.

Abstract: Techniques are disclosed relating to a database system. The database system includes multiple coordinator nodes storing replicas of a partition. Each partition describes the state of locks and transactions for keys covered by that partition of keys. Each partition is, in turn, replicated. The multiple coordinator nodes receive, from multiple worker nodes, requests to grant a lock for a key to permit a worker node to write a record for the key as part of executing a transaction. A given coordinator node of the multiple coordinator nodes sends an approval response for the lock to at most one of the worker nodes. A single worker node acquires the lock in response to receiving approval responses from a majority of the multiple coordinator nodes, and none of the multiple worker nodes acquire the lock in response to none of them receiving approval responses from a majority of the multiple coordinator nodes.

Abstract: Techniques are provided for managing LSM tree structures and enabling bulk insertion of data into LSM tree structures. An exemplary embodiment includes a method that is performed by a data management system. The data management system receives a plurality of data records generated at least in part by at least one background process, and generates a burst segment which comprises at least a portion of the plurality of data records. The data management system inserts the burst segment into a non-root level of a LSM tree structure.

Abstract: A system and method are provided that store electronic data describing events that have occurred in a computing system, index the electronic data to create indexed data records; and store the indexed data records in computer memory as part of a flat data structure.

Abstract: A computer-implemented method is presented for storing and querying records from a data segment. The objective of the disclosure is to find computer-implemented methods for storing and querying records from a data segment that are fast and space efficient storing and querying of APM data stored in a segmented database.

Abstract: Embodiments are directed to operating a cloud based product configurator. Embodiments store, as vectorized data in a vector database product, information corresponding to a first product to be configured. While configuring the first product, embodiments receive a query regarding the first product. Embodiments augment the query in response to a context based semantic search of the vector database using the query. Embodiments prompt a large language model (“LLM”) using the augmented query and receiving an LLM response. Embodiments the provide the LLM response in response to the query.

Abstract: A query coordinator can receive a query. The query coordinator can determine one or more data semantics of a first data processing system. The data semantics of the first data processing system may be based on execution of one or more queries by the first data processing system. The query coordinator can define a query processing scheme for obtaining and processing data based on the query. The query processing scheme may include instructions for a second data processing system to execute at least a portion of the query according to the data semantics of the first data processing system. The query coordinator can provide the query processing scheme to the second data processing system and obtain query results from the second data processing system.

Abstract: Disclosed herein are various embodiments of a location data processing system. An embodiment operates by configuring a column of a table to store data across a plurality of different coordinate systems. The data to be stored in the configured column is received. The received data is divided into a plurality of fragments, including a first fragment comprising a plurality of data entries. A first data entry in the first fragment includes a coordinate specification including metadata indicating how to evaluate corresponding data of a first coordinate system represented by the first data entry. A query for data from the first fragment is received. The plurality of data entries of the first fragment are evaluated based on the coordinate specification to identify data that satisfies the query. The data is returned responsive to the query.

Abstract: Provided are methods and systems comprising receiving a plurality of data tables, wherein each data table comprises at least one row, generating a first handle for a first value in a table of the plurality of tables, wherein the first handle comprises an indication of whether a row in each of the plurality of tables is associated or not associated with the first value, and performing one or more operations on the first handle.

Abstract: Methods and systems for key-value shard creation and management in a key-value store (319). A method includes the steps of partitioning the key-value store (319) including a plurality of keyspaces (364A, 364B) into a plurality of shards including a coordinator shard (362), receiving a first shard assignment value for a first key-value pair from a key-value store client, the key-value store client having access to the key-value store, creating a first shard (321A) using the first shard assignment value, the first shard including a data structure, assigning the first key-value pair to the first shard (321a) based on the first shard assignment value, updating the coordinator shard (362) to reference to the first shard (321A), writing the first shard (321A) to a storage device, and storing modifications to the first shard (321A) in a modified version of the first shard (321A) while the first shard (321A) remains unmodified.

Abstract: The current document is directed to methods and systems that efficiently transmit and process log/event messages within and among distributed computer facilities. By digesting and condensing log/event messages at the message-collector level, the volume of data transmitted from message collectors to message-ingestion-and-processing systems is greatly reduced, which increases system efficiencies by decreasing network overheads and which provides sufficient additional computational bandwidth at the message-collector level to allow message collectors to offload many message-processing tasks from message-ingestion-and-processing system and other downstream processing systems.

Abstract: A method for splitting a node of a metadata tree, the method includes (i) splitting the node of the metadata tree based on a lock contention parameter that differs from a fullness of the node, to provide split nodes; and (ii) preventing a size based merge of the split nodes for a time period following a period of a defined duration that starts after the splitting.