Abstract: A data processing service facilitates the creation and processing of data processing pipelines that process data processing jobs defined with respect to a set of tasks in a sequence and with data dependencies associated with each separate task such that the output from one task is used as input for a subsequent task. In various embodiments, the set of tasks include at least one cleanroom task that is executed in a cleanroom station and at least one non-cleanroom task executed in an execution environment of a user where each task is configured to read one or more input datasets and transform the one or more input datasets into one or more output datasets.

Abstract: A system and method for defining a relational schema for a non-relational database are disclosed. In one embodiment, the system identifies a non-relational database; obtains a data structure of the non-relational database; determines a relational schema type; generates a relational schema based on the determined schema type; determines whether the relational schema includes a plurality of relational tables; and responsive to determining that the relational schema includes a plurality of relational tables, defines a relation between at least two of the relational tables in the relational schema.

Abstract: Methods and systems for managing operation of a data pipeline are disclosed. To manage the operation, a system may include one or more data sources, a data manager, and one or more downstream consumers. Interruptions to the operation may impact provision of data processing services by the data pipeline and may cause the data processing services to no longer align with operation quality goals for the data pipeline. To maintain compliance with the operation quality goals, the operation may be monitored over time. Operation data may be obtained for the data pipeline and may be used to determine representations of operation quality of the data pipeline. The representations of operation quality of the data pipeline may be compared to the operation quality goals and actions may be performed to remediate differences between the representations of operation quality of the data pipeline and the operation quality goals.

Abstract: A data processing method for an electronic device includes: obtaining data rebalance information representing a new partition plan for a partition table, the partition table being a table for data distribution based on a partition key; creating, for a set specified in the partition plan, a corresponding partition space, and recording, in the corresponding partition space, a current data range of a set corresponding to the partition space, the data range comprising one of a value range of the partition key and a value list of the partition key; updating the data range recorded in the corresponding partition space based on the partition plan, and updating correspondence between the corresponding partition space and sets of the partition table by using the updated data range in the corresponding partition space; and updating data distribution in the sets of the partition table based on the correspondence.

Abstract: Disclosed are various embodiments for data anomaly detection. A variable profile is generated for each variable in source data. Then, the variable profiles are provided to each of a plurality of machine learning models. Next, it is determined, with each of the plurality of machine learning models, whether each variable profile is anomalous. The determination, from each of the plurality of machine learning models, whether each variable profile is anomalous is provided to an ensemble model. The ensemble model then generates a final determination whether each variable profile is anomalous. The final determination is then reported to an analysis service.

Abstract: A computer-implemented method, including receiving, by a processor set, a query including a query string for a system catalog; identifying, by the processor set, a default index structure of the system catalog; executing, by the processor set, the query based on the default index structure of each index in the system catalog; ranking, by the processor set, a performance of each execution of the query by each index; mapping, by the processor set, a query pattern to a corresponding index of the system catalog; selecting, by the processor set, the index to perform the query using a machine learning (ML) model trained with a knowledge base that includes the ranking and the mapping; executing, by the processor set, the query on the selected index; and in response to executing the query on the selected index, returning, by the processor set, a result of the query.

Abstract: Systems, methods, devices and non-transitory, computer-readable storage mediums are disclosed for a wearable multimedia device and cloud computing platform with an application ecosystem for processing multimedia data captured by the wearable multimedia device. In an embodiment, a method for using a machine learning model to provide data corresponding to a query comprises receiving a query for data stored in one or more databases. A prompt file is determined for the query. The prompt file and the query are provided as input for a machine learning model configured to generate a prediction for the query. Training the machine learning model can include updating network parameters in the machine learning model based on the prompt file. The prediction generated for the query is received. The prediction comprises a predicted database query corresponding to the query, and a level of accuracy is determined for the predicted database query.

Abstract: Methods, devices and computer program products facilitate the storage, access and management of log files that are associated with particular client devices. The log files provide a record of user or client device activities that are periodically sent to a data backup center. A dedicated log file server facilitates the processing and storage of an increasingly large number of log files that are generated by new and existing client devices. A storage server pre-processes the received log files to facilitate the processing and storage of the log files by the log file server. This Abstract is provided for the sole purpose of complying with the Abstract requirement rules. This Abstract is submitted with the explicit understanding that it will not be used to interpret or to limit the scope or the meaning of the claims.

Abstract: Techniques are described herein for a graph-organized file system (GOFS) that represents a data graph using a plurality of gnode data structures and a plurality of edge entry data structures. Gnodes and edge entries both store one or more “in-structure” metadata values for graph component properties. A GOFS partition includes dedicated storage for “out-of-structure” graph component metadata values that are accessed using graph component identifiers. Search operations may use the in-structure and/or out-of-structure metadata values to efficiently identify graph search results. Search criteria may involve in-structure metadata values for both nodes and relationships. Accessing in-structure metadata values for a particular search operation may be performed using an index or from within the graph component data structures themselves.

Abstract: When a transfer target file is updated during multiple part upload of the transfer target file to an external storage system, a file storage system stops the multiple part upload of the transfer target file, and performs multiple part upload of the updated file to the external storage system.

Abstract: A data processing service receives a request from a first collaborator to create a clean room for data sharing collaboration with at least a second collaborator. In response, the data processing service creates an execution environment separate from the data environment of the first collaborator and the second collaborator. The first and second collaborators can then add content into the clean room in the form of data tables and executable notebooks. Approval from each collaborator is required before a notebook can be executed using any data table shared into the clean room. Upon receiving notebook approval from each collaborator, the data processing service creates a notebook job to execute the notebook on one or more cluster computing resources of the data processing service to generate an output.

Abstract: Automation solutions for event logging and debugging on KUBERNETES are disclosed. A system for event logging and debugging on a distributed application cluster includes a user interface and at least one processor for executing instructions. The instructions, when executed, instruct the at least one processor to provide an edge module configured to deploy a Berkeley Packet Filter (BPF) probe and a BPF program in a computing environment. The BPF probe is triggered based on an event associated with a distributed application running in a user space of the computing environment. Data associated with the event is collected in a kernel space of the computing environment via the BPF program and transferred from the BPF program to the edge module. A query request associated with the collected data is sent to the edge module. A response corresponding to the collected data is received at the user interface.

Abstract: A file system and a related method are presented. The file system includes a data storage including a plurality of data blocks; a merge index including a plurality of namespace entries, wherein the plurality of namespace entries include a plurality of blockmap entries and a plurality of local reference entries; a deduplication database including a plurality of deduplication indices and a plurality of global reference entries for a plurality of datasets; and an indexing system configured to generate the plurality of namespace entries and the plurality of global reference entries.

Abstract: Systems and methods are described for maintaining bifurcated data management while labeling data for artificial intelligence model development. For example, the system may receive a first label for a first sample from a first dataset, wherein the first dataset is accessible to a first subset of a plurality of users, and wherein the first subset comprises a first attribute. The system may receive first version metadata of the first label, wherein the first version metadata comprises a proposed label for the first sample assigned by a first user. The system may determine, based on a first user input from the first user, a first grouping of source code files for storing the first version metadata, wherein the first grouping of source code files is accessible to a second subset of the plurality of users, and wherein the second subset comprises a second attribute.

Abstract: A method for file system destinations includes obtaining events for storage on one or more of the storage systems. For each event, the method includes extracting at least one field value from the event, comparing the at least one field value to configurations of the storage systems to identify at least one storage system of the plurality of storage systems having a matching configuration, transmitting the event to an ingest module queue for the at least one storage system, selecting a partition for the event based on the at least one field value to obtain a selected partition, mapping the selected partition to a file using a partition mapping, and appending the event to the file on the at least one storage system.

Abstract: Using container-centric managed access, an administrator is enabled to define a set of future grants for each object that will be created in the future in a container managed by the administrator. When a user creates a database object, the system checks the future grants to determine if any apply to the user, the database object, or the combination. Any applicable future grants are applied to the database object before the user is allowed to modify it. As a result, the administrator is enabled to control the privileges associated with the database object even before the database object is created, while restricting individual object owners from managing privileges on their owned objects.

Abstract: The present disclosure relates to methods, systems, and storage media for detecting events based on updates to node profiles from electronic activities. Exemplary implementations may access an electronic activity transmitted or received via an electronic account associated with a data source provider; generate a plurality of activity field-value pairs; maintain a plurality of node profiles; identify a first state of a first node profile of the plurality of node profiles; update the first node profile using the electronic activity; identify a second state of the first node profile subsequent to updating the first node profile using the electronic activity; detect a state change of the first node profile based on the first state and the second state; determine that the state change satisfies an event condition; and store an association between the first node profile and an event type corresponding to the event condition.

Abstract: Methods, devices and computer program products facilitate the storage, access and management of log files that are associated with particular client devices. The log files provide a record of user or client device activities that are periodically sent to a data backup center. A dedicated log file server facilitates the processing and storage of an increasingly large number of log files that are generated by new and existing client devices. A storage server pre-processes the received log files to facilitate the processing and storage of the log files by the log file server. This Abstract is provided for the sole purpose of complying with the Abstract requirement rules. This Abstract is submitted with the explicit understanding that it will not be used to interpret or to limit the scope or the meaning of the claims.

Abstract: A method for generating query response comprising receiving, by a processor, an input query from a first user; performing, by the processor, query parsing on the input query to generate a parsed query; determining, by the processor, a query type associated with the parsed query; for the query type being determined as initial query, performing: generating, by the processor, a first follow-up query to the input query based on the parsed query, and generating, by the processor, responses to the parsed query and the first follow-up query; and for the query type being determined as follow-up query to a previous query entered by a second user, performing: performing, by the processor, learning of a query sequence from the previous query to the input query, and generating, by the processor, responses to the parsed query and a second follow-up query.

Abstract: A method of generating a unified graph is described. First graph data is received from a first data source, nodes within the first graph data including first, second, and third fields. Second graph data is received from a second data source, nodes within the second graph data including first, second, and fourth fields. The unified graph is generated from the first and second graph data using aggregated fields of the first field and the third field as a unique identifier of a unified schema for nodes within the unified graph having a first type and aggregated fields of the first field and the fourth field as a unique identifier of the unified schema for nodes within the unified graph having a second type, wherein nodes of the unified graph represent entities and edges of the unified graph represent relationships among the entities.