Abstract: Systems and methods to analyze and significantly compress FastQ and/or FastA datasets are disclosed. The methodology includes algorithms to compress sequences, quality scores and identifiers of read files. The method relies on reducing the dimension and redundancy in genomic data in a unique and optimal way and in the binary format. The methodology also includes the decoding protocols to decompress the compressed data with zero loss.

Abstract: Techniques are described herein for automatically discovering undeclared PK-FK relationships between dimension tables and a fact table in situations where: the primary keys of the dimension tables (a) are not known, and (b) are composite keys, and the foreign keys of the fact table (a) are not known, and (b) are composite keys. The techniques involve removing from consideration all fact table columns that satisfy first removal criteria, and all dimension table columns that satisfy second removal criteria. Fingerprints are generated for the fact and dimension table columns that remain. Matched pairs (e.g. two fact table columns and two dimension table columns) are generated based on the columns that have not been filtered. The matched pairs are further filtered based on third removal criteria. The matched pairs that remain are then graded, and a PK-FK relationship is established based on the matched pair with the highest grade.

Abstract: A cloud migration framework may include an enterprise application data store that contains electronic records associated with enterprise applications. Each record may include, for example, an electronic record identifier and enterprise application parameters. A data repository stores a catalogue of cloud computing patterns. A back-end application computer server retrieves information from the enterprise application data store and, based on enterprise application parameters, creates a move group representing a subset of the enterprise applications. For each application in the move group, the computer server identifies whether an appropriate cloud computing pattern exists in the catalogue. If an appropriate cloud computing pattern is identified, it is used to automatically create a reference implementation of the enterprise application in a cloud computing environment.

Abstract: Systems and methods for tracking data lineage and record lifecycle using a distributed ledger are disclosed. In one embodiment, a method for tracking record lifecycle events may include: (1) creating, by a record lifecycle tool, a recordable artifact for a record lifecycle event in a record lifecycle, the recordable artifact comprising data for the record lifecycle event; (2) generating, by the record lifecycle tool, a hash of the data; (3) creating, by the record lifecycle tool, record lifecycle event metadata for the recordable artifact; (4) signing, by the record lifecycle tool, the record lifecycle event metadata; (5) writing the record lifecycle event metadata to supply chain metadata storage at a storage location, wherein the supply chain metadata storage may be cryptographically verifiable and immutable; and (6) writing the hash and an identifier for the storage location in the supply chain metadata store to a present state database.

Abstract: The present disclosure relates to the technical field of information retrieval, and discloses a method for constructing an entity relationship graph, including: performing entity recognition and an operation of uniting like terms on the text to be processed to obtain a standard text entity set; performing grammatical structure analysis on the text to be processed, to obtain a grammatical relationship graph; selecting the node connected to the connection edge in the grammatical relationship graph as a target connection node and performing vector conversion to obtain a text vector, and analyzing a text entity relationship between the target connection nodes according to the text vectors; constructing a knowledge graph according to the standard text entities and the text entity relationship to obtain the entity relationship graph corresponding to the text to be processed. The present disclosure can improve the accuracy of the text entity relationship graph.

Abstract: A storage system is provided. The storage system include a primary storage node that includes a primary processing device. The primary storage node is communicatively coupled to a secondary storage node. The secondary storage node includes a secondary processing device and a set of non-volatile memory modules. The primary processing device is to identify one or more storage operations to be performed on the set of non-volatile memory modules of the secondary storage node and transmit one or more instructions to the secondary storage node to perform the one or more storage operations, the one or more storage operations performed by the secondary processing device.

Abstract: A method and related system for generating and presenting queries is disclosed. The method includes selecting a first context based on a set of inputs including a first query and determining a match between the first context and a stored context associated with a schema. The method further includes generating a second query based on the set of inputs with a first language model in response to the determination of the match and identifying an indicated portion of the second query by providing, as a second input, the second query to a second language model. The method also includes generating data, the data including an issue indicator of the second query based on the indicated portion; and sending the data to the client device.

Abstract: Systems and methods for object-based data storage are provided. A queue may be maintained of operations relating to a plurality of documents operable to be maintained at an object-based data storage. An independent operation may be identified in the queue that must be processed prior to processing at least one dependent operation to thereby enable parallelization of processing of operations in the queue. The identified independent operation may then be processed. Subsequently, the dependent operations may be processed.

Abstract: A data processing service facilitates the creation and processing of data processing pipelines that process data processing jobs defined with respect to a set of tasks in a sequence and with data dependencies associated with each separate task such that the output from one task is used as input for a subsequent task. In various embodiments, the set of tasks include at least one cleanroom task that is executed in a cleanroom station and at least one non-cleanroom task executed in an execution environment of a user where each task is configured to read one or more input datasets and transform the one or more input datasets into one or more output datasets.

Abstract: A system and method for defining a relational schema for a non-relational database are disclosed. In one embodiment, the system identifies a non-relational database; obtains a data structure of the non-relational database; determines a relational schema type; generates a relational schema based on the determined schema type; determines whether the relational schema includes a plurality of relational tables; and responsive to determining that the relational schema includes a plurality of relational tables, defines a relation between at least two of the relational tables in the relational schema.

Abstract: Methods and systems for managing operation of a data pipeline are disclosed. To manage the operation, a system may include one or more data sources, a data manager, and one or more downstream consumers. Interruptions to the operation may impact provision of data processing services by the data pipeline and may cause the data processing services to no longer align with operation quality goals for the data pipeline. To maintain compliance with the operation quality goals, the operation may be monitored over time. Operation data may be obtained for the data pipeline and may be used to determine representations of operation quality of the data pipeline. The representations of operation quality of the data pipeline may be compared to the operation quality goals and actions may be performed to remediate differences between the representations of operation quality of the data pipeline and the operation quality goals.

Abstract: A data processing method for an electronic device includes: obtaining data rebalance information representing a new partition plan for a partition table, the partition table being a table for data distribution based on a partition key; creating, for a set specified in the partition plan, a corresponding partition space, and recording, in the corresponding partition space, a current data range of a set corresponding to the partition space, the data range comprising one of a value range of the partition key and a value list of the partition key; updating the data range recorded in the corresponding partition space based on the partition plan, and updating correspondence between the corresponding partition space and sets of the partition table by using the updated data range in the corresponding partition space; and updating data distribution in the sets of the partition table based on the correspondence.

Abstract: Disclosed are various embodiments for data anomaly detection. A variable profile is generated for each variable in source data. Then, the variable profiles are provided to each of a plurality of machine learning models. Next, it is determined, with each of the plurality of machine learning models, whether each variable profile is anomalous. The determination, from each of the plurality of machine learning models, whether each variable profile is anomalous is provided to an ensemble model. The ensemble model then generates a final determination whether each variable profile is anomalous. The final determination is then reported to an analysis service.

Abstract: A computer-implemented method, including receiving, by a processor set, a query including a query string for a system catalog; identifying, by the processor set, a default index structure of the system catalog; executing, by the processor set, the query based on the default index structure of each index in the system catalog; ranking, by the processor set, a performance of each execution of the query by each index; mapping, by the processor set, a query pattern to a corresponding index of the system catalog; selecting, by the processor set, the index to perform the query using a machine learning (ML) model trained with a knowledge base that includes the ranking and the mapping; executing, by the processor set, the query on the selected index; and in response to executing the query on the selected index, returning, by the processor set, a result of the query.

Abstract: Systems, methods, devices and non-transitory, computer-readable storage mediums are disclosed for a wearable multimedia device and cloud computing platform with an application ecosystem for processing multimedia data captured by the wearable multimedia device. In an embodiment, a method for using a machine learning model to provide data corresponding to a query comprises receiving a query for data stored in one or more databases. A prompt file is determined for the query. The prompt file and the query are provided as input for a machine learning model configured to generate a prediction for the query. Training the machine learning model can include updating network parameters in the machine learning model based on the prompt file. The prediction generated for the query is received. The prediction comprises a predicted database query corresponding to the query, and a level of accuracy is determined for the predicted database query.

Abstract: Methods, devices and computer program products facilitate the storage, access and management of log files that are associated with particular client devices. The log files provide a record of user or client device activities that are periodically sent to a data backup center. A dedicated log file server facilitates the processing and storage of an increasingly large number of log files that are generated by new and existing client devices. A storage server pre-processes the received log files to facilitate the processing and storage of the log files by the log file server. This Abstract is provided for the sole purpose of complying with the Abstract requirement rules. This Abstract is submitted with the explicit understanding that it will not be used to interpret or to limit the scope or the meaning of the claims.

Abstract: Techniques are described herein for a graph-organized file system (GOFS) that represents a data graph using a plurality of gnode data structures and a plurality of edge entry data structures. Gnodes and edge entries both store one or more “in-structure” metadata values for graph component properties. A GOFS partition includes dedicated storage for “out-of-structure” graph component metadata values that are accessed using graph component identifiers. Search operations may use the in-structure and/or out-of-structure metadata values to efficiently identify graph search results. Search criteria may involve in-structure metadata values for both nodes and relationships. Accessing in-structure metadata values for a particular search operation may be performed using an index or from within the graph component data structures themselves.

Abstract: A data processing service receives a request from a first collaborator to create a clean room for data sharing collaboration with at least a second collaborator. In response, the data processing service creates an execution environment separate from the data environment of the first collaborator and the second collaborator. The first and second collaborators can then add content into the clean room in the form of data tables and executable notebooks. Approval from each collaborator is required before a notebook can be executed using any data table shared into the clean room. Upon receiving notebook approval from each collaborator, the data processing service creates a notebook job to execute the notebook on one or more cluster computing resources of the data processing service to generate an output.

Abstract: When a transfer target file is updated during multiple part upload of the transfer target file to an external storage system, a file storage system stops the multiple part upload of the transfer target file, and performs multiple part upload of the updated file to the external storage system.

Abstract: Automation solutions for event logging and debugging on KUBERNETES are disclosed. A system for event logging and debugging on a distributed application cluster includes a user interface and at least one processor for executing instructions. The instructions, when executed, instruct the at least one processor to provide an edge module configured to deploy a Berkeley Packet Filter (BPF) probe and a BPF program in a computing environment. The BPF probe is triggered based on an event associated with a distributed application running in a user space of the computing environment. Data associated with the event is collected in a kernel space of the computing environment via the BPF program and transferred from the BPF program to the edge module. A query request associated with the collected data is sent to the edge module. A response corresponding to the collected data is received at the user interface.