Abstract: Techniques are disclosed for data management in an information processing system. For example, a method comprises analyzing one or more data sources, wherein each of the one or more data sources comprise a set of metadata and usage information associated with the set of metadata. The method then determines at least one of data definitions and data classifications for the one or more sets of metadata across the one or more data sources, and stores the at least one of data definitions and data classifications for the one or more sets of metadata in a repository.

Abstract: Systems and methods for pattern-based multi-stage deterministic data classification that may reduce processing and memory overhead while providing more accurate data classifications.

Abstract: The present embodiments relate to data processing model recommendation and simulation. A model catalog executing on a cloud infrastructure (CI) system can parse data from an obtained dataset identifying aspects of the dataset. The parsed data from the dataset can be compared with a plurality of potential models stored in a domain ontology store of the model catalog to identify one or more recommended models. A simulation of any of the recommended models can be executed using a portion of the dataset to provide insights into output data resulting from executing the recommended model. The output data resulting from the simulation of the recommended model can be provided to the client for the client to either accept or reject the model.

Abstract: Methods and systems are configured to determine a semantic meaning for data and generate data processing rules based on the semantic meaning of the data. The semantic meaning includes syntactical or contextual meaning for the data that is determined, for example, by profiling, by the data processing system, values stored in a field included in data records of one or more datasets; applying, by the data processing system, one or more classifiers to the profiled values; identifying, based on applying the one or more classifiers, one or more attributes indicative of a logical or syntactical characteristic for the values of the field, with each of the one or more attributes having a respective confidence level that is based on an output of each of the one or more classifiers. The attributes are associated with the fields and are used for generating data processing rules and processing the data.

Abstract: In an aspect, an apparatus for generating an ingredient chain is presented. The apparatus includes at least a processor and a memory communicatively connected to the at least a processor. A memory contains instructions configuring at least a processor receive recipe data from a suer. At least a processor is configured to extract a plurality of ingredients from recipe data. At least a processor is configured to classify, utilizing an ingredient classifier, a plurality of ingredients to a plurality of impact factors. At least a processor is configured to generate, as a function of impact factors, an ingredient chain for a user.

Abstract: A data classification method and a device thereof for hierarchical multi-label classification are proposed. The data classification device includes an encoder configured to vectorize data, a decoder configured to receive the vectorized data from the encoder, and classification data stored in the decoder, wherein the decoder starts classification of the vectorized data on the basis of a first node in the classification data, allocates a plurality of nodes included in the classification data to the vectorized data, and forms partial layers based on the allocated nodes with respect to the vectorized data, and when terminal nodes or temporary nodes are allocated among the nodes, the decoder ends the allocation to the partial layers to which the terminal nodes or the temporary nodes are allocated and allocates final partial layers formed on the basis of the partial layers to the data.

Abstract: In some examples, a system performs data deduplication using a deduplication fingerprint index in a hash data structure comprising a plurality of blocks, wherein a block of the plurality of blocks comprises fingerprints computed based on content of respective data values. The system merges, in a merge operation, updates for the deduplication fingerprint index to the hash data structure stored in a persistent storage. As part of the merge operation, the system mirrors the updates to a cached copy of the hash data structure in a cache memory, and updates, in an indirect block, information regarding locations of blocks in the cached copy of the hash data structure.

Abstract: Disclosed are a data processing method and device, the method including: setting the scale of data; analyzing and computing a correlation between the data according to the scale of the data; and processing the data on the basis of the correlation according to a processing method corresponding to a set processing type. According to different processing types, disclosed are: a data management and storage method; a method for establishing indexes and analyzing management data; a method for managing and storing data on the basis of indexes; a method for displaying data; a method for analyzing and computing data; and a method for progressive transmission of data.

Abstract: A computer-implemented technique is described herein for assisting a user in advancing a task objective. The technique uses a suggestion-generating system (SGS) to provide one or more suggestions to a user in response to at least a last-submitted query provided by the user. The SGS may correspond to a classification-type or generative-type neural network. The SGS uses a machine-trained model that is trained using a multi-task training framework based on plural groups of training examples, which, in turn, are produced using different respective example-generating methods. One such example-generating method constructs a training example from queries in a search session. It operates by identifying the task-related intent the queries, and then identifying at least one sequence of queries in the search session that exhibits a coherent task-related intent. A training example is constructed based on queries in such a sequence.

Abstract: An adaptive multi-model item selection method, comprising: receiving, from one of a plurality of client devices, a request including a client-side feature vector representing a state of the client device; determining, by an advocate model, a probability distribution of a plurality of specialist cluster models from the client-side feature vector; choosing, by a use case selector, a cluster corresponding to a use case from the probability distribution; and obtaining, by the use case selector based on the cluster (i.e., the cluster that was sampled by the user case selector), a specialist cluster model from the plurality of specialist cluster models.

Abstract: A system can a divide database into a group of shards distributed among a group of data centers, wherein the group of shards comprises respective leader replicas. The system can determine respective correlation values between pairs of shards of the group of shards. The system can examine the pairs of shards in a descending order of respective correlation values, comprising, in response to determining that a respective pair of shards of the pairs of shards has a first correlation value greater than a predetermined threshold value, and that at least one shard of the respective pair of shards is unlocked, reassigning leader replicas of the respective pair of shards to be stored in a same data center of the group of data centers, and locking the leader replicas of the respective pair of shards from being reassigned to another data center of the group of data centers during the examining.

Abstract: Computing systems, methods, and non-transitory computer-readable storage media are provided. An average percentage of identifier collisions are determined in historical event data based on short identifiers associated with a first event location or a second event location geographically dispersed from each other. Behavioral models are generated based on behavior analytics of short identifiers in the historical event data. Adjusted behavioral models are generated based on the average percentage of identifier collisions. A client device obtains a short identifier and an adjusted behavioral model is determined. Whether the short identifier is associated with a previous event of a first event type at the first event location is determined. A new event value or an intermediate event value for the first event type is calculated. The client device is notified using the new event value or the intermediate event value, whereby the client device displays a notification.

Abstract: An apparatus and method for generating an instruction set for a user is provided. The apparatus includes at least a processor and a memory connected to the processor. The memory contains instructions configuring the at least a processor to receive a client datum from a client, where the client datum describes resources of the client, and to receive a user datum from the user, where user datum includes a target datum that describes resource transfer data from the client to the user. Initiation of resource transfer described by the target datum is triggered by the pattern exceeding a threshold. In addition, the memory contains instructions configuring the at least a processor to generate an interface query data structure including an input field and to display the first transfer datum and the second transfer datum hierarchically based on a user-input datum input to the input field.

Abstract: A computer system and method for verifying a data lineage of a data element is provided. The system comprises at least processor and a memory storing instructions which when executed by the processor cause the processor to perform the method. The method comprises extracting data transformation code from programming logic, extracting and describing data transformation semantics from the code using a data transformation ontology, filtering out non-business-significant transformations, and determining that remaining business-significant transformations match a business ontology.

Abstract: A data management platform for intelligently managing data is provided. The data management platform includes an ETL module configured to extract, cleanse, transform, or load data; a data lake configured to store a first group of data formed by extracting raw data from a plurality of data sources by the ETL module; a data warehouse configured to store a second group of data formed by cleansing and standardizing on the first group of data; a general data layer configured to store a third group of data formed by subjecting the second group of data to data fusion; and a data mart configured to store a fourth group of data formed by transforming the third group of data by the ETL module. The general data layer is a distributed data storage storing information available for querying. The data mart is a database of NoSQL type storing information available for computational processing.

Abstract: Methods and systems are provided for improved access to rows of data in a distributed data system. Each data row is associated with a partition. Data rows are distributed in one or more files and an impure file includes data rows associated multiple partitions. A clustering set is generated from a plurality of impure files by selecting a candidate impure file based on file access activity metrics and one or more neighbor impure files. Data rows of the impure files included in the clustering set are sorted according to their respective associated partitions. A set of disjoint partition range files are generated based on the sorted data rows of the impure files included in the clustering set. Each file of the set of disjoint partition range files is transferred to a respective target partition.

Abstract: Provided is a method of extracting a mediator variable and a mediation influence from a multivariate set. The method includes: receiving a database in which a set of N multiple variables (N is a natural number) including at least one of discrete time series data or continuous time series data is stored; selecting a test variable set including an independent variable, a dependent variable, and a mediator variable from the database in which the multivariate set is stored; determining the mediator variable using transfer entropy and conditional transfer entropy for the selected test variable set; and calculating a mediation influence of the determined mediator variable, wherein the mediation influence is separately calculated as an individual causal influence of the mediator variable and a group causal influence of mediator variables.

Abstract: Presented herein are systems and methods for regularly updating computer-form files. A method may include obtaining, by a computer, raw data containing a plurality of data records associated with a customer from a plurality of databases, for each data record, determining, by the computer, a data category indicating one or more computer-form files for a data entry of the data record based upon a preconfigured mapping between a type of data of the data entry mapped to the data category, in response to detecting a new customer data requirement, updating, by the computer, the one or more computer-form files associated with each data category according to each data record of each daily interval, and at a preconfigured time, generating, by the computer, the one or more computer-form files based upon each data category using each data record of each daily interval.

Abstract: Apparatus and methods are disclosed for using machine learning models with private and public domains. Operations can be applied to transform input to a machine learning model in a private domain that is kept secret or otherwise made unavailable to third parties. In one example of the disclosed technology, a method includes applying a private transform to produce transformed input, providing the transformed input to a machine learning model that was trained using a training set modified by the private transform, and generating inferences with the machine learning model using the transformed input. Examples of suitable transforms that can be employed include matrix multiplication, time or spatial domain to frequency domains, and partitioning a neural network model such that an input and at least one hidden layer form part of the private domain, while the remaining layers form part of the public domain.

Abstract: A method includes, for each key of a plurality of keys, identifying from a set of buckets a first bucket for the key based on a first hash function, and identifying from the set of buckets a second bucket for the key based on a second hash function. An entry for the key is stored in a bucket selected from one of the first bucket and the second bucket. The entry is inserted in a sequence of entries in a memory block. A position of the entry in the sequence of entries corresponds to the selected bucket. For each bucket in the set of buckets, an indication of a number of entries in the bucket is recorded.