Abstract: A system for data comparison is disclosed. The system may receive a source configuration metadata. The system may configure a data extraction module to extract data from a data set in response to the source configuration metadata. The system may generate a pre-work data from the data set. The system may compare the pre-work data to generate a post-process data set. The system may generate a report corresponding to the post-process data set.

Abstract: A system may store a plurality of first records comprising a first variable in a first data storage format. The system may also store a plurality of second records comprising a second variable in a second data storage format. The first data storage format may have a different data structure than the second data storage format. The system may then generate a catalog including a first location of the first variable and a second location of the second variable. The first location identifies the first data storage format, which is where the first variable is stored. The second location identifies the second data storage format, which is where the second variable is stored. The system may then receive a request to access at least one of the first variable or the second variable and access the first variable and/or the second variable in response to the request.

Abstract: A system may receive a request for access to a first variable. The request may include a requested action and a variable identifier for the first variable. The request may also have a syntax that is incompatible with first data storage format. The system may parse the request to capture the variable identifier from the request. The system may also look up a location of the first variable in a catalog using the variable identifier. The location of the first variable may include the first data storage format. The system may generate a query to access the first variable. The syntax of the query may be compatible with the first data storage format. The system may then submit the query to access the first data storage format. The query may be configured to complete the requested action.

Abstract: A system for data comparison is disclosed. The system may receive a source configuration metadata. The system may configure a data extraction module to extract data from a data set in response to the source configuration metadata. The system may generate a pre-work data from the data set. The system may compare the pre-work data to generate a post-process data set. The system may generate a report corresponding to the post-process data set.

Abstract: Systems and methods for identifying data of interest are disclosed. The system may retrieve unstructured data from an internet data source via an alert system or RSS feed. The system may input the unstructured data into various models and scoring systems to determine whether the data is of interest. The models and scoring systems may be executed in order or in parallel. For example, the system may input the unstructured data into a Naïve Bayes machine learning model, a long short-term memory (LSTM) machine learning model, a named entity recognition (NER) model, a semantic role labeling (SRL) model, a sentiment scoring algorithm, and/or a gradient boosted regression tree (GBRT) machine learning model. Based on determining that the unstructured data is of interest, a data alert may be generated and transmitted for manual review or as part of an automated decisioning process.

Abstract: A system and method may allow for improved accuracy for name matching. The system may receive a name input and preprocess the name input. The name input may be tokenized to create a name token. The name token may be compared to a stored name. The system may calculate a first name matching score based on the comparison. The system may permute the name token to form a second level permutation name, compare the second level permutation name with the stored name, and calculate a second name matching score based on the comparison. The first name matching score may be compared with the second name matching score to improve accuracy in name matching.

Abstract: A system may receive a request to derive an output variable from a source variable. The request may include proposed logic to derive the output variable from the source variable. The system may then compare the proposed logic to existing logic to determine the proposed logic is new. In response to the proposed logic being new, the system may generate transformation code configured to execute the proposed logic. The system may further schedule the transformation code for execution at a predetermined time, and then execute the transformation code to generate data for the output variable.

Abstract: A distributed file system may store a plurality of entity attributes. A node consolidating system may identify similarities between entity attributes for different entities. The node consolidating system may generate an entity graph which displays relationships and similarities between entities in a graphical user interface.

Abstract: A system may retrieve a pattern from a pattern database with the pattern identifying a type of sensitive data. The system may also retrieve data identified by a variable from a big data management system. The system may then match the data to the pattern to detect the type of sensitive data in the data. An output may be generated in response to the data matching the pattern. A variable access permission may be retrieved for the variable from a permissions repository, a sensitive data permission may be retrieved for the type of sensitive data from the permissions repository, and the variable access permission may be compared to the sensitive data permission to detect a discrepancy.

Abstract: The system may validate a data source having a structured format and a grammar that includes tags. The system may identify a tag in the grammar. The system may parse the data source to extract attributes and/or values associated with the tags in response to successful validation. The system may also write the attributes and/or values to an output file separated by a preselected delimiter. A configuration file may identify the grammar, the preselected delimiter, and/or the data source. The data source may be in an XML format or a JSON format. The system may generate execution ready code in response to the validating the data source and the grammar. The output file may be a load ready file for ingestion into a big data storage format. The tag may include a parent tag and a sub tag corresponding to a hierarchy in the data source.

Abstract: A system may register a use case with the use case including an application. An application identifier may be assigned to the application. The system may generate a transformation associated with the use case. The transformation may include logic to derive an output variable from a source variable. The system may also execute the transformation to derive output data for the output variable from source data of the source variable. The system may further lookup an access permission for the application using the application identifier in response to an access request.

Abstract: A system may read an input file having an input file size and including a first record and a second record. The first and second record may each have a record length. The system may parse the input file into a first split file and a second split file, with the first split file including the first record and the second split file including the second record. The system may distribute the first split file to a first node to generate a first output file and the second split file to a second node to generate a second output file. Any number of additional split files may be distributed to generate any number output files. The system may combine the output files to generate a converted data file.

Abstract: A system may receive a request for access to a first variable. The request may include a requested action and a variable identifier for the first variable. The request may also have a syntax that is incompatible with first data storage format. The system may parse the request to capture the variable identifier from the request. The system may also look up a location of the first variable in a catalog using the variable identifier. The location of the first variable may include the first data storage format. The system may generate a query to access the first variable. The syntax of the query may be compatible with the first data storage format. The system may then submit the query to access the first data storage format. The query may be configured to complete the requested action.

Abstract: A system may receive a query configured to access a column in a data table. The data table may be in a flat file in a big data storage format. The system may detect the language type of the query and validate the query against the syntax of the language type. The system may also validate an access permission for data in the data table. The access permission may be stored in a permissions database. The system may generate a temporary table including the column with the temporary table configured to support the language type. The system may execute the query against the temporary table to generate a query result based on the temporary table as though it were the underlying data table. The system may enforce column-level or row-level access permissions by excluding columns or rows from the temporary table.

Abstract: A system may store a plurality of first records comprising a first variable in a first data storage format. The system may also store a plurality of second records comprising a second variable in a second data storage format. The first data storage format may have a different data structure than the second data storage format. The system may then generate a catalog including a first location of the first variable and a second location of the second variable. The first location identifies the first data storage format, which is where the first variable is stored. The second location identifies the second data storage format, which is where the second variable is stored. The system may then receive a request to access at least one of the first variable or the second variable and access the first variable and/or the second variable in response to the request.

Abstract: A system may receive a request for access to a first variable. The request may include a requested action and a variable identifier for the first variable. The request may also have a syntax that is incompatible with first data storage format. The system may parse the request to capture the variable identifier from the request. The system may also look up a location of the first variable in a catalog using the variable identifier. The location of the first variable may include the first data storage format. The system may generate a query to access the first variable. The syntax of the query may be compatible with the first data storage format. The system may then submit the query to the first data storage format. The query may be configured to complete the requested action.

Abstract: A system may receive a request to derive an output variable from a source variable. The request may include proposed logic to derive the output variable from the source variable. The system may then compare the proposed logic to existing logic to determine the proposed logic is new. In response to the proposed logic being new, the system may generate transformation code configured to execute the proposed logic. The system may further schedule the transformation code for execution at a predetermined time, and then execute the transformation code to generate data for the output variable.

Abstract: A system may read source data corresponding to a source variable and apply a transformation to the source variable to generate an output variable. The transformation may include logic, and the output variable may be configured for ingestion into a big data storage format. The system may record lineage data of the output variable that comprises the transformation and/or the source variable. The system may also receive a request to generate a requested output variable. The requested output variable may be generated from a second transformation that is the same as the first transformation. The system may thus match the first transformation to the second transformation using the lineage data. In response to matching the first transformation to the second transformation, the system may deny the request. The original output variable may be returned in response to the matching the first transformation to the second transformation.

Abstract: A system may receive a query configured to access a column in a data table. The data table may be in a flat file in a big data storage format. The system may detect the language type of the query and validate the query against the syntax of the language type. The system may also validate an access permission for data in the data table. The access permission may be stored in a permissions database. The system may generate a temporary table including the column with the temporary table configured to support the language type. The system may execute the query against the temporary table to generate a query result based on the temporary table as though it were the underlying data table. The system may enforce column-level or row-level access permissions by excluding columns or rows from the temporary table.

Abstract: A system may receive a request to derive an output variable from a source variable. The request may include proposed logic to derive the output variable from the source variable. The system may then compare the proposed logic to existing logic to determine the proposed logic is new. In response to the proposed logic being new, the system may generate transformation code configured to execute the proposed logic. The system may further schedule the transformation code for execution at a predetermined time, and then execute the transformation code to generate data for the output variable.