Abstract: A data structure management system includes a first database, a second database, and a processing engine. The first database includes a first file with a first term and a corresponding first metadata, and a second file with the first term and a corresponding second metadata. The processing engine extracts the first file and the second file from the first database in a first format. It links the first term with the first metadata from the first file and the second metadata from the second file. It transforms the extracted first file and second file from the first format into a second format while maintaining the link between the first term, the first metadata, and the second metadata. It then exports the transformed first file and second file to a second database in the second format with the link between the first term, the first metadata, and the second metadata intact.

Abstract: A parallel processing device includes a parallel processing engine implemented by a processor. The parallel processing engine is configured to execute a shell script for each particular processing job in a queue of processing jobs to run. The shell script is configured to dynamically generate a configuration file for each particular processing job. The configuration file instructs a network of computing systems to run the particular processing job using a particular number of parallel partitions corresponding to a parallel partitions parameter associated with the particular job. The configuration file includes randomized scratch directories for computing nodes within the network of computing systems and a calculated container size for the particular processing job. Each processing job is run on the network of computing systems according to the dynamically-generated configuration file of the particular processing job.

Abstract: A system for communication between two or more computer programs is disclosed. The system includes a memory, an interface, and a processor. The memory stores a first file, expected metadata for the first file, and expected metadata for one or more fields in the first file. The interface receives a file from a computer program. The file comprises fields that each comprise information provided by one or more sources. The processor executes a second computer program which extracts a first set of file metadata from the received file, compares the extracted first set of file metadata to the expected metadata, and determines if the extracted first set of file metadata corresponds to the expected metadata. If the extracted first set of file metadata corresponds to the expected metadata for the first file, then the processor performs analogous comparisons at a field level and stores the first file in the memory.

Abstract: A data structure management system includes a first database, a second database, and a processing engine. The first database includes a first file with a first term and a corresponding first metadata, and a second file with the first term and a corresponding second metadata. The processing engine extracts the first file and the second file from the first database in a first format. It links the first term with the first metadata from the first file and the second metadata from the second file. It transforms the extracted first file and second file from the first format into a second format while maintaining the link between the first term, the first metadata, and the second metadata. It then exports the transformed first file and second file to a second database in the second format with the link between the first term, the first metadata, and the second metadata intact.

Abstract: A parallel processing device includes a parallel processing engine implemented by a processor. The parallel processing engine is configured to execute a shell script for each particular processing job in a queue of processing jobs to run. The shell script is configured to dynamically generate a configuration file for each particular processing job. The configuration file instructs a network of computing systems to run the particular processing job using a particular number of parallel partitions corresponding to a parallel partitions parameter associated with the particular job. The configuration file includes randomized scratch directories for computing nodes within the network of computing systems and a calculated container size for the particular processing job. Each processing job is run on the network of computing systems according to the dynamically-generated configuration file of the particular processing job.

Abstract: A data structure management system includes a first database, a second database, and a processing engine. The first database includes a first file with a first term and a corresponding first metadata, and a second file with the first term and a corresponding second metadata. The processing engine extracts the first file and the second file from the first database in a first format. It links the first term with the first metadata from the first file and the second metadata from the second file. It transforms the extracted first file and second file from the first format into a second format while maintaining the link between the first term, the first metadata, and the second metadata. It then exports the transformed first file and second file to a second database in the second format with the link between the first term, the first metadata, and the second metadata intact.

Abstract: A parallel processing device includes a parallel processing engine implemented by a processor. The parallel processing engine is configured to execute a shell script for each particular processing job in a queue of processing jobs to run. The shell script is configured to dynamically generate a configuration file for each particular processing job. The configuration file instructs a network of computing systems to run the particular processing job using a particular number of parallel partitions corresponding to a parallel partitions parameter associated with the particular job. The configuration file includes randomized scratch directories for computing nodes within the network of computing systems and a calculated container size for the particular processing job. Each processing job is run on the network of computing systems according to the dynamically-generated configuration file of the particular processing job.

Abstract: A parallel processing device includes a parallel processing engine implemented by a processor. The parallel processing engine is configured to execute a shell script for each particular processing job in a queue of processing jobs to run. The shell script is configured to dynamically generate a configuration file for each particular processing job. The configuration file instructs a network of computing systems to run the particular processing job using a particular number of parallel partitions corresponding to a parallel partitions parameter associated with the particular job. The configuration file includes randomized scratch directories for computing nodes within the network of computing systems and a calculated container size for the particular processing job. Each processing job is run on the network of computing systems according to the dynamically-generated configuration file of the particular processing job.

Abstract: A system for communication between two or more computer programs is disclosed. The system includes a memory, an interface, and a processor. The memory stores a first file, expected metadata for the first file, and expected metadata for one or more fields in the first file. The interface receives a file from a computer program. The file comprises fields that each comprise information provided by one or more sources. The processor executes a second computer program which extracts a first set of file metadata from the received file, compares the extracted first set of file metadata to the expected metadata, and determines if the extracted first set of file metadata corresponds to the expected metadata. If the extracted first set of file metadata corresponds to the expected metadata for the first file, then the processor performs analogous comparisons at a field level and stores the first file in the memory.

Abstract: An adaptive machine learning system for predicting file controls includes a memory, an interface, and a processor. The memory stores a plurality of controls for incoming files and the interface receives a first file and a second file. The first file has a first property and the second file has a second property. The processor determines a type for each of the first property and the second property, wherein the type of each property is related to a first file control. The processor also determines that the first property and the second property each satisfy the first file control. If the value of the first property and the second property are above a first threshold, the processor changes a value of the first control for incoming files.

Abstract: A system for communication between two or more computer programs is disclosed. The system includes a memory, an interface, and a processor. The memory stores a first file, expected metadata for the first file, and expected metadata for one or more fields in the first file. The interface receives a file from a computer program. The file comprises fields that each comprise information provided by one or more sources. The processor executes a second computer program which extracts a first set of file metadata from the received file, compares the extracted first set of file metadata to the expected metadata, and determines if the extracted first set of file metadata corresponds to the expected metadata. If the extracted first set of file metadata corresponds to the expected metadata for the first file, then the processor performs analogous comparisons at a field level and stores the first file in the memory.

Abstract: A system for file management in data structures is disclosed. The system includes a memory, an extraction engine, an enrichment engine, a portal, and a transmission engine. The memory stores a first database and a memory. The extraction engine extracts columns and fields and associates them with extracted terms. The enrichment engine determines an end-to-end dataflow of the data from extracted metadata and loads the associated data into the memory. Enrichment engine performs either a full load comprising loading all the associated data onto the memory or a delta load comprising comparing the extracted data with data stored in the memory and loading any different data from the extracted data onto the memory. The enrichment engine also generates a journal recording metadata associated with the full or delta load. A portal displays the end-to-end dataflow of the associated data and a transmission engine transmits communication identifying incomplete associated data.

Abstract: A data structure management system includes a first database, a second database, and a processing engine. The first database includes a first file with a first term and a corresponding first metadata, and a second file with the first term and a corresponding second metadata. The processing engine extracts the first file and the second file from the first database in a first format. It links the first term with the first metadata from the first file and the second metadata from the second file. It transforms the extracted first file and second file from the first format into a second format while maintaining the link between the first term, the first metadata, and the second metadata. It then exports the transformed first file and second file to a second database in the second format with the link between the first term, the first metadata, and the second metadata intact.

Abstract: An adaptive machine learning system for predicting file controls includes a memory, an interface, and a processor. The memory stores a plurality of controls for incoming files and the interface receives a first file and a second file. The first file has a first property and the second file has a second property. The processor determines a type for each of the first property and the second property, wherein the type of each property is related to a first file control. The processor also determines that the first property and the second property each satisfy the first file control. If the value of the first property and the second property are above a first threshold, the processor changes a value of the first control for incoming files.

Abstract: A system for communication between two or more computer programs is disclosed. The system includes a memory, an interface, and a processor. The memory stores a first file, expected metadata for the first file, and expected metadata for one or more fields in the first file. The interface receives a file from a computer program. The file comprises fields that each comprise information provided by one or more sources. The processor executes a second computer program which extracts a first set of file metadata from the received file, compares the extracted first set of file metadata to the expected metadata, and determines if the extracted first set of file metadata corresponds to the expected metadata. If the extracted first set of file metadata corresponds to the expected metadata for the first file, then the processor performs analogous comparisons at a field level and stores the first file in the memory.

Abstract: A system for file management in data structures is disclosed. The system includes a memory, an extraction engine, an enrichment engine, a portal, and a transmission engine. The memory stores a first database and a memory. The extraction engine extracts columns and fields and associates them with extracted terms. The enrichment engine determines an end-to-end dataflow of the data from extracted metadata and loads the associated data into the memory. Enrichment engine performs either a full load comprising loading all the associated data onto the memory or a delta load comprising comparing the extracted data with data stored in the memory and loading any different data from the extracted data onto the memory. The enrichment engine also generates a journal recording metadata associated with the full or delta load. A portal displays the end-to-end dataflow of the associated data and a transmission engine transmits communication identifying incomplete associated data.