Abstract: A data structure management system includes a first database, a second database, and a processing engine. The first database includes a first file with a first term and a corresponding first metadata, and a second file with the first term and a corresponding second metadata. The processing engine extracts the first file and the second file from the first database in a first format. It links the first term with the first metadata from the first file and the second metadata from the second file. It transforms the extracted first file and second file from the first format into a second format while maintaining the link between the first term, the first metadata, and the second metadata. It then exports the transformed first file and second file to a second database in the second format with the link between the first term, the first metadata, and the second metadata intact.

Abstract: A parallel processing device includes a parallel processing engine implemented by a processor. The parallel processing engine is configured to execute a shell script for each particular processing job in a queue of processing jobs to run. The shell script is configured to dynamically generate a configuration file for each particular processing job. The configuration file instructs a network of computing systems to run the particular processing job using a particular number of parallel partitions corresponding to a parallel partitions parameter associated with the particular job. The configuration file includes randomized scratch directories for computing nodes within the network of computing systems and a calculated container size for the particular processing job. Each processing job is run on the network of computing systems according to the dynamically-generated configuration file of the particular processing job.

Abstract: A system for communication between two or more computer programs is disclosed. The system includes a memory, an interface, and a processor. The memory stores a first file, expected metadata for the first file, and expected metadata for one or more fields in the first file. The interface receives a file from a computer program. The file comprises fields that each comprise information provided by one or more sources. The processor executes a second computer program which extracts a first set of file metadata from the received file, compares the extracted first set of file metadata to the expected metadata, and determines if the extracted first set of file metadata corresponds to the expected metadata. If the extracted first set of file metadata corresponds to the expected metadata for the first file, then the processor performs analogous comparisons at a field level and stores the first file in the memory.

Abstract: A data structure management system includes a first database, a second database, and a processing engine. The first database includes a first file with a first term and a corresponding first metadata, and a second file with the first term and a corresponding second metadata. The processing engine extracts the first file and the second file from the first database in a first format. It links the first term with the first metadata from the first file and the second metadata from the second file. It transforms the extracted first file and second file from the first format into a second format while maintaining the link between the first term, the first metadata, and the second metadata. It then exports the transformed first file and second file to a second database in the second format with the link between the first term, the first metadata, and the second metadata intact.

Abstract: A parallel processing device includes a parallel processing engine implemented by a processor. The parallel processing engine is configured to execute a shell script for each particular processing job in a queue of processing jobs to run. The shell script is configured to dynamically generate a configuration file for each particular processing job. The configuration file instructs a network of computing systems to run the particular processing job using a particular number of parallel partitions corresponding to a parallel partitions parameter associated with the particular job. The configuration file includes randomized scratch directories for computing nodes within the network of computing systems and a calculated container size for the particular processing job. Each processing job is run on the network of computing systems according to the dynamically-generated configuration file of the particular processing job.

Abstract: A data structure management system includes a first database, a second database, and a processing engine. The first database includes a first file with a first term and a corresponding first metadata, and a second file with the first term and a corresponding second metadata. The processing engine extracts the first file and the second file from the first database in a first format. It links the first term with the first metadata from the first file and the second metadata from the second file. It transforms the extracted first file and second file from the first format into a second format while maintaining the link between the first term, the first metadata, and the second metadata. It then exports the transformed first file and second file to a second database in the second format with the link between the first term, the first metadata, and the second metadata intact.

Abstract: A parallel processing device includes a parallel processing engine implemented by a processor. The parallel processing engine is configured to execute a shell script for each particular processing job in a queue of processing jobs to run. The shell script is configured to dynamically generate a configuration file for each particular processing job. The configuration file instructs a network of computing systems to run the particular processing job using a particular number of parallel partitions corresponding to a parallel partitions parameter associated with the particular job. The configuration file includes randomized scratch directories for computing nodes within the network of computing systems and a calculated container size for the particular processing job. Each processing job is run on the network of computing systems according to the dynamically-generated configuration file of the particular processing job.

Abstract: A parallel processing device includes a parallel processing engine implemented by a processor. The parallel processing engine is configured to execute a shell script for each particular processing job in a queue of processing jobs to run. The shell script is configured to dynamically generate a configuration file for each particular processing job. The configuration file instructs a network of computing systems to run the particular processing job using a particular number of parallel partitions corresponding to a parallel partitions parameter associated with the particular job. The configuration file includes randomized scratch directories for computing nodes within the network of computing systems and a calculated container size for the particular processing job. Each processing job is run on the network of computing systems according to the dynamically-generated configuration file of the particular processing job.

Abstract: A system for communication between two or more computer programs is disclosed. The system includes a memory, an interface, and a processor. The memory stores a first file, expected metadata for the first file, and expected metadata for one or more fields in the first file. The interface receives a file from a computer program. The file comprises fields that each comprise information provided by one or more sources. The processor executes a second computer program which extracts a first set of file metadata from the received file, compares the extracted first set of file metadata to the expected metadata, and determines if the extracted first set of file metadata corresponds to the expected metadata. If the extracted first set of file metadata corresponds to the expected metadata for the first file, then the processor performs analogous comparisons at a field level and stores the first file in the memory.

Abstract: An adaptive machine learning system for predicting file controls includes a memory, an interface, and a processor. The memory stores a plurality of controls for incoming files and the interface receives a first file and a second file. The first file has a first property and the second file has a second property. The processor determines a type for each of the first property and the second property, wherein the type of each property is related to a first file control. The processor also determines that the first property and the second property each satisfy the first file control. If the value of the first property and the second property are above a first threshold, the processor changes a value of the first control for incoming files.

Abstract: A system for communication between two or more computer programs is disclosed. The system includes a memory, an interface, and a processor. The memory stores a first file, expected metadata for the first file, and expected metadata for one or more fields in the first file. The interface receives a file from a computer program. The file comprises fields that each comprise information provided by one or more sources. The processor executes a second computer program which extracts a first set of file metadata from the received file, compares the extracted first set of file metadata to the expected metadata, and determines if the extracted first set of file metadata corresponds to the expected metadata. If the extracted first set of file metadata corresponds to the expected metadata for the first file, then the processor performs analogous comparisons at a field level and stores the first file in the memory.

Abstract: A system for file management in data structures is disclosed. The system includes a memory, an extraction engine, an enrichment engine, a portal, and a transmission engine. The memory stores a first database and a memory. The extraction engine extracts columns and fields and associates them with extracted terms. The enrichment engine determines an end-to-end dataflow of the data from extracted metadata and loads the associated data into the memory. Enrichment engine performs either a full load comprising loading all the associated data onto the memory or a delta load comprising comparing the extracted data with data stored in the memory and loading any different data from the extracted data onto the memory. The enrichment engine also generates a journal recording metadata associated with the full or delta load. A portal displays the end-to-end dataflow of the associated data and a transmission engine transmits communication identifying incomplete associated data.

Abstract: According to one embodiment, a system for converting data integration system (“DIS”) files comprises a memory operable to store data associated with at least one DIS and a processor communicatively coupled to the memory and operable to convert files associated with a first DIS to files associated with a second DIS. The operating system used by the first DIS is different from the operating system used by the second DIS. The processor converts the files by being operable to determine differences between the first DIS and the second DIS, determine a set of transformation rules based on the differences, create a conversion algorithm based on the set of transformation rules, and execute the conversion algorithm to convert the files. The system is further operable to execute the second DIS such that the second DIS uses the converted data integration files.

Abstract: A system for communication between two or more computer programs is disclosed. The system includes a memory, an interface, and a processor. The memory stores a first file, expected metadata for the first file, and expected metadata for one or more fields in the first file. The interface receives a file from a computer program. The file comprises fields that each comprise information provided by one or more sources. The processor executes a second computer program which extracts a first set of file metadata from the received file, compares the extracted first set of file metadata to the expected metadata, and determines if the extracted first set of file metadata corresponds to the expected metadata. If the extracted first set of file metadata corresponds to the expected metadata for the first file, then the processor performs analogous comparisons at a field level and stores the first file in the memory.

Abstract: An adaptive machine learning system for predicting file controls includes a memory, an interface, and a processor. The memory stores a plurality of controls for incoming files and the interface receives a first file and a second file. The first file has a first property and the second file has a second property. The processor determines a type for each of the first property and the second property, wherein the type of each property is related to a first file control. The processor also determines that the first property and the second property each satisfy the first file control. If the value of the first property and the second property are above a first threshold, the processor changes a value of the first control for incoming files.

Abstract: A data structure management system includes a first database, a second database, and a processing engine. The first database includes a first file with a first term and a corresponding first metadata, and a second file with the first term and a corresponding second metadata. The processing engine extracts the first file and the second file from the first database in a first format. It links the first term with the first metadata from the first file and the second metadata from the second file. It transforms the extracted first file and second file from the first format into a second format while maintaining the link between the first term, the first metadata, and the second metadata. It then exports the transformed first file and second file to a second database in the second format with the link between the first term, the first metadata, and the second metadata intact.

Abstract: A system for file management in data structures is disclosed. The system includes a memory, an extraction engine, an enrichment engine, a portal, and a transmission engine. The memory stores a first database and a memory. The extraction engine extracts columns and fields and associates them with extracted terms. The enrichment engine determines an end-to-end dataflow of the data from extracted metadata and loads the associated data into the memory. Enrichment engine performs either a full load comprising loading all the associated data onto the memory or a delta load comprising comparing the extracted data with data stored in the memory and loading any different data from the extracted data onto the memory. The enrichment engine also generates a journal recording metadata associated with the full or delta load. A portal displays the end-to-end dataflow of the associated data and a transmission engine transmits communication identifying incomplete associated data.

Abstract: According to one embodiment, a method of copying a dataset associated with a first extract, transform, and load (ETL) job in a first data integration system to a second data integration system comprises copying executable code associated with the first ETL job from the first to the second system. Operating system software, integration system software, and file system structure are substantially identical between the first and second systems. The method further comprises executing the second ETL job to read the dataset from the first data integration system and write the dataset to the second data integration system. The second ETL job is associated with configuration parameters specifying storage resources in the first system associated with the dataset and destination parameters specifying storage resources in the second system. The method further comprises copying metadata generated by the second ETL job from the first to the second data integration system.

Abstract: According to one embodiment, a system for converting data integration system (“DIS”) files comprises a memory operable to store data associated with at least one DIS and a processor communicatively coupled to the memory and operable to convert files associated with a first DIS to files associated with a second DIS. The operating system used by the first DIS is different from the operating system used by the second DIS. The processor converts the files by being operable to determine differences between the first DIS and the second DIS, determine a set of transformation rules based on the differences, create a conversion algorithm based on the set of transformation rules, and execute the conversion algorithm to convert the files. The system is further operable to execute the second DIS such that the second DIS uses the converted data integration files.

Abstract: According to one embodiment, a method of copying a dataset associated with a first extract, transform, and load (ETL) job in a first data integration system to a second data integration system comprises copying executable code associated with the first ETL job from the first to the second system. Operating system software, integration system software, and file system structure are substantially identical between the first and second systems. The method further comprises executing the second ETL job to read the dataset from the first data integration system and write the dataset to the second data integration system. The second ETL job is associated with configuration parameters specifying storage resources in the first system associated with the dataset and destination parameters specifying storage resources in the second system. The method further comprises copying metadata generated by the second ETL job from the first to the second data integration system.