Abstract: The present invention relates, in an embodiment, to a system for automatically ingesting data into a data lake. In an embodiment of the present invention, the system comprises computer readable memory having recorded thereon instructions for execution by a processor having an ingestion engine and a metadata model. In an embodiment of the present invention, the instructions are configured to determine, via the metadata model, a type of metadata the ingested data contains; to generate guidelines for processing and transforming the ingested data based on the determined metadata; to apply the guidelines at the ingestion engine for how the ingestion engine processes and transforms the ingested data based on the determined metadata; and to store the transformed ingested data to a storage repository.

Abstract: Embodiments of the present invention relate to systems and methods for ingesting input data containing a plurality of records into a data lake. In an embodiment, the method comprises splitting the input data into a plurality of input splits consisting of a balanced number of records; reading the records from the plurality of input splits in parallel, regardless of the format and encoding of the input source; converting the input data within the records into at least one key/value pair; transforming the values input data into a serializable format; sorting the key/value pairs of the transformed values such that the records are sorted in the same order as they were read; writing the transformed values to an output file; and storing the output file to the data lake.

Abstract: A system and method to automatically generate source code for moving and transforming data stored in a source database defined according to a first data model to a target database defined according to a second data model. At least one data mapping expression between the first data model to a conceptual module and at least one mapping and transformation definition between the conceptual model and the second data model are provided. Data Definition Language code is automatically generated for defining the target database based on the at least one data mapping expression and the second data model. Extract, Transform, Load (ETL) code is automatically generated for extracting data from the source database, transforming the data, and loading the transformed data into the target database. The ETL code based on the at least one mapping expression and the at least one mapping and transformation definition.

Abstract: The present invention relates, in an embodiment, to a method for ingesting input data containing a plurality of records into a data lake. In an embodiment, the method comprises splitting the input data into a plurality of input splits consisting of a balanced number of records; reading the records from the plurality of input splits in parallel, regardless of the format and encoding of the input source; converting the input data within the records into at least one key/value pair; transforming the values input data into a serializable format; sorting the key/value pairs of the transformed values such that the records are sorted in the same order as they were read; writing the transformed values to an output file; and storing the output file to the data lake. The present invention also relates, in another embodiment, to a system for ingesting input data containing a plurality of records into a data lake.

Abstract: The present invention relates, in an embodiment, to a system for automatically ingesting data into a data lake. In an embodiment of the present invention, the system comprises computer readable memory having recorded thereon instructions for execution by a processor having an ingestion engine and a metadata model. In an embodiment of the present invention, the instructions are configured to determine, via the metadata model, a type of metadata the ingested data contains; to generate guidelines for processing and transforming the ingested data based on the determined metadata; to apply the guidelines at the ingestion engine for how the ingestion engine processes and transforms the ingested data based on the determined metadata; and to store the transformed ingested data to a storage repository.