Abstract: The present invention relates, in an embodiment, to a system for automatically ingesting data into a data lake. In an embodiment of the present invention, the system comprises computer readable memory having recorded thereon instructions for execution by a processor having an ingestion engine and a metadata model. In an embodiment of the present invention, the instructions are configured to determine, via the metadata model, a type of metadata the ingested data contains; to generate guidelines for processing and transforming the ingested data based on the determined metadata; to apply the guidelines at the ingestion engine for how the ingestion engine processes and transforms the ingested data based on the determined metadata; and to store the transformed ingested data to a storage repository.

Abstract: Embodiments of the present invention relate to systems and methods for ingesting input data containing a plurality of records into a data lake. In an embodiment, the method comprises splitting the input data into a plurality of input splits consisting of a balanced number of records; reading the records from the plurality of input splits in parallel, regardless of the format and encoding of the input source; converting the input data within the records into at least one key/value pair; transforming the values input data into a serializable format; sorting the key/value pairs of the transformed values such that the records are sorted in the same order as they were read; writing the transformed values to an output file; and storing the output file to the data lake.