Abstract: An apparatus, computer-readable medium, and computer-implemented method for ingesting semi-structured data in a columnar format, including parsing a data record, the data record comprising semi-structured data including one or more data fields and one or more data values corresponding to the one or more data fields, generating a schema corresponding to the data record, the schema comprising the one or more data fields and one or more data types corresponding to the one or more data fields, determining whether the schema corresponding to the data record is homogenous with a schema corresponding to a current row group, wherein the current row group comprises one or more current column names and one or more current column types, and storing the data record as a new record in the current row group based at least in part on a determination that the schema is homogenous.

Abstract: An apparatus, computer-readable medium, and computer-implemented method for encoding repetition and definition level values for a semi-structured data record, including identifying a leaf of an entry in a data record, the entry specifying one or more data fields in the data record and the leaf corresponding to a last data field in the entry, storing a value of the leaf in a column which corresponds to a nesting path of the leaf within the data record, the column being in a current row group, determining a repetition level of the entry for the column, determining a definition level of the entry for the column based at least in part on a nesting level of the leaf in the data record, and storing the repetition level and definition level of the entry for the column in a table of repetition and definition levels for the current row group.

Abstract: An apparatus, computer-readable medium, and computer-implemented method for ingesting semi-structured data in a columnar format, including parsing a data record, the data record comprising semi-structured data including one or more data fields and one or more data values corresponding to the one or more data fields, generating a schema corresponding to the data record, the schema comprising the one or more data fields and one or more data types corresponding to the one or more data fields, determining whether the schema corresponding to the data record is homogenous with a schema corresponding to a current row group, wherein the current row group comprises one or more current column names and one or more current column types, and storing the data record as a new record in the current row group based at least in part on a determination that the schema is homogenous.