Abstract: Apparatus, methods, and computer-readable media for providing frameworks for data source representation and compression using an index file format are disclosed herein. The index file format separate information about symbols in a data file and information about the corresponding location of those symbols in the data file. The described techniques provide mechanisms for reducing the size associated with the representation of the symbols information and/or the size associated with the representation of the location information.

Abstract: Disclosed are system and methods for processing and storing data files, using a data edge file format. The data edge file format separates information about what symbols are in a data file and information about the corresponding location of those symbols in the data file. Examples convert a source file comprising symbols into a data edge index having a manifest portion, a symbol portion, and a locality portion. The symbol portion contains a sorted unique set of symbols from the source file, and the locality portion contains a plurality of location values referencing the symbol portion. Examples include normalizing structured data from the source file by modifying the locality manifest portion of the data edge file to include a description of at least one nonexistent column empty locality value at a respective position within the locality file representing an omission of data at an associated position in the source file.

Abstract: Apparatus, methods, and computer-readable media facilitating efficiently scaling result caching are disclosed herein. An example method includes generating an index based on a plurality of source data objects in an object storage system. The generated index comprises a manifest, at least one symbol file, and at least one locality file. The example method also includes receiving a search query for the plurality of source data objects stored in the object storage system, and querying the generated index based on the search query and a manifest root file of the manifest. Additionally, the example method includes generating a materialized view of a result set of the search query based on the querying of the generated index. The example method also includes storing a cached manifest file at the generated index, the cached manifest file mapping the search query to a segment of the generated index based on the result set.

Abstract: Apparatus, methods, and computer-readable media facilitating efficiently scaling real-time indexing are disclosed herein. An example method includes generating a first plurality of source data objects based on source files having data received at the object storage system. The example method also includes generating one or more real-time manifest files based on the first plurality of source data objects. Additionally, the example method includes updating the index to include the one or more real-time manifest files. The example method also includes receiving a search query for at least one of the first plurality of source data objects and the second plurality of source data objects stored at the object storage system. Additionally, the example method includes generating a materialized view of a result set of the search query based on querying the index based on the search query, the manifest file, and the one or more real-time manifest files.

Abstract: Disclosed are system and methods for processing and storing data files, using a data edge file format. The data edge file separates information about what symbols are in a data file and information about the corresponding location of those symbols in the data file. The described technique for converting a source file comprising symbols into a data edge file includes: generating a locality file of symbol location from the source file to identify locations of the symbols in the source file, generating a symbol file to identify symbols in the source file, and then modifying the locality file of symbol location to associate each symbol from the symbol file with a location in the source file.

Abstract: Disclosed are system and methods for processing and storing data files, using a data edge file format. The data edge file separates information about what symbols are in a data file and information about the corresponding location of those symbols in the data file. The described technique for converting a source file comprising symbols into a data edge file includes: generating a locality file of symbol location from the source file to identify locations of the symbols in the source file, generating a symbol file to identify symbols in the source file, and then modifying the locality file of symbol location to associate each symbol from the symbol file with a location in the source file.

Abstract: Disclosed are system and methods for processing and storing data files, using a data edge file format. The data edge file separates information about what symbols are in a data file and information about the corresponding location of those symbols in the data file. The described technique converts a source file comprising symbols into a data edge index having a manifest portion, a symbol portion, and a locality portion. The symbol portion contains a sorted unique set of the symbols from the source file, and the locality portion contains a plurality of location values referencing the symbol portion. The technique includes normalizing the structured data from the source file by modifying the locality manifest portion of the data edge file to include a description of at least one nonexistent column empty locality value at a respective position within the locality file representing an omission of data at an associated the respective position in the source file.

Abstract: Apparatus, methods, and computer-readable media facilitating efficiently scaling result caching are disclosed herein. An example method includes generating an index based on a plurality of source data objects in an object storage system. The generated index comprises a manifest, at least one symbol file, and at least one locality file. The example method also includes receiving a search query for the plurality of source data objects stored in the object storage system, and querying the generated index based on the search query and a manifest root file of the manifest. Additionally, the example method includes generating a materialized view of a result set of the search query based on the querying of the generated index. The example method also includes storing a cached manifest file at the generated index, the cached manifest file mapping the search query to a segment of the generated index based on the result set.

Abstract: Disclosed are system and methods for processing and storing data files, using a data edge file format. The data edge file separates information about what symbols are in a data file and information about the corresponding location of those symbols in the data file. An index for the data files can be generated according to the data edge file format. Using the data edge index, a materialized view of a result set can be generated in response to a search query for the source data objects stored in object storage.