Abstract: Disclosed are system and methods for processing and storing data files, using a data edge file format. The data edge file format separates information about what symbols are in a data file and information about the corresponding location of those symbols in the data file. Examples convert a source file comprising symbols into a data edge index having a manifest portion, a symbol portion, and a locality portion. The symbol portion contains a sorted unique set of symbols from the source file, and the locality portion contains a plurality of location values referencing the symbol portion. Examples include normalizing structured data from the source file by modifying the locality manifest portion of the data edge file to include a description of at least one nonexistent column empty locality value at a respective position within the locality file representing an omission of data at an associated position in the source file.

Abstract: Apparatus, methods, and computer-readable media facilitating efficiently scaling result caching are disclosed herein. An example method includes generating an index based on a plurality of source data objects in an object storage system. The generated index comprises a manifest, at least one symbol file, and at least one locality file. The example method also includes receiving a search query for the plurality of source data objects stored in the object storage system, and querying the generated index based on the search query and a manifest root file of the manifest. Additionally, the example method includes generating a materialized view of a result set of the search query based on the querying of the generated index. The example method also includes storing a cached manifest file at the generated index, the cached manifest file mapping the search query to a segment of the generated index based on the result set.

Abstract: Apparatus, methods, and computer-readable media facilitating efficiently scaling real-time indexing are disclosed herein. An example method includes generating a first plurality of source data objects based on source files having data received at the object storage system. The example method also includes generating one or more real-time manifest files based on the first plurality of source data objects. Additionally, the example method includes updating the index to include the one or more real-time manifest files. The example method also includes receiving a search query for at least one of the first plurality of source data objects and the second plurality of source data objects stored at the object storage system. Additionally, the example method includes generating a materialized view of a result set of the search query based on querying the index based on the search query, the manifest file, and the one or more real-time manifest files.

Abstract: Apparatus, methods, and computer-readable media facilitating efficiently scaling result caching are disclosed herein. An example method includes generating an index based on a plurality of source data objects in an object storage system. The generated index comprises a manifest, at least one symbol file, and at least one locality file. The example method also includes receiving a search query for the plurality of source data objects stored in the object storage system, and querying the generated index based on the search query and a manifest root file of the manifest. Additionally, the example method includes generating a materialized view of a result set of the search query based on the querying of the generated index. The example method also includes storing a cached manifest file at the generated index, the cached manifest file mapping the search query to a segment of the generated index based on the result set.

Abstract: Disclosed are system and methods for processing and storing data files, using a data edge file format. The data edge file separates information about what symbols are in a data file and information about the corresponding location of those symbols in the data file. The described technique for converting a source file comprising symbols into a data edge file includes: generating a locality file of symbol location from the source file to identify locations of the symbols in the source file, generating a symbol file to identify symbols in the source file, and then modifying the locality file of symbol location to associate each symbol from the symbol file with a location in the source file.

Abstract: Disclosed are system and methods for processing and storing data files, using a data edge file format. The data edge file separates information about what symbols are in a data file and information about the corresponding location of those symbols in the data file. The described technique for converting a source file comprising symbols into a data edge file includes: generating a locality file of symbol location from the source file to identify locations of the symbols in the source file, generating a symbol file to identify symbols in the source file, and then modifying the locality file of symbol location to associate each symbol from the symbol file with a location in the source file.

Abstract: Disclosed are system and methods for processing and storing data files, using a data edge file format. The data edge file separates information about what symbols are in a data file and information about the corresponding location of those symbols in the data file. The described technique converts a source file comprising symbols into a data edge index having a manifest portion, a symbol portion, and a locality portion. The symbol portion contains a sorted unique set of the symbols from the source file, and the locality portion contains a plurality of location values referencing the symbol portion. The technique includes normalizing the structured data from the source file by modifying the locality manifest portion of the data edge file to include a description of at least one nonexistent column empty locality value at a respective position within the locality file representing an omission of data at an associated the respective position in the source file.

Abstract: Disclosed are system and methods for processing and storing data files, using a data edge file format. The data edge file separates information about what symbols are in a data file and information about the corresponding location of those symbols in the data file. The described technique converts a source file comprising symbols into a data edge index having a manifest portion, a symbol portion, and a locality portion. The symbol portion contains a sorted unique set of the symbols from the source file, and the locality portion contains a plurality of location values referencing the symbol portion. The technique includes normalizing the structured data from the source file by modifying the locality manifest portion of the data edge file to include a description of at least one nonexistent column empty locality value at a respective position within the locality file representing an omission of data at an associated the respective position in the source file.

Abstract: Apparatus, methods, and computer-readable media facilitating efficiently scaling result caching are disclosed herein. An example method includes generating an index based on a plurality of source data objects in an object storage system. The generated index comprises a manifest, at least one symbol file, and at least one locality file. The example method also includes receiving a search query for the plurality of source data objects stored in the object storage system, and querying the generated index based on the search query and a manifest root file of the manifest. Additionally, the example method includes generating a materialized view of a result set of the search query based on the querying of the generated index. The example method also includes storing a cached manifest file at the generated index, the cached manifest file mapping the search query to a segment of the generated index based on the result set.

Abstract: Disclosed are system and methods for processing and storing data files, using a data edge file format. The data edge file separates information about what symbols are in a data file and information about the corresponding location of those symbols in the data file. The described technique for converting a source file comprising symbols into a data edge file includes: generating a locality file of symbol location from the source file to identify locations of the symbols in the source file, generating a symbol file to identify symbols in the source file, and then modifying the locality file of symbol location to associate each symbol from the symbol file with a location in the source file.

Abstract: Disclosed are system and methods for processing and storing data files, using a data edge file format. The data edge file separates information about what symbols are in a data file and information about the corresponding location of those symbols in the data file. An index for the data files can be generated according to the data edge file format. Using the data edge index, a materialized view of a result set can be generated in response to a search query for the source data objects stored in object storage.

Abstract: Disclosed are system and methods for processing and storing data files, using a data edge file format. The data edge file separates information about what symbols are in a data file and information about the corresponding location of those symbols in the data file. The described technique converts a source file comprising symbols into a data edge index having a manifest portion, a symbol portion, and a locality portion. The symbol portion contains a sorted unique set of the symbols from the source file, and the locality portion contains a plurality of location values referencing the symbol portion. The technique includes normalizing the structured data from the source file by modifying the locality manifest portion of the data edge file to include a description of at least one nonexistent column empty locality value at a respective position within the locality file representing an omission of data at an associated the respective position in the source file.

Abstract: Disclosed are system and methods for processing and storing data files, using a data edge file format. The data edge file separates information about what symbols are in a data file and information about the corresponding location of those symbols in the data file. An index for the data files can be generated according to the data edge file format. Using the data edge index, a materialized view of a result set can be generated in response to a search query for the source data objects stored in object storage.

Abstract: Disclosed are system and methods for processing and storing data files, using a data edge file format. The data edge file separates information about what symbols are in a data file and information about the corresponding location of those symbols in the data file. The described technique for converting a source file comprising symbols into a data edge file includes: generating a locality file of symbol location from the source file to identify locations of the symbols in the source file, generating a symbol file to identify symbols in the source file, and then modifying the locality file of symbol location to associate each symbol from the symbol file with a location in the source file.

Abstract: A method, a system, and a computer program product for adaptively managing information in a database management system are provided. The system generates a model associated with the database management system. The system receives information for performing a database. The system determines, based on the generated model and the database transaction, whether to adjust an attribute associated with the database management system.

Abstract: A method, system, and computer program product for error detection and correction in append-only data-stores. A user or agent input is received. Error detection and correction involving at least one datastore is begun based on the user or agent input. An error detection and correction state is created and error detection and correction is performed. A file sealing indication and/or a clean shutdown indication is interpreted, the error-detection and correction is ended, and the state of the error-detection and correction is written to memory in an append-only manner.

Abstract: A remote customisation module for a logic engine, running upon a server remote from said logic engine, comprises a communication interface arranged to communicate with the logic engine and with an application running upon said server. The module also comprises a logic element arranged to receive an incoming request for data from the logic engine via the communication interface. The logic element is arranged to interrogate the application in order to extract data required to fulfill said request. The logic element is also arranged to process said extracted data to generate response data and to pass said response data to the logic engine.

Abstract: A remote customisation module for a logic engine, running upon a server remote from said logic engine, comprises a communication interface arranged to communicate with the logic engine and with an application running upon said server. The module also comprises a logic element arranged to receive an incoming request for data from the logic engine via the communication interface. The logic element is arranged to interrogate the application in order to extract data required to fulfil said request. The logic element is also arranged to process said extracted data to generate response data and to pass said response data to the logic engine.