Abstract: Techniques and architectures for data ingestion in an environment having a distributed data storage system. A request to import data into the environment. The data to be imported from an external source through an application programming interface (API). The request is analyzed to determine if the request corresponds to a request to import a large data load. The data is staged if the request is for the large data load. Data management jobs are created to cause the data to be stored in one or more nodes of the distributed data storage system. The data management jobs are transmitted to corresponding nodes in the distributed data storage system. The jobs are executed asynchronously to cause the data to be stored in the nodes.

Abstract: Disclosed are examples of systems, apparatus, methods and computer program products for providing run-time querying of multi-tenant non-relational database objects. A database system maintains a multi-tenant non-relational database associated with a number of enterprises, a number of records, and a number of data objects for each of the enterprises, each data object having at least one composite key field. A request is received during runtime to query data related to a data object, and the system determines that the data object is defined to be used in associated with a non-relational database. The system then retrieves a metadata model of the data object, determines that the request includes at least one composite key field for the data object, and processes the requery to query data in synchronous fashion. Finally, a query response is provided to the user.

Abstract: Systems, methods, and computer-readable media for asynchronous (async) querying are described. In embodiments, a system may convert a user-issued query into a distributed execution instruction set (DEIS), and issue the DEIS to data stores that may have various database structures. The system may obtain database objects from the data stores, and store a result set indicating the obtained database objects in a location specified by the user-issued query. The system may also provide mechanisms to allow users to view progress of their async query jobs and/or cancel ongoing async query jobs. Other embodiments may be described and/or claimed.

Abstract: A method is provided for processing a query. In some embodiments, the method includes creating a historical schema defining an ordered set of fields based on a current schema of an object and a field modification of the current schema. The method includes inserting, based on the field modification, a new field at an end of the historical schema and archiving records into a second storage device. The method includes receiving a query, which includes a command specifying a subset of fields and which indicates a non-current field. The method includes searching in the second storage device for archive records having a second schema including the subset of fields and the non-current field. The first and second schemas follow the field order in accordance with the historical schema. The method further includes accessing the subset of fields and the non-current field in accordance with the command.

Abstract: Systems, apparatus, methods and computer program products for deleting data of an object within a multi-tenant database via an efficient delete operation are described herein. The new delete operation (or method) may be configured to identify one or more records to be deleted within a data structure such as an object. In order to ensure that the operation is efficient, the mechanism may determine characteristics of a data store and determine whether the information provided within the data structure provides an efficient identification of the data to be deleted. Upon a successful validation, an initial delete request may be translated to an appropriate delete operation for the underlying database.

Abstract: A method is provided for processing a query. In some embodiments, the method includes creating a historical schema defining an ordered set of fields based on a current schema of an object and a field modification of the current schema. The method includes inserting, based on the field modification, a new field at an end of the historical schema and archiving records into a second storage device. The method includes receiving a query, which includes a command specifying a subset of fields and which indicates a non-current field. The method includes searching in the second storage device for archive records having a second schema including the subset of fields and the non-current field. The first and second schemas follow the field order in accordance with the historical schema. The method further includes accessing the subset of fields and the non-current field in accordance with the command.

Abstract: Techniques and architectures for data ingestion in an environment having a distributed data storage system. A request to import data into the environment. The data to be imported from an external source through an application programming interface (API). The request is analyzed to determine if the request corresponds to a request to import a large data load. The data is staged if the request is for the large data load. Data management jobs are created to cause the data to be stored in one or more nodes of the distributed data storage system. The data management jobs are transmitted to corresponding nodes in the distributed data storage system. The jobs are executed asynchronously to cause the data to be stored in the nodes.

Abstract: Disclosed are examples of systems, apparatus, methods and computer program products for deleting data of an object within a multi-tenant database. Described is a mechanism for performing operations such as an efficient delete operation by introducing a new delete operation (or method) that is configured to allow a data structure such as an object to identify one or more records to be deleted. In order to ensure that the operation is efficient, the mechanism may determine characteristics of a data store and determine whether the information provided within the data structure provides an efficient identification of the data to be deleted. Upon a successful validation, an initial delete request may be translated to an appropriate delete operation for the underlying database.

Abstract: A method is provided for processing a query. In some embodiments, the method includes creating a historical schema defining an ordered set of fields based on a current schema of an object and a field modification of the current schema. The method includes inserting, based on the field modification, a new field at an end of the historical schema and archiving records into a second storage device. The method includes receiving a query, which includes a command specifying a subset of fields and which indicates a non-current field. The method includes searching in the second storage device for archive records having a second schema including the subset of fields and the non-current field. The first and second schemas follow the field order in accordance with the historical schema. The method further includes accessing the subset of fields and the non-current field in accordance with the command.

Abstract: Disclosed are examples of systems, apparatus, methods and computer program products for deleting data of an object within a multi-tenant database. Described is a mechanism for performing operations such as an efficient delete operation by introducing a new delete operation (or method) that is configured to allow a data structure such as an object to identify one or more records to be deleted. In order to ensure that the operation is efficient, the mechanism may determine characteristics of a data store and determine whether the information provided within the data structure provides an efficient identification of the data to be deleted. Upon a successful validation, an initial delete request may be translated to an appropriate delete operation for the underlying database.

Abstract: Techniques and architectures for data ingestion in an environment having a distributed data storage system. A request to import data into the environment. The data to be imported from an external source through an application programming interface (API). The request is analyzed to determine if the request corresponds to a request to import a large data load. The data is staged if the request is for the large data load. Data management jobs are created to cause the data to be stored in one or more nodes of the distributed data storage system. The data management jobs are transmitted to corresponding nodes in the distributed data storage system. The jobs are executed asynchronously to cause the data to be stored in the nodes.

Abstract: Computer-implemented methods and systems are provided for writing events to a data store. An application server generates events, the data store that stores the events, and a temporary events file storage system (TEFSS) temporarily stores groups of events as events files. When events are unable to be written directly to the data store, an indirect events writer is invoked that includes event capture threads each being configured to generate a particular events file, and write it to the TEFSS. Each events file includes a plurality of events flushed from an in-memory buffer service. An events file uploader service reads the events file(s) from the TEFSS, and then writes the events from each of the events files to the data store.

Abstract: Disclosed are examples of systems, apparatus, methods and computer program products for deleting data of an object within a multi-tenant database. Described is a mechanism for performing operations such as an efficient delete operation by introducing a new delete operation (or method) that is configured to allow a data structure such as an object to identify one or more records to be deleted. In order to ensure that the operation is efficient, the mechanism may perform a validation of the values provided to the delete operation based on the type of data store used for storing the records. For example, the validation may ensure that the values provided to the method correspond to a composite primary key when records are stored in a non-relational data store. Upon a successful validation, an initial delete request may be translated to an appropriate delete operation for the underlying database.

Abstract: A method and apparatus for a mechanism of disaster recovery and instance refresh in an event recordation system are described. A first request for events associated with a first topic is received. A first set of events associated with the first topic are retrieved from a primary event recordation system. Each event from the first set of events includes a first commit identifier indicating the order with which each event is stored in the primary event recordation system. For each event from the first set of events a first replay identifier is determined based on the first commit identifier and a first value of a low commit identifier. The first replay identifier is exposed to event consumers for identifying the event. In response to the first request each event from the first set of events is transmitted with the replay identifier.

Abstract: A method is provided for processing a query. In some embodiments, the method includes creating a historical schema defining an ordered set of fields based on a current schema of an object and a field modification of the current schema. The method includes inserting, based on the field modification, a new field at an end of the historical schema and archiving records into a second storage device. The method includes receiving a query, which includes a command specifying a subset of fields and which indicates a non-current field. The method includes searching in the second storage device for archive records having a second schema including the subset of fields and the non-current field. The first and second schemas follow the field order in accordance with the historical schema. The method further includes accessing the subset of fields and the non-current field in accordance with the command.

Abstract: A database system may determine that a database is unavailable to generate an identifier for an event or that events may not be written to the database. As a result, the database system may write the event to a low-latency, supplementary file system. The database system may determine that the database is available at a later time, and read the event from the supplementary file system, store the event in a memory component, and then delete the event from the supplementary file system. The database system may then access the database to generate an identifier for the event, write the event to the database, and then delete the event from the memory component.

Abstract: In an example, a computing system is configured to detect data to temporarily store in a group of buffers using an in-memory buffer service; correlate, to the detected data, one or more identifiers of a plurality of identifiers based on a characteristic of the detected data, wherein a first identifier of the plurality corresponds to a first buffer type and a second different identifier of the plurality corresponds to a second buffer type; in response to the data correlated to a single identifier of the identifiers, create a first data object and place the first data object in one of the buffers of the corresponding buffer type; and in response to the data correlated to more than one of the identifiers, create a second data object for each one of the identifiers and place the second data objects in ones of the buffers of the corresponding buffer types, respectively.

Abstract: A system derives a first schema that is specific to a first log entry type associated with a log code, a second schema that is specific to a second log entry type associated with the log code, and a common schema for the first log entry type and the second log entry type. The system stores the first schema and the common schema in a container for the first log entry type, and the second schema and the common schema in a container for the second log entry type. The system identifies a schema identifier in a log entry corresponding to a system user event. The schema identifier corresponds to a schema in the container for the first log entry type or the container for the second log entry type. The system identifies log data by applying the corresponding schema to the log entry, and outputs the log data.

Abstract: A method and apparatus for a mechanism of disaster recovery and instance refresh in an event recordation system are described. A first request for events associated with a first topic is received. A first set of events associated with the first topic are retrieved from a primary event recordation system. Each event from the first set of events includes a first commit identifier indicating the order with which each event is stored in the primary event recordation system. For each event from the first set of events a first replay identifier is determined based on the first commit identifier and a first value of a low commit identifier. The first replay identifier is exposed to event consumers for identifying the event. In response to the first request each event from the first set of events is transmitted with the replay identifier.

Abstract: Disclosed are some examples of database systems, methods, and computer program products for processing log files. In some implementations, a server of a database system accesses a metadata file indicating algorithms that can be applied to data of log files. The server generates customer-facing log files using the log file and metadata file. The customer-facing log files include new data derived from using the algorithms and the data of the log files.