Abstract: A system and method for rotating private encryption keys for tenants of a database system has been developed. First, three separate public-private encryption keys are generated for a tenant of the database system. The three separate private encryption keys for the tenant are then stored in cloud-based storage. A defined cadence is created to rotate the private encryption keys for the tenant. The three separate private encryption keys for the tenant are defined as a a past private key, a present private key and a future private key. Next, the public encryption key is stored for the tenant in a global tenant directory. The present private key and the public encryption key are retrieved to encrypt and decrypt data from the tenant. The three separate private encryption keys are rotated at the defined cadence, where the past private key is discarded, the present private key becomes a new past private key, the future private key becomes a new present private key, and a new future private key is generated.

Abstract: Techniques and mechanisms for ingesting data through an atomic transaction are disclosed. Raw data is received from multiple disparate sources to be consumed in an environment that does not support atomic write operations to data consumers. The environment has at least a data table and a notification table. A write to an entry in the data table having an associated version is attempted. The data table entry corresponds to the data to be consumed. A write to a corresponding entry to the notification table is attempted in response to a successful write attempt to the data table. The notification table entry includes information about the corresponding data table entry. The version associated with the data table is modified in response to successful writes of both the data table entry and the notification table entry. At least one data consumer is notified that the data table version has been modified.

Abstract: Described is a method and system for verifying the end-to-end distribution of messages within an on-demand services platform. To provide the ability to monitor and trace such messages, the system may include a specialized and queryable datastore (or database) that allows the system to track the distribution and acknowledgement of messages throughout the services platform. Accordingly, a specialized verification process may be initiated to query the database to verify the receipt of a message has been acknowledged by various components within the system. In addition, the verification process may automatically retry the distribution the messages to ensure particular downstream components have received the message. Accordingly, the system alleviates the need to manual re-crawl various message source to ensure the end-to-end distribution of such messages.

Abstract: Various techniques and mechanisms for sharing remote resources among a trusted group are disclosed. A credential management agent utilizes a resource credential for a first user to access a secure resource corresponding to the first user for a second user by at least validating a second user and validating a consent of the first user to allow the second user to access the secure resource using the resource credential for the first user. The secure resource resides on a remote server system accessible via one or more application program interfaces (APIs). A platform management agent provides an interface for shared resource-agnostic credential sharing. The platform management agent validates credentials for the second user as belonging to a trusted group and forwards a request for access to the secure resource for the second user to the credential management agent.

Abstract: Techniques and mechanisms to manage deletions from data tables are disclosed. A request to delete data from at least one data table in an environment having tables storing data from multiple disparate sources is received. The environment can also have a delete request status table and a notification table. Processing of the delete request is managed utilizing a multi-stage workflow where stages of the multistage workflow are tracked by updating entries to the delete request status table. Completion of the delete request is verified by checking at least one entry in the delete request status table corresponding to the delete request. A corresponding entry is written to the notification table in response to a successful verified completion of the delete request.

Abstract: Managing mutations in a data lake environment. A mutation request to cause write operations that modify data objects or structures within an environment for collecting unformatted raw data is received. The environment has at least a data table and a notification table. An entry is written to the data table with a streaming job configured to receive and process the mutation request. Entries to the data table specify at least records indicating changes to objects in the environment based on ingestion processing for the environment for collecting unformatted raw data and based on the mutation request. A corresponding entry is written to the notification table in response to a successful write attempt to the data table. The notification table entry has information about data table entries for a specified period. At least one data consumer is notified that the data table has been modified.

Abstract: Architectures and techniques to provide an extract-once framework for data ingestion into a data lake. A data consumption job to ingest data to multiple tables within a data collection platform is started. Checkpoint metadata corresponding to the data consumption job is retrieved from a checkpoint metadata store. A subset of processes from the data consumption job are performed. Checkpoint metadata is updated in response to completion of the subset of processes. A subsequent subset of processes from the data consumption job is performed. Checkpoint metadata is updated in response to completion of each of the at least one subsequent subset of processes from the data consumption job. Batch metadata is updated in response to completion of the data consumption job.

Abstract: Managing mutations in a data lake environment. A mutation request to cause write operations that modify data objects or structures within an environment for collecting unformatted raw data is received. The environment has at least a data table and a notification table. An entry is written to the data table with a streaming job configured to receive and process the mutation request. Entries to the data table specify at least records indicating changes to objects in the environment based on ingestion processing for the environment for collecting unformatted raw data and based on the mutation request. A corresponding entry is written to the notification table in response to a successful write attempt to the data table. The notification table entry has information about data table entries for a specified period. At least one data consumer is notified that the data table has been modified.

Abstract: Techniques and mechanisms for incremental data ingestion are disclosed. Raw data is received from multiple disparate sources to be consumed in an environment for collecting unformatted raw data. The environment has at least a delta data table and a delta notification table. A write to an entry in the delta data table is attempted. Entries to the delta data table specify at least records indicating changes to objects in the environment. A write a corresponding entry to the delta notification table is attempted in response to a successful write attempt to the delta data table. The delta notification table entry includes information about delta data table entries for a specified period. At least one data consumer is notified that the delta data table has been modified.

Abstract: Techniques and mechanisms for ingesting data through an atomic transaction are disclosed. Raw data is received from multiple disparate sources to be consumed in an environment that does not support atomic write operations to data consumers. The environment has at least a data table and a notification table. A write to an entry in the data table having an associated version is attempted. The data table entry corresponds to the data to be consumed. A write to a corresponding entry to the notification table is attempted in response to a successful write attempt to the data table. The notification table entry includes information about the corresponding data table entry. The version associated with the data table is modified in response to successful writes of both the data table entry and the notification table entry. At least one data consumer is notified that the data table version has been modified.

Abstract: Described is a system and method for compacting data into customized (e.g. optimal) file sizes for processing by computing resources. The mechanism may leverage various computing resources such as a cluster computing frameworks combined with a stream processing platform to efficiently process the activity data. For example, activity data of an organization may be processed by a set of jobs (or sub-jobs) as part of a data stream by a set of distributed computing resources. In order to efficiently process such data, the mechanism may compact the data into customized (e.g. optimal) file sizes. For example, the customized file sizes may provide an optimal (or near optimal) amount of data to be processed by each job, for example, to improve performance.

Abstract: Described is a method and system for verifying the end-to-end distribution of messages within an on-demand services platform. To provide the ability to monitor and trace such messages, the system may include a specialized and queryable datastore (or database) that allows the system to track the distribution and acknowledgement of messages throughout the services platform. Accordingly, a specialized verification process may be initiated to query the database to verify the receipt of a message has been acknowledged by various components within the system. In addition, the verification process may automatically retry the distribution the messages to ensure particular downstream components have received the message. Accordingly, the system alleviates the need to manual re-crawl various message source to ensure the end-to-end distribution of such messages.

Abstract: Described is a system and method for compacting data into customized (e.g. optimal) file sizes for processing by computing resources. The mechanism may leverage various computing resources such as a cluster computing frameworks combined with a stream processing platform to efficiently process the activity data. For example, activity data of an organization may be processed by a set of jobs (or sub-jobs) as part of a data stream by a set of distributed computing resources. In order to efficiently process such data, the mechanism may compact the data into customized (e.g. optimal) file sizes. For example, the customized file sizes may provide an optimal (or near optimal) amount of data to be processed by each job, for example, to improve performance.