Abstract: A request may be received to transform records in a data lake that match one or more query criteria. Data lake records that match the criteria may be retrieved. The retrieved records may include two data lake records associated with the same data lake record identifier. A transformed record may be generated based on time stamps associated with the retrieved records.

Abstract: A method of syncing data across multiple systems includes: receiving a plurality of calendar events from a plurality of independent calendar systems that use different calendar system specific schemas; aggregating the calendar events at a unifying communication system; converting the calendar events from a calendar system specific schema to a unifying communication system specific schema; storing the plurality of calendar events in the unifying communication system specific schema; converting a calendar event received from a non-master calendar system to the master calendar system specific schema; and sending the converted calendar event to the master calendar system; wherein copies of the received calendar events that are formatted according to the calendar system specific schema of the master calendar system are stored with the master calendar system, and copies of the calendar events that are formatted according to the unifying communication system specific schema are stored with the unifying communication

Abstract: A method of syncing data across multiple systems includes: receiving a plurality of calendar events from a plurality of independent calendar systems that use different calendar system specific schemas; aggregating the calendar events at a unifying communication system; converting the calendar events from a calendar system specific schema to a unifying communication system specific schema; storing the plurality of calendar events in the unifying communication system specific schema; converting a calendar event received from a non-master calendar system to the master calendar system specific schema; and sending the converted calendar event to the master calendar system; wherein copies of the received calendar events that are formatted according to the calendar system specific schema of the master calendar system are stored with the master calendar system, and copies of the calendar events that are formatted according to the unifying communication system specific schema are stored with the unifying communication

Abstract: A total number of messages in a set of messages that are processed by a computing node in a virtual computer cluster during a time interval is determined. The virtual computer cluster is deployed with a cloud computing service and includes the computing node and other computing nodes at an end time of the time interval. It is determined whether the total number of messages in the set of messages processed by the computing node is no less than a maximum per-interval message number threshold. In response to determining that the total number of messages is no less than a maximum per-interval message number threshold, the cloud computing service is caused to start an additional computing node in the virtual computer cluster after the time interval, wherein the additional computing node is used to process one or more subsequent messages to be processed after the time interval.

Abstract: Mechanisms and techniques for providing continuous integration and continuous deployment (CI/CD) for data lake applications are disclosed. Assembly of code for an app is managed with a CI platform to create a container within a shared environment within which the app runs. The container is isolated from other containers and bundles software, libraries and configuration files and can communicate with other containers through defined channels. The shared environment provides a platform for running the app. The app writes to one or more tables maintained in the shared environment. Assembly of subsequent versions of code for the app is managed by the CI platform. Deployment of the assembled subsequent version of the code to the container is managed by the CI platform. Integration tests are run on the deployed subsequent version of the code with the CI platform. The subsequent version of the code replaces the app in the shared environment when integration testing is complete.

Abstract: Techniques and mechanisms for incremental data ingestion are disclosed. Raw data is received from multiple disparate sources to be consumed in an environment for collecting unformatted raw data. The environment has at least a delta data table and a delta notification table. A write to an entry in the delta data table is attempted. Entries to the delta data table specify at least records indicating changes to objects in the environment. A write a corresponding entry to the delta notification table is attempted in response to a successful write attempt to the delta data table. The delta notification table entry includes information about delta data table entries for a specified period. At least one data consumer is notified that the delta data table has been modified.

Abstract: A method by a computing system implementing a node within a cluster of nodes to load balance among the cluster of nodes using retroactive refusal. Each of the nodes within the cluster of nodes is capable of processing work items. The method includes accepting a work item to be processed by the node, monitoring a resource utilization of the node, determining whether the resource utilization of the node exceeds a predetermined threshold, and retroactively refusing the work item in response to a determination that the resource utilization of the node exceeds the predetermined threshold, wherein the retroactive refusal causes the work item to be reassigned from the node to another node within the cluster of nodes.

Abstract: A request may be received to transform records in a data lake that match one or more query criteria. Data lake records that match the criteria may be retrieved. The retrieved records may include two data lake records associated with the same data lake record identifier. A transformed record may be generated based on time stamps associated with the retrieved records.

Abstract: A data lake partition identifier may be retrieved from a data lake update service. The data lake partition identifier may identify a partition of a data lake that stores a data lake records. Records may be retrieved using a query that includes one of the identifiers. Retrieved records may be transformed and transmitted to a downstream data service.

Abstract: Dynamic selection of data sources for streaming dynamic data is described. A data streaming service receives a dynamic selection of a data source after the data streaming service begins executing, the dynamic selection of the data source including an identifier of a host of a multi-tenant database and an identifier of a tenant of multiple tenants storing data in the multi-tenant database. The data streaming service streams dynamic data from the data source to a data destination.

Abstract: Embodiments regard transfer of data streaming services to provide continuous data flow. An embodiment of an apparatus includes a processor to process data for streaming to one or more organizations; and a memory to store data for streaming to the one or more organizations, wherein the apparatus is to provide a centralized work distribution service to track status of each of a plurality of data streams to the one or more organizations, and a plurality of nodes, each node being a virtual machine to stream one or more data streams to the one or more organizations, each node including a first daemon service to monitor connectivity of the node to dependency services for the node and, upon detecting a loss of connection to one or more of the dependency services, the node to discontinue ownership of the one or more data streams of the node and a second daemon service to poll the centralized work distribution service for data streams that are not assigned.

Abstract: A cloud computing service is used to deploy a virtual computer cluster. The virtual computer cluster is initialized with a set of one or more streaming nodes for processing first messages of one or more streaming jobs. It is determined whether the virtual computer cluster is to process second messages of a non-streaming job. In response to determining that the virtual computer cluster is to process the second messages of the non-streaming job, for example using control messages, the cloud computing service is caused to start a non-streaming node in the virtual computer cluster. The non-streaming node is tasked to process the second messages of the non-streaming job.

Abstract: Embodiments regard transfer of data streaming services to provide continuous data flow. An embodiment of an apparatus includes a processor to process data for streaming to one or more organizations; and a memory to store data for streaming to the one or more organizations, wherein the apparatus is to provide a centralized work distribution service to track status of each of a plurality of data streams to the one or more organizations, and a plurality of nodes, each node being a virtual machine to stream one or more data streams to the one or more organizations, each node including a first daemon service to monitor connectivity of the node to dependency services for the node and, upon detecting a loss of connection to one or more of the dependency services, the node to discontinue ownership of the one or more data streams of the node and a second daemon service to poll the centralized work distribution service for data streams that are not assigned.

Abstract: A method is described for collapsing a plurality of detected changes to data in a database into a reduced set of changes. The method includes detecting a plurality of changes to the database; determining that one or more changes in the plurality of changes are superfluous to demonstrating the current state of data in the database to an external system; determining, in response to determining that one or more changes in the plurality of changes are superfluous, a reduced set of changes that demonstrates the current state of data in the database based on the plurality of changes, wherein a number of changes in the plurality of changes is greater than a number of changes in the reduced set of changes; and determining a set of messages to generate based on the reduced set of changes, wherein the set of messages demonstrate the current state of data.

Abstract: In accordance with embodiments, there are provided mechanisms and methods for facilitating smart streaming of data between external systems and service providers in an on-demand services environment according to one embodiment. In one embodiment and by way of example, a method comprises splitting, by a streaming server computing device (“streaming device”), a file including messages associated with multiple tenants in a multitenant environment based on growth of the file over a time-period. The method may further include grouping, by the streaming device, the messages into multiple files based on intelligence fields associated with one or more of the messages and the tenants, where each file of the multiple files corresponds to a tenant of the multiple tenants. The method may further include uploading, by the streaming device, the multiple files to a database device associated with a service provider and coupled to the streaming server device.

Abstract: A total number of messages in a set of messages that are processed by a computing node in a virtual computer cluster during a time interval is determined. The virtual computer cluster is deployed with a cloud computing service and includes the computing node and other computing nodes at an end time of the time interval. It is determined whether the total number of messages in the set of messages processed by the computing node is no less than a maximum per-interval message number threshold. In response to determining that the total number of messages is no less than a maximum per-interval message number threshold, the cloud computing service is caused to start an additional computing node in the virtual computer cluster after the time interval, wherein the additional computing node is used to process one or more subsequent messages to be processed after the time interval.

Abstract: A cloud computing service is used to deploy a virtual computer cluster. The virtual computer cluster is initialized with a set of one or more streaming nodes for processing first messages of one or more streaming jobs. It is determined whether the virtual computer cluster is to process second messages of a non-streaming job. In response to determining that the virtual computer cluster is to process the second messages of the non-streaming job, for example using control messages, the cloud computing service is caused to start a non-streaming node in the virtual computer cluster. The non-streaming node is tasked to process the second messages of the non-streaming job.

Abstract: Techniques are disclosed relating to bulk data extraction systems. In some embodiments, a streaming server system may receive a first request, from a data storage system, that is sent prior to initiation of a bulk data extraction for a first group of users. In response to the first request, the streaming server system may receive, from the data storage system, a first notification message that includes a particular event identifier for a most recent data event generated at the data storage system. The streaming server system may receive, from the data storage system, those messages associated with the bulk data extraction for the first group. Subsequent to completion of the bulk data extraction, the streaming server system may send, to the data storage system, a request to subscribe to notification messages for data events associated with the first group.

Abstract: Techniques are disclosed relating to verifying the sequence of messages sent via data streaming systems. In some embodiments, a streaming server system may receive a plurality of messages associated with events at a data storage system that is accessible to various groups of users. In some embodiments, each of the plurality of messages may include an event identifier for an associated event and a group-specific sequence number. The streaming server system may detect, based on group-specific sequence numbers included in a set of messages, that a particular message for a particular group has not been received. The streaming server system may send, to the data storage system, a request for the particular message and, in response to the request, may receive the particular message from the data storage system.

Abstract: Embodiments regard transfer of data streaming services to provide continuous data flow. An embodiment of an apparatus includes a processor to process data for streaming to one or more organizations; and a memory to store data for streaming to the one or more organizations, wherein the apparatus is to provide a centralized work distribution service to track status of each of a plurality of data streams to the one or more organizations, and a plurality of nodes, each node being a virtual machine to stream one or more data streams to the one or more organizations, each node including a first daemon service to monitor connectivity of the node to dependency services for the node and, upon detecting a loss of connection to one or more of the dependency services, the node to discontinue ownership of the one or more data streams of the node and a second daemon service to poll the centralized work distribution service for data streams that are not assigned.