Abstract: Techniques are disclosed for utilizing directed acyclic graphs for deployment instructions. A computer-implemented method can include various operations. Instructions may be executed by a computing device to perform parses of configuration data associated with a deployment. The computing device may cause a first directed acyclic graph (DAG) to be generated, the first DAG being utilized for deploying a first resource based on the parses. A second DAG may be generated for deploying execution targets based on the parses, the second DAG specifying dependencies between execution targets of the deployment. The computing device may generate a linked list data structure based on the parses and may deploy the computing system by traversal of the linked list data structure.

Abstract: Techniques for implementing an infrastructure orchestration service are described. In some examples, a declarative provisioner of the infrastructure orchestration service receives instructions for deployment of a resource. The declarative provisioner identifies that the deployment of the resource is a long-running task stores state information corresponding to the deployment of the resource. In certain embodiments, upon identifying that the deployment of the resource is a long-running task, the declarative provisioner pauses its execution of the long-running task. Responsive to a trigger received from the infrastructure orchestration service, the declarative provisioner resumes execution of the deployment of the resource using the state information and transmits deployment information corresponding to the deployment of the resource to the infrastructure orchestration service.

Abstract: Techniques for implementing an infrastructure orchestration service are described. A scheduler can receive a configuration file, which can include a first release identifier, for a first deployment of resources at an execution target. The resources can be deployed at the execution target according to the file. A current state of the resources can be stored. The scheduler can receive a second version of the file, which can include a second release identifier, for a new deployment at the execution target. At least one worker node can execute a plugin to compare the first identifier to the second identifier. If the first identifier is different than the second identifier, the plugin can compare a current state of the resources with a desired state according to the second identifier. If the desired state is different than the current state, the resources are deployed at the execution target according to the second identifier.

Abstract: Techniques for implementing an infrastructure orchestration service are described. A configuration file for a deployment to a first execution target and a second execution target can be received. A first safety plan can be generated for the first execution target that comprises a first list of resources and operations associated with deployment at the first execution target. Approval of the first safety plan can be received. A second safety plan can be generated for the second execution target that comprises a second list of resources and operations associated with deployment at the second execution target. A determination can be made whether the second safety plan is a subset of the first safety plan. If the determination is that the second safety plan is a subset of the first safety plan, the second safety plan can automatically be approved and transmitted to the second execution target for deployment.

Abstract: Techniques for resegmenting a partition in a distributed stream-processing platform are provided. The techniques include receiving a trigger to move a partition of the distributed stream-processing platform from a first broker on a first set of physical resources to a second broker on a second a set of physical resources. In response to the trigger, the partition is allocated on the second broker, and the first broker is configured to redirect, to the second broker, requests for new messages after a last offset in the partition without replicating older messages before the last offset to the second broker. Idempotent produce metadata for the partition from the first broker is then merged into the second broker. Finally, metadata for processing requests for the partition is updated to include the second broker.

Abstract: Techniques for resegmenting a partition in a distributed stream-processing platform are provided. The techniques include receiving a trigger to move a partition of the distributed stream-processing platform from a first broker on a first set of physical resources to a second broker on a second a set of physical resources. In response to the trigger, the partition is allocated on the second broker, and the first broker is configured to redirect, to the second broker, requests for new messages after a last offset in the partition without replicating older messages before the last offset to the second broker. Idempotent produce metadata for the partition from the first broker is then merged into the second broker. Finally, metadata for processing requests for the partition is updated to include the second broker.

Abstract: Techniques for resegmenting a partition in a distributed stream-processing platform are provided. The techniques include receiving a trigger to move a partition of the distributed stream-processing platform from a first broker on a first set of physical resources to a second broker on a second a set of physical resources. In response to the trigger, the partition is allocated on the second broker, and the first broker is configured to redirect, to the second broker, requests for new messages after a last offset in the partition without replicating older messages before the last offset to the second broker. Idempotent produce metadata for the partition from the first broker is then merged into the second broker. Finally, metadata for processing requests for the partition is updated to include the second broker.

Abstract: A replication chain comprising one or more replication nodes of a multi-tenant stream management system is assigned to store data records of a partition of a particular data stream. A data record of the partition is received at a selected replication node of the replication chain. In a sequential order, a respective replica of the data record is stored at each replication node of the chain. An acknowledgement of a successful storage of the data record is provided after the replications are completed.

Abstract: Techniques for resegmenting a partition in a distributed stream-processing platform are provided. The techniques include receiving a trigger to move a partition of the distributed stream-processing platform from a first broker on a first set of physical resources to a second broker on a second a set of physical resources. In response to the trigger, the partition is allocated on the second broker, and the first broker is configured to redirect, to the second broker, requests for new messages after a last offset in the partition without replicating older messages before the last offset to the second broker. Idempotent produce metadata for the partition from the first broker is then merged into the second broker. Finally, metadata for processing requests for the partition is updated to include the second broker.

Abstract: A replication chain comprising one or more replication nodes of a multi-tenant stream management system is assigned to store data records of a partition of a particular data stream. A data record of the partition is received at a selected replication node of the replication chain. In a sequential order, a respective replica of the data record is stored at each replication node of the chain. An acknowledgement of a successful storage of the data record is provided after the replications are completed.

Abstract: A replication chain comprising one or more replication nodes of a multi-tenant stream management system is assigned to store data records of a partition of a particular data stream. A data record of the partition is received at a selected replication node of the replication chain. In a sequential order, a respective replica of the data record is stored at each replication node of the chain. An acknowledgement of a successful storage of the data record is provided after the replications are completed.