Abstract: A method of scaling open-source container orchestration platforms is provided. The open-source container orchestration platform has a first plane that is associated with regional agents each having an object group is provided. A second plane is associated with each object group where the object groups have a pod group. Each pod group has an availability zone divided into sub-availability zones where the availability zones are associated with the second plane. Each of the availability zones has a first sub-availability zone configured to substitute functionality of a second sub-availability zone of the availability zones when the second sub-availability zone becomes non-functional. Moreover, each of the availability zones can be further subdivided to include additional sub-availability zones when demand increases thereby scaling an open-source container orchestration platform implementing the availability zones.

Abstract: Systems and methods are directed to seamless failback after an unplanned failover. The method involves executing a truncate-and-restore command on a primary topic of a primary cluster, performing checks on a secondary topic to ensure it is a mirror topic in a stopped state with valid identifiers and offsets, and transitioning the primary topic to immutable state. The method further includes comparing sequence numbers to ensure safe truncation, truncating partitions to match the secondary topic's log end offsets, and clamping consumer group offsets. The primary topic is then converted to a mirror state, enabling active mirroring. A reverse command is then executed to complete the failback process, restoring the primary topic to a writable state.

Abstract: Systems and methods are directed to secure hybrid data transfer through connection and request direction decoupling. Initially, a controller server in a source on-premises cluster establishes a persistent connection to a controller server in a destination cloud cluster. The connection is then reversed, which enables destination servers to request additional connections between any pair of source and destination servers dynamically from the source cluster. Each of these connections is established by a source server, which authenticates to a cloud (destination) server and then requests to reverse the connection. The reversed connections can be used to transfer data securely between the on-premises and cloud servers. The source server can be a broker in a source cluster located on-premises and the destination server can be a broker in a destination cluster in the cloud.

Abstract: Systems and methods are directed to transforming data into a distributed table format. The system accesses new data for a topic. One or more materializers generate data files from the new data, whereby each of the materializers can process a specific range of offsets within a partition of the topic. The materializers also generate materializer committables corresponding to the data files. A maintainer performs one or more maintenance operations on previously committed files and generates a maintainer committable for each maintenance operation performed. The maintenance operations can include merging previously committed smaller files into a larger file and deleting obsolete files. Subsequently, a committer scheduler schedules the maintainer and materializer committables for a commit process. Scheduling includes prioritizing which committables are applied first and ordering materializer committables based on their offsets.

Abstract: Systems and methods provide low-latency, ordered delivery of database change events to a watch client. In response to a database (DB) create, update, or delete (CUD) request, the system writes the request to a database. The database usually provides a change data capture stream of events to a publish/subscribe (pub/sub) system. This results in higher change event delivery latency and is referred to as a slow path. The system leverages a dual write approach, where the system asynchronously writes a corresponding change event directly to a fast path pub/sub system. A watch server consumes change events from the slow and fast path pub/sub systems and merges and orders the change events using a first-in first-out cache. An ordered event stream can be transmitted to the watch client with lower latency. The ordered event stream includes one or more batches, and each batch comprises a most recent change event version for each object that was changed.

Abstract: When a resource is being deployed to the cloud, there are many possible locations for the deployment. When a resource is created, it is placed under a parent resource. As discussed herein, a system for applying constraints to resources is used. A constraint is a type of metadata that helps manage, identify, organize, and filter resources. Defined constraints are a set of constraints that, for a particular resource, are either set or not set. Custom constraints are key/value pairs that have predefined keys, but variable values. New constraints may be added to the system without modifying source code. When a resource creation request is received, constraints of the resource being created are checked against the constraints of the candidate parent resources. The resource is then deployed to a parent resource for which it does not violate any constraints.

Abstract: A cross-region data distribution service enables eventually consistent replication of data objects between geographical regions while maintaining data ordering guarantees. For data replication between two regions, the system leverages a data stream-based replication model where destination region leverages a watch API exposed by origin regions. The origin region watch API provides a resumable data stream for changes happening in the origin region that need to be replicated to destination. The system exposes distribution ability through distribution policies which are global objects that encode the source datasets and desired distribution topology. In order to solve data fanout scalability challenges where same data needs to be replicated to many regions, system leverages caching at origin watch servers. The system maintains data ordering by enforcing ordered replication stream provided by watch servers and using version based conditional optimistic concurrency control writes in destination servers.

Abstract: A method of synchronizing container image lists is provided. A first container image list having first container images is accessed from a remote source. The first container images listed in the first container image list include a first tag and a first digest. A second container image list having second container images is accessed from a cloud-computing system registry. The second container images listed in the second container image list include a second tag and a second digest. Container images in the accessed container image lists are compared with each other. The comparison includes comparing the first tag with the second tag and, based on the tag comparison, the first digest with the second digest. The first container image list is synchronized with the second container image list based on the comparison of the container images.

Abstract: A system includes at least one processor and at least one memory including instructions that, when executed by the at least one processor, cause the at least one processor to perform operations. The operations include providing a secure link to an auditable, machine-readable dataset to a client device of a user, the auditable, machine-readable dataset comprising data. At least one policy constraint is provided to at least a portion of the data in the dataset. In response to detecting activation of the secure link at the client device, one or both of the user and the client device is authenticated based on the policy constraint. Streaming access is provided to the auditable, machine-readable dataset in real-time.

Abstract: A method of adjusting a number of virtual machines in a data plane is provided. A number of virtual machines in the data plane each having a data plane proxy is provisioned. The virtual machines provide data routing for a first number of operational pods in a deployment plane associated with the data plane. A status of the deployment plane is monitored. The status reflects the deployment plane has a second number of operational pods different from the first number of operational pods. The first number of operational pods is compared to the second number of operational pods. Based on the comparison, the number of virtual machines in the data plane is adjusted.

Abstract: A method of synchronizing container image lists is provided. A first container image list having first container images is accessed from a remote source. The first container images listed in the first container image list include a first tag and a first digest. A second container image list having second container images is accessed from a cloud-computing system registry. The second container images listed in the second container image list include a second tag and a second digest. Container images in the accessed container image lists are compared with each other. The comparison includes comparing the first tag with the second tag and, based on the tag comparison, the first digest with the second digest. The first container image list is synchronized with the second container image list based on the comparison of the container images.

Abstract: A request to configure a connect cluster including one or more connectors for a cloud computing system is received. The request include a desired connector state. A connector specification file is automatically generated based on the desired connector state for the connect cluster via a declarative application programming interface (API). Application resources associated with the connect cluster are automatically configured based on the specification file.

Abstract: A deployment request is received for a plurality of clusters within a plurality cloud computing environments. The deployment request relates to one or more operational actions for computing devices associated with the plurality of clusters. The plurality of clusters is associated with a single application programming interface (API). A set of standards are received for the plurality of clusters from the API. The plurality of clusters is configured according to the set of standards to create a configured plurality of clusters. The configured plurality of clusters are deployed within the plurality of cloud computing environments.

Abstract: A method for upgrading computing devices in a distributed computing environment is provided. The method includes receiving first and second rollout requests for a cluster relating to operational actions for computing devices associated with the cluster. A rollout status for the cluster is determined and the rollout requests are simultaneously performed on the cluster where the operational actions are simultaneously performed on the computing devices associated with the cluster. In addition, a rollout status is updated based on simultaneously performing the rollout requests for the cluster.

Abstract: A system includes at least one processor and at least one memory including instructions that, when executed by the at least one processor, cause the at least one processor to perform operations. The operations include providing a secure link to an auditable, machine-readable dataset to a client device of a user, the auditable, machine-readable dataset comprising data. At least one policy constraint is provided to at least a portion of the data in the dataset. In response to detecting activation of the secure link at the client device, one or both of the user and the client device is authenticated based on the policy constraint. Streaming access is provided to the auditable, machine-readable dataset in real-time.

Abstract: A method of synchronizing container image lists is provided. A first container image list having first container images is accessed from a remote source. The first container images listed in the first container image list include a first tag and a first digest. A second container image list having second container images is accessed from a cloud-computing system registry. The second container images listed in the second container image list include a second tag and a second digest. Container images in the accessed container image lists are compared with each other. The comparison includes comparing the first tag with the second tag and, based on the tag comparison, the first digest with the second digest. The first container image list is synchronized with the second container image list based on the comparison of the container images.

Abstract: A method of adjusting a number of virtual machines in a data plane is provided. A number of virtual machines in the data plane each having a data plane proxy is provisioned. The virtual machines provide data routing for a first number of operational pods in a deployment plane associated with the data plane. A status of the deployment plane is monitored. The status reflects the deployment plane has a second number of operational pods different from the first number of operational pods. The first number of operational pods is compared to the second number of operational pods. Based on the comparison, the number of virtual machines in the data plane is adjusted.

Abstract: A stream verification system for a distributed message queue system with metric collectors on each producer and consumer. A producer time stamp allows correlation of sent and received messages. Verification reports are organized by a message topic. A cumulative checksum allows detection of missing or corrupted messages. Verification messages are used to determine if a zero message report means no messages were sent, or rather that the messages weren't received.

Abstract: A stream verification system for a distributed message queue system with metric collectors on each producer and consumer. A producer time stamp allows correlation of sent and received messages. Verification reports are organized by a message topic. A cumulative checksum allows detection of missing or corrupted messages. Verification messages are used to determine if a zero message report means no messages were sent, or rather that the messages weren't received.