Abstract: A source worker node at a source site fetches a task from a message queue. The task specifies replicating a first object at the source site to a destination site. A request for a connection is issued from the source worker node to the destination site. The request is received by a load balancer at the destination site and assigned to a destination worker node. A connection is established between the source and destination worker nodes. A determination is made that the destination site does not include an object that is the same as the first object. Upon the determination, a deduplication is performed between the source and destination worker nodes of segments into which the first object has been divided. Deduplicated segments of the first object are transmitted from the source worker node to the destination worker node for storage at the destination site.

Abstract: A schedule is stored indicating a frequency of replication from source to destination sites. When a replication job is initiated, information identifying one or more objects at the source site to be replicated is copied into a snapshot without pausing user operations against the one or more objects. The snapshot is compared with a previous snapshot to generate replication tasks for the replication job. The replication tasks are placed onto a message queue at the source site, where a worker node at the source site retrieves a replication task from the message queue and processes the replication task in conjunction with a worker node at the destination site.

Abstract: A deduplicated microservices-based storage system and method running in a cloud includes frontend and backend microservice nodes that process requests from a plurality of clients and interface with a master node and one or more worker nodes connected as a network for deduplicating, storing and retrieving data from object storage. A custom metric compares demand on system resources to thresholds and dynamically scales resources to add or eliminate resources according to demand. The master node partitions the range of values identifying data similarity groups across the worker nodes, reassigns ranges of values using consistent hashing as worker nodes are scaled, and maintains a mapping of ranges of values to worker IP addresses.

Abstract: A controller at a source site generates a set of tasks associated with a replication job. Each task includes one or more of copying an object from the source to destination site, or deleting an object from the destination site. The tasks are placed onto a message queue at the source site. Source worker nodes at the source site retrieve the tasks from the source site message queue for processing in conjunction with destination worker nodes at the destination site. A destination worker node, upon receiving a task from a source worker nodes, places the task onto a message queue at the destination site for retrieval by a backend worker node that handles writing to an object store at the destination site.

Abstract: A source site includes a controller, a set of source worker nodes, and a message queue connected between the controller and source worker nodes. The message queue receives messages and stores the messages for retrieval. A destination site includes a set of destination worker nodes. Tasks are generated to replicate changes to objects at the source site to the destination site. The controller pushes messages corresponding to the tasks onto the message queue. A source worker node retrieves a message corresponding to a task from the message queue for processing in conjunction with a destination worker node. The message is indicated as having been retrieved from the message queue.

Abstract: A source site includes a controller, a set of source worker nodes, and a message queue connected between the controller and source worker nodes. A destination site includes a set of destination worker nodes. The controller identifies differences between a first snapshot created at the source site at a first time and a second snapshot created at a second time, after the first time. Based on the differences, a set of tasks are generated. The tasks include one or more of copying an object from the source to destination or deleting an object from the destination. The controller places the tasks onto the message queue. A first source worker node retrieves the first task and coordinates with a first destination worker node to perform the first task. A second source worker nodes retrieves the second task and coordinates with a second destination worker node to perform the second task.

Abstract: A controller at a source site generates a set of tasks associated with a replication job. Each task includes one or more of copying an object from the source to destination site, or deleting an object from the destination site. The tasks are placed onto a message queue at the source site. Source worker nodes at the source site retrieve the tasks from the source site message queue for processing in conjunction with destination worker nodes at the destination site. A destination worker node, upon receiving a task from a source worker nodes, places the task onto a message queue at the destination site for retrieval by a backend worker node that handles writing to an object store at the destination site.

Abstract: A schedule is stored indicating a frequency of replication from source to destination sites. When a replication job is initiated, information identifying one or more objects at the source site to be replicated is copied into a snapshot without pausing user operations against the one or more objects. The snapshot is compared with a previous snapshot to generate replication tasks for the replication job. The replication tasks are placed onto a message queue at the source site, where a worker node at the source site retrieves a replication task from the message queue and processes the replication task in conjunction with a worker node at the destination site.

Abstract: A controller at a source site generates a set of tasks associated with a replication job. Each task involves a source worker node from among a set of source worker nodes at the source site, a destination worker node from among a set of destination worker nodes at the destination site, and includes one or more of copying an object from the source to destination site, or deleting an object from the destination site. Status update messages concerning the tasks are received at a message queue connected between the controller and the set of source worker nodes. The status update messages are logged into a persistent key-value store. Upon a failure to complete the replication job, the key-value store is accessed to identify tasks that were and were not completed before the failure. The tasks that were not completed are resent to the source worker nodes.

Abstract: Write performance is accelerated in a microservices deduplication system having a master node and a worker node, where the worker node receives and writes updates to data similarity groups to a cache and to a primary write-ahead log. Periodically, the worker writes updated similarity groups from cache to object storage, and flushes the contents of the primary write-ahead log. During flushing of the primary write-ahead log, the worker writes similarity group updates to a secondary write-ahead log which is swapped for the primary write-ahead log and which becomes a new primary write-ahead log for updates during flushing and during a subsequent time period. The previous primary write-ahead log becomes the new secondary write-ahead log, and the log swapping is repeated during subsequent time periods.

Abstract: A deduplicated microservices-based storage system and method running in a cloud includes frontend and backend microservice nodes that process requests from a plurality of clients and interface with a master node and one or more worker nodes connected as a network for deduplicating, storing and retrieving data from object storage. A custom metric compares demand on system resources to thresholds and dynamically scales resources to add or eliminate resources according to demand. The master node partitions the range of values identifying data similarity groups across the worker nodes, reassigns ranges of values using consistent hashing as worker nodes are scaled, and maintains a mapping of ranges of values to worker IP addresses.

Abstract: In a deduplicated microservices-based storage system and method running in a cloud includes frontend and backend microservice nodes that process requests from a plurality of clients and interface with a master node and one or more worker nodes connected as a network for deduplicating, storing and retrieving data from object storage, a worker node receives queries and fingerprints for data segments of a data similarity group to be deduplicated, obtains and returns the similarity group from its cache or from object storage using a key. The system compares the received fingerprints to stored fingerprints to determine non-duplicates for deduplication. The worker node updates similarity groups in its cache and in a write-ahead log with a reference to a compression region of non-duplicate segments, and periodically writes modified objects from its cache to object storage.

Abstract: In a deduplicated microservices-based storage system and method running in a cloud includes frontend and backend microservice nodes that process requests from a plurality of clients and interface with a master node and one or more worker nodes connected as a network for deduplicating, storing and retrieving data from object storage, a worker node receives queries and fingerprints for data segments of a data similarity group to be deduplicated, obtains and returns the similarity group from its cache or from object storage using a key. The system compares the received fingerprints to stored fingerprints to determine non-duplicates for deduplication. The worker node updates similarity groups in its cache and in a write-ahead log with a reference to a compression region of non-duplicate segments, and periodically writes modified objects from its cache to object storage.

Abstract: A set of tasks, associated with a replication job, are generated for replicating from a source to destination site. An initial number of worker nodes are allocated to process the tasks. Each task involves a first type of worker node at the source site, a second type of worker node at the destination site, and includes one or more of copying an object from the source site to the destination site, or deleting an object from the destination site. The replication job is monitored. Based on the monitoring, a number of worker nodes is adjusted from the initial number to a new number, different from the initial number.

Abstract: A controller at a source site generates a set of tasks associated with a replication job. Each task involves a source worker node from among a set of source worker nodes at the source site, a destination worker node from among a set of destination worker nodes at the destination site, and includes one or more of copying an object from the source to destination site, or deleting an object from the destination site. Status update messages concerning the tasks are received at a message queue connected between the controller and the set of source worker nodes. The status update messages are logged into a persistent key-value store. Upon a failure to complete the replication job, the key-value store is accessed to identify tasks that were and were not completed before the failure. The tasks that were not completed are resent to the source worker nodes.

Abstract: A set of tasks, associated with a replication job, are generated for replicating from a source to destination site. An initial number of worker nodes are allocated to process the tasks. Each task involves a first type of worker node at the source site, a second type of worker node at the destination site, and includes one or more of copying an object from the source site to the destination site, or deleting an object from the destination site. The replication job is monitored. Based on the monitoring, a number of worker nodes is adjusted from the initial number to a new number, different from the initial number.

Abstract: A source site includes a controller, a set of source worker nodes, and a message queue connected between the controller and source worker nodes. A destination site includes a set of destination worker nodes. The controller identifies differences between a first snapshot created at the source site at a first time and a second snapshot created at a second time, after the first time. Based on the differences, a set of tasks are generated. The tasks include one or more of copying an object from the source to destination or deleting an object from the destination. The controller places the tasks onto the message queue. A first source worker node retrieves the first task and coordinates with a first destination worker node to perform the first task. A second source worker nodes retrieves the second task and coordinates with a second destination worker node to perform the second task.

Abstract: A source worker node at a source site fetches a task from a message queue. The task specifies replicating a first object at the source site to a destination site. A request for a connection is issued from the source worker node to the destination site. The request is received by a load balancer at the destination site and assigned to a destination worker node. A connection is established between the source and destination worker nodes. A determination is made that the destination site does not include an object that is the same as the first object. Upon the determination, a deduplication is performed between the source and destination worker nodes of segments into which the first object has been divided. Deduplicated segments of the first object are transmitted from the source worker node to the destination worker node for storage at the destination site.