Abstract: A technique improves storage efficiency of an object store configured to maintain numerous snapshots for long-term storage in an archival storage system by efficiently determining data that is exclusively owned by an expiring snapshot to allow deletion of the expiring snapshot from the object store. The technique involves managing index data structures to enable efficient garbage collection across a very large number of data objects. When a snapshot expires, the technique obviates the need to scan the numerous snapshot data objects to determine which index structures are no longer needed and can be reclaimed (garbage collected). The technique is directed to management of underlying storage based on different sets of policies. When certain snapshots expire and are ready for deletion, the technique is directed to finding those data blocks that are no longer referenced (used) by any valid snapshots.

Abstract: A technique improves storage efficiency of an object store configured to maintain numerous snapshots for long-term storage in an archival storage system by efficiently determining data that is exclusively owned by an expiring snapshot to allow deletion of the expiring snapshot from the object store. The technique involves managing index data structures to enable efficient garbage collection across a very large number of data objects. When a snapshot expires, the technique obviates the need to scan the numerous snapshot data objects to determine which index structures are no longer needed and can be reclaimed (garbage collected). The technique is directed to management of underlying storage based on different sets of policies. When certain snapshots expire and are ready for deletion, the technique is directed to finding those data blocks that are no longer referenced (used) by any valid snapshots.

Abstract: An indexing technique provides an index data structure for efficient retrieval of a snapshot from a long-term storage service (LTSS) of an archival storage system. The snapshot is generated from typed data of a logical entity, such as a virtual disk (vdisk). The data of the snapshot is replicated to a frontend data service of the LTSS sequentially and organized as one or more data objects for storage by a backend data service of LTSS in an object store of the archival storage system. Metadata associated with the snapshot (i.e., snapshot metadata) is recorded as a log and persistently stored on storage media local to the frontend data service. The snapshot metadata includes information describing the snapshot data, e.g., a logical offset range of a snapshot of the vdisk and, thus, is used to construct the index data structure. Notably, construction of the index data structure is deferred until after the entirety of the snapshot data has been replicated and received by the frontend data service.

Abstract: A high frequency snapshot technique improves data replication in a disaster recovery (DR) environment. A base snapshot is generated from failover data at a primary site and replicated to a placeholder file at a secondary site. Upon commencement of the base snapshot generation and replication, incremental light weight snapshots (LWSs) of the failover data are captured and replicated to the secondary site. A staging file at the secondary site accumulates the replicated LWSs (“high-frequency snapshots”). The staging file is populated with the LWSs in parallel with the replication of the base snapshot at the placeholder file. At a subsequent predetermined time interval, the accumulated LWSs are synthesized to capture a “checkpoint” snapshot by applying and pruning the accumulated LWSs at the staging file. Once the base snapshot is fully replicated, the pruned LWSs are merged to the base snapshot to synchronize the replicated failover data.

Abstract: A system and method for capturing high frequency snapshots of an entity includes creating, by a storage sub-system associated with a distributed storage system of a virtualized environment, a cached configuration of an entity within the storage sub-system, capturing, by the storage sub-system, a snapshot of the entity from the cached configuration based on a schedule of a snapshot policy, including sending, by the storage sub-system, the snapshot to a repository indicated in the snapshot policy, and determining, by the storage sub-system, that a configuration file associated with the entity has been updated. The system and method also includes updating, by the storage sub-system, the cached configuration based upon the configuration file that has been updated.

Abstract: A system and method for capturing high frequency snapshots of an entity includes creating, by a storage sub-system associated with a distributed storage system of a virtualized environment, a cached configuration of an entity within the storage sub-system, capturing, by the storage sub-system, a snapshot of the entity from the cached configuration based on a schedule of a snapshot policy, including sending, by the storage sub-system, the snapshot to a repository indicated in the snapshot policy, and determining, by the storage sub-system, that a configuration file associated with the entity has been updated. The system and method also includes updating, by the storage sub-system, the cached configuration based upon the configuration file that has been updated.

Abstract: A computer system maintains identifiers that identify changed blocks of virtual machine (VM) storage. The computer system accesses a stable VM checkpoint comprising a restorable VM image at a time, and that stores a representation of data of at least one block as it existed at the time. The computer system converts the checkpoint to a reference point. Reference point information is transferable with the VM, such that if the VM is moved to a different computing system, any data identified by the reference point is recoverable. The conversion includes querying the storage to determine an identifier corresponding to the block of the checkpoint at the time, storing this identifier as a part of the reference point, and releasing the representation of the data of the block from the checkpoint. The computer system then uses the reference point to identify changes in the blocks of the storage since the time.

Abstract: Embodiments are directed to backing up a virtual machine cluster and to determining virtual machine node ownership prior to backing up a virtual machine cluster. In one scenario, a computer system determines which virtual machines nodes are part of the virtual machine cluster, determines which shared storage resources are part of the virtual machine cluster and determines which virtual machine nodes own the shared storage resources. The computer system then indicates to the virtual machine node owners that at least one specified application is to be quiesced over the nodes of the virtual machine cluster, such that a consistent, cluster-wide checkpoint can be created. The computer system further creates a cluster-wide checkpoint which includes a checkpoint for each virtual machine in the virtual machine cluster.

Abstract: A method and system for incremental backup is disclosed. The method includes writing a first update to a secondary node and writing a second update to the secondary node. The secondary node comprises a replication storage group (RSG). Further, the order in which the updates are written at the secondary node is consistent with the order in which the updates were written at a primary node.

Abstract: The writing of data to a storage system such that change tracking is efficiently performed. If a portion is to be written to the storage system, the system writes a write record indicating that a group of portions (that includes the particular portion) of the storage system is to be written to the storage system. This is represented even though those other portions are not being contemporaneously written to the storage system, and may in fact never be written. The particular portion is then written to the storage system. At some point thereafter, perhaps in the background, a change tracking structure is changed to reflect that the particular portion is written to the storage system, but without reflecting writes of all of the group of portions. The write record may then be invalidated. This reduces latency in systems that track changes with small cost at the time of backup.

Abstract: A computer system maintains identifiers that identify changed blocks of virtual machine (VM) storage. The computer system accesses a stable VM checkpoint comprising a restorable VM image at a time, and that stores a representation of data of at least one block as it existed at the time. The computer system converts the checkpoint to a reference point. Reference point information is transferable with the VM, such that if the VM is moved to a different computing system, any data identified by the reference point is recoverable. The conversion includes querying the storage to determine an identifier corresponding to the block of the checkpoint at the time, storing this identifier as a part of the reference point, and releasing the representation of the data of the block from the checkpoint. The computer system then uses the reference point to identify changes in the blocks of the storage since the time.

Abstract: A method, system and computer-readable medium for providing rapid failback of a computer system is described. The method, which operates during failback of a secondary computer to a primary computer, accesses a map to determine a location of a latest version of data corresponding to a read request, where the location may be within either a primary data storage or a secondary data storage. The system comprises a primary computer coupled to a primary data storage and a secondary computer coupled to a secondary data storage. The primary computer maintains a write log and the secondary computer maintains a map. The computer-readable medium contains instructions, which, when executed by a processor, performs the steps embodied by the method.

Abstract: Embodiments are directed to backing up a virtual machine cluster and to determining virtual machine node ownership prior to backing up a virtual machine cluster. In one scenario, a computer system determines which virtual machines nodes are part of the virtual machine cluster, determines which shared storage resources are part of the virtual machine cluster and determines which virtual machine nodes own the shared storage resources. The computer system then indicates to the virtual machine node owners that at least one specified application is to be quiesced over the nodes of the virtual machine cluster, such that a consistent, cluster-wide checkpoint can be created. The computer system further creates a cluster-wide checkpoint which includes a checkpoint for each virtual machine in the virtual machine cluster.

Abstract: Tracking changes amongst unit portions (e.g., blocks or files) of a storage system. A logical time identifier is associated with each unit portion and is included within a logical time identifier structure. When writing to a particular write portion, the mechanism updates the appropriate logical time identifiers, calculates redundancy data of a group of one or more logical time identifiers associated with the unit portion(s) of the write portion. Furthermore, the write portion of the storage system is written. In addition, the corresponding redundancy data for that write portion is written to the logical time identifier structure. Later, for a given write portion, the redundancy data is verified to be consistent or inconsistent with the group of one or more logical time identifiers associated with the write portion. If the redundancy data is not consistent, then a current logical time identifier is assigned to each of the logical time identifiers.

Abstract: Embodiments are directed to establishing efficient virtual machine reference points and to specifying a virtual machine reference point to query incremental changes. In one scenario, a computer system accesses a stable virtual machine checkpoint that includes portions of underlying data stored in data storage, where the checkpoint is associated with a specific point in time. The computer system then queries the data storage to determine data storage identifiers that reference the point in time associated with the checkpoint and stores the determined data storage identifiers as a virtual machine reference point, where each subsequent change to the data storage results in an update to the data storage identifier, so that virtual machine reference point is usable to identify incremental changes from specific points in time on.

Abstract: The writing of data to a storage system such that change tracking is efficiently performed. If a portion is to be written to the storage system, the system writes a write record indicating that a group of portions (that includes the particular portion) of the storage system is to be written to the storage system. This is represented even though those other portions are not being contemporaneously written to the storage system, and may in fact never be written. The particular portion is then written to the storage system. At some point thereafter, perhaps in the background, a change tracking structure is changed to reflect that the particular portion is written to the storage system, but without reflecting writes of all of the group of portions. The write record may then be invalidated. This reduces latency in systems that track changes with small cost at the time of backup.

Abstract: Embodiments are directed to backing up a virtual machine cluster and to determining virtual machine node ownership prior to backing up a virtual machine cluster. In one scenario, a computer system determines which virtual machines nodes are part of the virtual machine cluster, determines which shared storage resources are part of the virtual machine cluster and determines which virtual machine nodes own the shared storage resources. The computer system then indicates to the virtual machine node owners that at least one specified application is to be quiesced over the nodes of the virtual machine cluster, such that a consistent, cluster-wide checkpoint can be created. The computer system further creates a cluster-wide checkpoint which includes a checkpoint for each virtual machine in the virtual machine cluster.

Abstract: Tracking changes amongst unit portions (e.g., blocks or files) of a storage system. A logical time identifier is associated with each unit portion and is included within a logical time identifier structure. When writing to a particular write portion, the mechanism updates the appropriate logical time identifiers, calculates redundancy data of a group of one or more logical time identifiers associated with the unit portion(s) of the write portion. Furthermore, the write portion of the storage system is written. In addition, the corresponding redundancy data for that write portion is written to the logical time identifier structure. Later, for a given write portion, the redundancy data is verified to be consistent or inconsistent with the group of one or more logical time identifiers associated with the write portion. If the redundancy data is not consistent, then a current logical time identifier is assigned to each of the logical time identifiers.

Abstract: Embodiments are directed to establishing efficient virtual machine reference points and to specifying a virtual machine reference point to query incremental changes. In one scenario, a computer system accesses a stable virtual machine checkpoint that includes portions of underlying data stored in data storage, where the checkpoint is associated with a specific point in time. The computer system then queries the data storage to determine data storage identifiers that reference the point in time associated with the checkpoint and stores the determined data storage identifiers as a virtual machine reference point, where each subsequent change to the data storage results in an update to the data storage identifier, so that virtual machine reference point is usable to identify incremental changes from specific points in time on.

Abstract: Embodiments are directed to backing up a virtual machine cluster and to determining virtual machine node ownership prior to backing up a virtual machine cluster. In one scenario, a computer system determines which virtual machines nodes are part of the virtual machine cluster, determines which shared storage resources are part of the virtual machine cluster and determines which virtual machine nodes own the shared storage resources. The computer system then indicates to the virtual machine node owners that at least one specified application is to be quiesced over the nodes of the virtual machine cluster, such that a consistent, cluster-wide checkpoint can be created. The computer system further creates a cluster-wide checkpoint which includes a checkpoint for each virtual machine in the virtual machine cluster.