Abstract: Data in a storage system is deduplicated after receiving from at least one writing entity requests for a plurality of write operations for a corresponding plurality of data blocks in a storage object. The received blocks are buffered and sorted in order and a sequence of clumps is created from the buffered blocks, where each clump comprises a grouping of at least one of the sorted, buffered blocks. A boundary is determined between at least one pair of clumps based at least in part on the content of at least one of the buffered blocks, and it is then determined whether at least one of the clumps is a duplicate of a previously stored clump.

Abstract: Techniques for replicating virtual machine data is provided. A plurality of compute nodes running on a primary cluster determine the amount of virtual machine data cached within each compute node. Based on the amount of virtual machine data for a particular virtual machine, a particular compute node is assigned to replicate the data to a secondary cluster. The amount of particular virtual machine data copied to the secondary cluster is based on updated virtual machine data that belongs to a particular state of the virtual machine. The destination of the particular virtual machine data is based on available cache space and prior replication statistics for target compute nodes on the secondary cluster.

Abstract: Techniques for scheduling replication events may be based upon establishing a plurality of policy groups. Each policy group has a replication schedule that defines when to initiate replication events and a membership selection pattern used to determine which virtual machines belong to which policy group. The policy groups may contain a first policy group and a second policy group, where each policy group has a unique replication schedule and a unique selection pattern. The system may assign a first set of virtual machines to the first policy group based upon the first selection pattern. A second set of virtual machines may be assigned to a second policy group based upon the second selection pattern. Each of the virtual machines in the first policy group may be assigned a first replication schedule and each of the virtual machines in the second policy group may be assigned a second replication schedule.

Abstract: A data storage system includes a plurality of hosts, each of which includes at least one processor and communicates over a network with a plurality of storage nodes, at least one of which has at least one storage device, at least one storage controller, and at least one non-volatile memory. At least one process within a host issues data storage read/write requests. At least one of the hosts has a cache for caching data stored in at least one of the plurality of storage nodes. The host writes data corresponding to a write request to at least one remote non-volatile memory and carries out at least one storage processing function; data in the written-to node may then be made available for subsequent reading by a different one of the hosts. Examples of the storage processing function include compression, ECC computation, deduplicating, garbage collection, write logging, reconstruction, rebalancing, and scrubbing.

Abstract: Exemplary methods, apparatuses, and systems maintain a plurality of summary data structures corresponding to a plurality of logical file system namespaces representing a plurality of hierarchies of one or more directories having one or more files, each file being stored in the storage system as a plurality of segments in a deduplicated manner. In response to a request to estimate a storage usage by a first of the file system namespace, identify a first of the summary data structures corresponding to the first file system namespace, wherein the first summary data structure stores information summarizing deduplicated segments referenced by one or more files of the first file system namespace. Estimate the storage usage of the first file system namespace based on the first summary data structure and a global summary data structure, wherein the global summary data structure stores information summarizing deduplicated segments referenced by all of the file system namespaces.

Abstract: Techniques for scheduling replication events may be based upon establishing a plurality of policy groups. Each policy group has a replication schedule that defines when to initiate replication events and a membership selection pattern used to determine which virtual machines belong to which policy group. The policy groups may contain a first policy group and a second policy group, where each policy group has a unique replication schedule and a unique selection pattern. The system may assign a first set of virtual machines to the first policy group based upon the first selection pattern. A second set of virtual machines may be assigned to a second policy group based upon the second selection pattern. Each of the virtual machines in the first policy group may be assigned a first replication schedule and each of the virtual machines in the second policy group may be assigned a second replication schedule.

Abstract: Techniques for replicating virtual machine data is provided. A plurality of compute nodes running on a primary cluster determine the amount of virtual machine data cached within each compute node. Based on the amount of virtual machine data for a particular virtual machine, a particular compute node is assigned to replicate the data to a secondary cluster. The amount of particular virtual machine data copied to the secondary cluster is based on updated virtual machine data that belongs to a particular state of the virtual machine. The destination of the particular virtual machine data is based on available cache space and prior replication statistics for target compute nodes on the secondary cluster.

Abstract: Data in a storage system is deduplicated after receiving from at least one writing entity requests for a plurality of write operations for a corresponding plurality of data blocks in a storage object. The received blocks are buffered and sorted in order and a sequence of clumps is created from the buffered blocks, where each clump comprises a grouping of at least one of the sorted, buffered blocks. A boundary is determined between at least one pair of clumps based at least in part on the content of at least one of the buffered blocks, and it is then determined whether at least one of the clumps is a duplicate of a previously stored clump.

Abstract: A data storage system includes a plurality of hosts, each of which includes at least one processor and communicates over a network with a plurality of storage nodes, at least one of which has at least one storage device, at least one storage controller, and at least one non-volatile memory. At least one process within a host issues data storage read/write requests. At least one of the hosts has a cache for caching data stored in at least one of the plurality of storage nodes. The host writes data corresponding to a write request to at least one remote non-volatile memory and carries out at least one storage processing function; data in the written-to node may then be made available for subsequent reading by a different one of the hosts. Examples of the storage processing function include compression, ECC computation, deduplicating, garbage collection, write logging, reconstruction, rebalancing, and scrubbing.

Abstract: Data in a storage system is deduplicated after receiving from at least one writing entity requests for a plurality of write operations for a corresponding plurality of data blocks in a storage object. The received blocks are buffered and sorted in order and a sequence of clumps is created from the buffered blocks, where each clump comprises a grouping of at least one of the sorted, buffered blocks. A boundary is determined between at least one pair of clumps based at least in part on the content of at least one of the buffered blocks, and it is then determined whether at least one of the clumps is a duplicate of a previously stored clump.

Abstract: A data storage system includes a plurality of hosts, each of which includes at least one processor and communicates over a network with a plurality of storage nodes, at least one of which has at least one storage device, at least one storage controller, and at least one non-volatile memory. At least one process within a host issues data storage read/write requests. At least one of the hosts has a cache for caching data stored in at least one of the storage nodes. The host writes data corresponding to a write request to at least one remote non-volatile memory and carries out at least one storage processing function; data in the written-to node may then be made available for subsequent reading by a different one of the hosts. Examples of the storage processing function include compression, ECC computation, deduplicating, garbage collection, write logging, reconstruction, rebalancing, and scrubbing.

Abstract: Embodiments of a distributed virtual array data storage system and method are disclosed. Storage nodes made up of relatively unsophisticated disks with associated processors are scalable to store very large amounts of data. The storage nodes communicate with servers directly over a network through, for example, an Ethernet connection. Control of the storage nodes and access to the storage nodes is handled entirely on the server side of the system by distributed virtual array (DVA) software running on the server side and employing a particular protocol over the standard network connection. In an embodiment, server-side virtual machine (VM) hosts host application VMs that are associated with vDisks. The DVA software distributes data for the vDisk over the storage nodes. In the case of failure of one or more of the storage nodes, the DVA software reconstructs the data, for example by reading redundant data from surviving storage nodes.

Abstract: A distributed storage system that dispatches an input/output request is described. In an exemplary embodiment, a storage controller client receives the input/output request, wherein the distributed storage system includes the storage controller client, a plurality of storage controller servers, a plurality of virtual nodes distributed among a plurality of physical nodes, and each of the plurality of physical nodes is hosted on one of the plurality of storage controller servers. The storage controller client further computes a target virtual node for the input/output request, where the target virtual node is one of the plurality of virtual nodes. Using the computed target virtual node, the storage controller client determines a target physical node that corresponds to the target virtual node, where the target physical node is one of the plurality of physical nodes.

Abstract: Data in a storage system is deduplicated after receiving from at least one writing entity requests for a plurality of write operations for a corresponding plurality of data blocks in a storage object. The received blocks are buffered and sorted in order and a sequence of clumps is created from the buffered blocks, where each clump comprises a grouping of at least one of the sorted, buffered blocks. A boundary is determined between at least one pair of clumps based at least in part on the content of at least one of the buffered blocks, and it is then determined whether at least one of the clumps is a duplicate of a previously stored clump.

Abstract: A distributed storage system that dispatches an input/output request is described. In an exemplary embodiment, a storage controller client receives the input/output request, wherein the distributed storage system includes the storage controller client, a plurality of storage controller servers, a plurality of virtual nodes distributed among a plurality of physical nodes, and each of the plurality of physical nodes is hosted on one of the plurality of storage controller servers. The storage controller client further computes a target virtual node for the input/output request, where the target virtual node is one of the plurality of virtual nodes. Using the computed target virtual node, the storage controller client determines a target physical node that corresponds to the target virtual node, where the target physical node is one of the plurality of physical nodes.

Abstract: Cluster storage is disclosed. A data stream or a data block is received. The data stream or the data block is broken into segments. For each segment, a cluster node is selected, and a portion of the segment smaller than the segment is identified that is a duplicate of a portion of a segment already managed by the cluster node.

Abstract: A data storage system includes a plurality of hosts, each of which includes at least one processor and communicates over a network with a plurality of storage nodes, at least one of which has at least one storage device, at least one storage controller, and at least one non-volatile memory. At least one process within a host issues data storage read/write requests. At least one of the hosts has a cache for caching data stored in at least one of the storage nodes. The host writes data corresponding to a write request to at least one remote non-volatile memory and carries out at least one storage processing function; data in the written-to node may then be made available for subsequent reading by a different one of the hosts. Examples of the storage processing function include compression, ECC computation, deduplicating, garbage collection, write logging, reconstruction, rebalancing, and scrubbing.

Abstract: Cluster storage is disclosed. A data stream or a data block is received. The data stream or the data block is broken into segments. For each segment, a cluster node is selected, and a portion of the segment smaller than the segment is identified that is a duplicate of a portion of a segment already managed by the cluster node.

Abstract: Exemplary methods, apparatuses, and systems maintain a plurality of summary data structures corresponding to a plurality of logical file system namespaces representing a plurality of hierarchies of one or more directories having one or more files, each file being stored in the storage system as a plurality of segments in a deduplicated manner. In response to a request to estimate a storage usage by a first of the file system namespace, identify a first of the summary data structures corresponding to the first file system namespace, wherein the first summary data structure stores information summarizing deduplicated segments referenced by one or more files of the first file system namespace. Estimate the storage usage of the first file system namespace based on the first summary data structure and a global summary data structure, wherein the global summary data structure stores information summarizing deduplicated segments referenced by all of the file system namespaces.

Abstract: Cluster storage is disclosed. A data stream or a data block is received. The data stream or the data block is broken into segments. For each segment, a cluster node is selected, and a portion of the segment smaller than the segment is identified that is a duplicate of a portion of a segment already managed by the cluster node.