Abstract: The present disclosure is related to methods, systems, and machine-readable media for log-structured file system management operations. An aggregate amount of over-provisioned computing resources assigned to a plurality of log-structured file system (LFS) objects that are allocated for performance of memory management operations associated with a virtual storage area network (vSAN) can be determined. A subset of LFS objects that are candidates for performance of a particular memory management operation based on an amount of over-provisioned computing resources consumed by one or more LFS objects among the plurality of LFS objects exceeding a resource consumption threshold associated with the determined amount of over-provisioned computing resources assigned to the plurality of LFS objects can be selected. The particular memory management operation using one or more of the candidate LFS objects can be performed.

Abstract: The present disclosure is related to methods, systems, and machine-readable media for log-structured file system management operations. An aggregate amount of over-provisioned computing resources assigned to a plurality of log-structured file system (LFS) objects that are allocated for performance of memory management operations associated with a virtual storage area network (vSAN) can be determined. A subset of LFS objects that are candidates for performance of a particular memory management operation based on an amount of over-provisioned computing resources consumed by one or more LFS objects among the plurality of LFS objects exceeding a resource consumption threshold associated with the determined amount of over-provisioned computing resources assigned to the plurality of LFS objects can be selected. The particular memory management operation using one or more of the candidate LFS objects can be performed.

Abstract: The present disclosure is related to methods, systems, and machine-readable media for supporting clones with consolidated snapshots. A clone of a virtual computing instance can be created in a snapshot tree provided by a software defined data center. A consolidated snapshot including tuples from a diskchain of the clone in the snapshot tree can be created responsive to a determination that the snapshot tree exceeds a threshold.

Abstract: A technique to name data is disclosed to allow preservation of storage efficiency over a link between a source and a destination in a replication relationship as well as in storage at the destination. The technique allows the source to send named data to the destination once and refer to it by name multiple times in the future, without having to resend the data. The technique also allows the transmission of data extents to be decoupled from the logical containers that refer to the data extents. Additionally, the technique allows a replication system to accommodate different extent sizes between replication source and destination while preserving storage efficiency.

Abstract: A technique to name data is disclosed to allow preservation of storage efficiency over a link between a source and a destination in a replication relationship as well as in storage at the destination. The technique allows the source to send named data to the destination once and refer to it by name multiple times in the future, without having to resend the data. The technique also allows the transmission of data extents to be decoupled from the logical containers that refer to the data extents. Additionally, the technique allows a replication system to accommodate different extent sizes between replication source and destination while preserving storage efficiency.

Abstract: A technique to name data is disclosed to allow preservation of storage efficiency over a link between a source and a destination in a replication relationship as well as in storage at the destination. The technique allows the source to send named data to the destination once and refer to it by name multiple times in the future, without having to resend the data. The technique also allows the transmission of data extents to be decoupled from the logical containers that refer to the data extents. Additionally, the technique allows a replication system to accommodate different extent sizes between replication source and destination while preserving storage efficiency.

Abstract: A system can maintain multiple queues for deduplication requests of different priorities. The system can also designate priority of storage units. The scheduling priority of a deduplication request is based on the priority of the storage unit indicated in the deduplication request and a trigger for the deduplication request.

Abstract: A system can maintain multiple queues for deduplication requests of different priorities. The system can also designate priority of storage units. The scheduling priority of a deduplication request is based on the priority of the storage unit indicated in the deduplication request and a trigger for the deduplication request.

Abstract: A system can maintain multiple queues for deduplication requests of different priorities. The system can also designate priority of storage units. The scheduling priority of a deduplication request is based on the priority of the storage unit indicated in the deduplication request and a trigger for the deduplication request.

Abstract: A technique to name data is disclosed to allow preservation of storage efficiency over a link between a source and a destination in a replication relationship as well as in storage at the destination. The technique allows the source to send named data to the destination once and refer to it by name multiple times in the future, without having to resend the data. The technique also allows the transmission of data blocks to be decoupled from the logical containers that refer to the data blocks. The technique creates a name-to-data mapping at the destination, which maps source data block names to destination data block addresses. An invalidation of the data block namespace can be detected, in which case appropriate steps can be taken to rebuild the name-to-data mapping.

Abstract: Systems for deduplicating one or more storage units of a storage system provide a scheduler, which is operable to select at least one storage unit (e.g. a storage volume) for deduplication and perform a deduplication process, which removes duplicate data blocks from the selected storage volume. The systems are operable to determine the state of one or more storage units and manage deduplication requests in part based state information. The system is further operable to manage user generated requests and manage deduplication requests in part based on user input information. The system may include a rules engine which prioritizes system operations including determining an order in which to perform state-gathering information and determining an order in which to perform deduplication. The system is further operable to determine the order in which storage units are processed.

Abstract: A system can maintain multiple queues for deduplication requests of different priorities. The system can also designate priority of storage units. The scheduling priority of a deduplication request is based on the priority of the storage unit indicated in the deduplication request and a trigger for the deduplication request.

Abstract: It is determined that a first data block contains the same data as a second data block. The first data block is associated with a first extent and the second data block is associated with a second extent. In response to determining that the first data block contains the same data as the second data block, the second data block is associated with the first extent and the first data block is disassociated with the second extent.

Abstract: An extent-based storage architecture is implemented by a storage server receiving a read request for an extent from a client, wherein the extent includes a group of contiguous blocks and the read request includes a file block number. The storage server retrieves an extent identifier from a first sorted data structure, wherein the storage server uses the received file block number to traverse the first sorted data structure to the extent identifier. The storage server retrieves a reference to the extent from a second sorted data structure, wherein the storage server uses the retrieved extent identifier to traverse the second sorted data structure to the reference, and wherein the second sorted data structure is global across a plurality of volumes. The storage server retrieves the extent from a storage device using the reference and returns the extent to the client.

Abstract: An extent-based storage architecture is implemented by a storage server. The storage server generates a new extent identifier for cloning a source extent identified by a source extent identifier and stored at a source data structure that includes a length value providing a length of the source extent, an offset value and a reference count value that provides a number of data containers that reference the source extent identifier. The storage server updates a data structure for a cloned version of the data container for storing the new extent identifier that points to the source extent identifier and includes an extent length value and offset value different from length value and the offset value of the source data structure.

Abstract: Systems for deduplicating one or more storage units of a storage system provide a scheduler, which is operable to select at least one storage unit (e.g. a storage volume) for deduplication and perform a deduplication process, which removes duplicate data blocks from the selected storage volume. The systems are operable to determine the state of one or more storage units and manage deduplication requests in part based state information. The system is further operable to manage user generated requests and manage deduplication requests in part based on user input information. The system may include a rules engine which prioritizes system operations including determining an order in which to perform state-gathering information and determining an order in which to perform deduplication. The system is further operable to determine the order in which storage units are processed.

Abstract: A method for efficiently handling partial write requests in a storage system includes allocating a new block of data for the new partial data, and allocating a record in an extent map to record the location of the new partial data block, the location of the old partial data block and the offset length for each data block. Data blocks can be repackaged in the background when system resources are available. A full, but misaligned write request is also efficiently handled by writing the new data to a newly allocated data block and allocating new records in an extent map to record information corresponding to two partial write operations.

Abstract: Overwriting part of compressed data without decompressing on-disk compressed data is includes by receiving a write request for a block of data in a compression group from a client, wherein the compression group comprises a group of data blocks that is compressed, wherein the block of data is uncompressed. The storage server partially overwrites the compression group, wherein the compression group remains compressed while the partial overwriting is performed. The storage server determines whether the partially overwritten compression group including the uncompressed block of data should be compressed. The storage server defers compression of the partially overwritten compression group if the partially overwritten compression group should not be compressed. The storage server compresses the partially overwritten compression group if the partially overwritten compression group should be compressed.

Abstract: A request is received to remove duplicate data. A log data container associated with a storage volume in a storage server is accessed. The log data container includes a plurality of entries. Each entry is identified by an extent identifier in a data structures stored in a volume associated with the storage server. For each entry in the log data container, a determination is made if the entry matches another entry in the log data container. If the entry matches another entry in the log data container, a determination is made of a donor extent and a recipient extent. If an external reference count associated with the recipient extent equals a first predetermined value, block sharing is performed for the donor extent and the recipient extent. A determination is made if the reference count of the donor extent equals a second predetermined value. If the reference count of the donor extent equals the second predetermined value, the donor extent is freed.

Abstract: An extent-based storage architecture is implemented by a storage server receiving a read request for an extent from a client, wherein the extent includes a group of contiguous blocks and the read request includes a file block number. The storage server retrieves an extent identifier from a first sorted data structure, wherein the storage server uses the received file block number to traverse the first sorted data structure to the extent identifier. The storage server retrieves a reference to the extent from a second sorted data structure, wherein the storage server uses the retrieved extent identifier to traverse the second sorted data structure to the reference, and wherein the second sorted data structure is global across a plurality of volumes. The storage server retrieves the extent from a storage device using the reference and returns the extent to the client.