Abstract: Embodiments of the disclosure provide techniques managing a log-structured solid state drive (SSD) format in a distributed storage system. SSDs in the distributed storage system maintains a journal of logical changes to storage objects to persist prepared and committed changes in the latency path. The journal includes metadata entries that describe changes and reference data pages. Dense data structures (such as a logical block addressing table) index the metadata entries. To reduce the amount of overhead in I/O operations, the distributed storage system maintains the dense data structures in memory rather than on disk.

Abstract: Techniques are disclosed for maintaining high availability (HA) for virtual machines (VMs) running on host systems of a host cluster, where each host system executes a HA module in a plurality of HA modules and a storage module in a plurality of storage modules, where the host cluster aggregates, via the plurality of storage modules, locally-attached storage resources of the host systems to provide logical data store, and where persistent data for the VMs is stored across the locally-attached storage resources comprising the logical data store.

Abstract: Embodiments of the disclosure provide techniques managing a log-structured solid state drive (SSD) format in a distributed storage system. SSDs in the distributed storage system maintains a journal of logical changes to storage objects to persist prepared and committed changes in the latency path. The journal includes metadata entries that describe changes and reference data pages. Dense data structures (such as a logical block addressing table) index the metadata entries. To reduce the amount of overhead in I/O operations, the distributed storage system maintains the dense data structures in memory rather than on disk.

Abstract: A method includes obtaining a plurality of representations corresponding respectively to a plurality of blocks of data stored on a source node. A plurality of data pairs are sent to a destination node, where each data pair includes a logical address associated with a block of data from the plurality of blocks of data and the corresponding representation of the block of data. A determination is made whether the blocks of data associated with the respective logical addresses are duplicates of data stored on the destination node. In accordance with an affirmative determination, a reference to a physical address of the block of data stored on the destination node is stored. In accordance with a negative determination, an indication that the data corresponding to the respective logical address is not a duplicate is stored. The data indicated as not being a duplicate is written to the destination node.

Abstract: A method and system are disclosed for storing client data objects in a deduplicated storage system. Deduplicated data may be stored in a plurality of physical data blocks. A content map layer can provide a mapping between the physical data blocks to logical map objects associated with the client data objects. The deduplicated data may be mapped to logical data blocks that comprise the client data objects.

Abstract: In a storage cluster having nodes, blocks of a logical storage space of a storage object are allocated flexibly by a parent node to component nodes that are backed by physical storage. The method includes maintaining a first allocation map for the parent node, and second and third allocation maps for the first and second component nodes, respectively, executing a first write operation on the first component node and updating the second allocation map to indicate that the first block is a written block, selecting the second component node for executing a second write operation, and executing the second write operation on the second component node. Upon execution of the second write operation, the third allocation map is updated to indicate that the second block is a written block and the first allocation map is updated to indicate that the second block is allocated to the second component node.

Abstract: In a storage cluster having nodes, blocks of a logical storage space of a storage object are allocated flexibly by a parent node to component nodes that are backed by physical storage. The method includes maintaining a first allocation map for the parent node, and second and third allocation maps for the first and second component nodes, respectively, executing a first write operation on the first component node and updating the second allocation map to indicate that the first block is a written block, selecting the second component node for executing a second write operation, and executing the second write operation on the second component node. Upon execution of the second write operation, the third allocation map is updated to indicate that the second block is a written block and the first allocation map is updated to indicate that the second block is allocated to the second component node.

Abstract: In a storage cluster having nodes, blocks of a logical storage space of a storage object are allocated flexibly by a parent node to component nodes that are backed by physical storage. The method includes maintaining a first allocation map for the parent node, and second and third allocation maps for the first and second component nodes, respectively, executing a first write operation on the first component node and updating the second allocation map to indicate that the first block is a written block, and upon detecting that the first component node is offline, executing a second write operation that targets a second block of the logical storage space, which is allocated to the first component node, on the second component node and updating the third allocation map to indicate that the second block is a written block.

Abstract: System and method for managing multiple data storages using a file system of a computer system utilize a primary data storage to cache objects of logical object containers stored in a secondary data storage in caching-tier volumes. When an access request for an object stored in the secondary data storage is received at the file system and the object is not currently cached in the primary data storage, a caching-tier volume in the primary data storage is created that corresponds to a logical object container in the secondary data storage that includes the requested object. The caching-tier volume is used to cache the object as an inflated file so that the inflated file is available at the primary data storage in the caching-tier volume for a subsequent access request for the object stored in the secondary data storage.

Abstract: Exemplary methods, apparatuses, and systems include a first layer of a virtual storage area network (VSAN) module receiving a write request from a data compute node. The write request includes data to be written and the VSAN module is distributed across a plurality of computers to provide an aggregate object store using storage attached to each of the plurality of computers. The first layer of the VSAN module calculates a checksum for the data to be written and passes the data to be written and the checksum to a second layer of the VSAN module. The second layer of the VSAN module calculates a first verification checksum for the data to be written. The data and the checksum are written to persistent storage in response to determining the first verification checksum matches the checksum passed by the first layer of the VSAN module.

Abstract: Exemplary methods, apparatuses, and systems maintain hole boundary information by calculating a block attribute parity value. For example, a request is received to write to a first block of a stripe of data. A block attribute of a second block is determined. The block attribute of the second block indicates whether the second block includes written data or is a hole. A block attribute parity value is calculated based upon both the block attribute of the first block and the block attribute of the second block. The block attribute of the first block indicates the first block includes written data based upon the received request. The block attribute parity value and the data parity value are stored on one of the physical storage devices in response to the received write request. As a result, if a disk is lost, holes can be recovered using the block attribute parity value.

Abstract: A deduplication storage system with snapshot and clone capability includes storing logical pointer objects and organizing a first set of the logical pointer objects into a hierarchical structure. A second set of the logical pointer objects may be associated with corresponding logical data blocks of a client data object. The second set of the logical pointer objects may point to physical data blocks having deduplicated data that comprise data of the corresponding logical data blocks. Some of the logical pointer objects in the first set may point to the logical pointer objects in the second set, so that the hierarchical structure represents the client data object. A root of the hierarchical structure may be associated with the client data object. A snapshot or clone may be created by making a copy of the root and associating the copied root with the snapshot or clone.

Abstract: A method for restoring a data volume using incremental snapshots of the data volume includes creating a first series of incremental snapshots according to a first predefined interval. The method further includes creating a second series of incremental snapshots according to a second predefined interval that is an integer multiple of the first predefined interval. The method also includes receiving a request to restore the data volume to a point-in-time. The method further includes restoring the data volume to the point-in-time using none or some of the snapshots in the first series that were created at or prior to the point-in-time, and all of the snapshots in the second series that were created at or prior to the point-in-time.

Abstract: Embodiments of the disclosure provide techniques for measuring congestion and controlling quality of service to a shared resource. A module that interfaces with the shared resource monitors the usage of the shared resource by accessing clients. Upon detecting that the rate of usage of the shared resource has exceeded a maximum rate supported by the shared resource, the module determines and transmits a congestion metric to clients that are currently attempting to access the shared resource. Clients, in turn determine a delay period based on the congestion metric prior to attempting another access of the shared resource.

Abstract: A given host machine in a virtualization system having a virtual distributed storage system may receive an iSCSI protocol packet from a computer system separate from the given host machine. Processing the iSCSI protocol may include accessing distributed storage device (iSCSI target) comprising storage connected to the two or more host machines in the virtualization system. The given host machine may generate an outbound iSCSI protocol packet comprising return data received from the target and send the outbound iSCSI protocol packet to the computer system.

Abstract: Techniques are disclosed for providing a file system interface for an object store intended to support simultaneous access to objects stored in the object store by multiple clients. In accordance with one method, an abstraction of a root directory to a hierarchical namespace for the object store is exposed to clients. The object store is backed by a plurality of physical storage devices housed in or directly attached to the plurality of host computers and internally tracks its stored objects using a flat namespace that maps unique identifiers to the stored objects. The creation of top-level objects appearing as subdirectories of the root directory is enabled, wherein each top-level object represents a separate abstraction of a storage device having a separate namespace that can be organized in accordance with any designated file system.

Abstract: Embodiments of the disclosure provide techniques for updating a distributed transaction log on a previously offline resource object component using distributed transaction logs from active host computer nodes from separate RAID mirror configurations. Each component object maintains a journal (log) where distributed transactions are recorded. If a component object goes offline and subsequently returns (e.g., if the node hosting the component object reboots), the component object is marked as stale. To return the component object to an active state, a distributed resources module retrieves the journals from other resource component objects from other RAID configurations where the data is mirrored. The module filters corresponding data that is missing in the journal of the previously offline corresponding object and merges the filtered data to the journal.

Abstract: Embodiments of the disclosure provide techniques for measuring congestion and controlling quality of service to a shared resource. A module that interfaces with the shared resource monitors the usage of the shared resource by accessing clients. Upon detecting that the rate of usage of the shared resource has exceeded a maximum rate supported by the shared resource, the module determines and transmits a congestion metric to clients that are currently attempting to access the shared resource. Clients, in turn determine a delay period based on the congestion metric prior to attempting another access of the shared resource.

Abstract: Embodiments of the disclosure provide techniques for updating a distributed transaction log on a previously offline resource object component using distributed transaction logs from active host computer nodes from separate RAID mirror configurations. Each component object maintains a journal (log) where distributed transactions are recorded. If a component object goes offline and subsequently returns (e.g., if the node hosting the component object reboots), the component object is marked as stale. To return the component object to an active state, a distributed resources module retrieves the journals from other resource component objects from other RAID configurations where the data is mirrored. The module filters corresponding data that is missing in the journal of the previously offline corresponding object and merges the filtered data to the journal.

Abstract: Techniques are disclosed for providing a file system interface for an object store intended to support simultaneous access to objects stored in the object store by multiple clients. In accordance with one method, an abstraction of a root directory to a hierarchical namespace for the object store is exposed to clients. The object store is backed by a plurality of physical storage devices housed in or directly attached to the plurality of host computers and internally tracks its stored objects using a flat namespace that maps unique identifiers to the stored objects. The creation of top-level objects appearing as subdirectories of the root directory is enabled, wherein each top-level object represents a separate abstraction of a storage device having a separate namespace that can be organized in accordance with any designated file system.