Abstract: Aspects of the subject matter described herein relate to virtualization and offload reads and writes. In aspects, an offload read allows a requestor to obtain a token that represents data while an offload write allows the requestor to request that the data (or a part thereof) represented by a token be logically written. Offload reads and writes may be used to perform various actions for virtual environments.

Abstract: Techniques for recovery and redistribution of data from a virtual disk storage system are described herein. In one or more implementations, a storage scheme derived for a virtual disk configuration is configured to implement various recovery and redistribution designed to improve recovery performance. The storage scheme implements one or more allocation techniques to produce substantially uniform or nearly uniform distributions of data across physical storage devices associated with a virtual disk. The allocation facilitates concurrent regeneration and rebalancing operations for recovery of data in the event of failures. Additionally, the storage scheme is configured to implements parallelization techniques to perform the concurrent operations including but not limited to controlling multiple parallel read/writes during recovery.

Abstract: A set of storage devices may interoperate to share a pool of storage space, such as in a Redundant Array of Inexpensive Disks (RAID) scheme. However, the details of the representation of the pool and the allocation of capacity to the pool may enable advantages and/or impose limitations on the storage set. Presented herein are techniques for generating a representing a pooled partition on one or more storage devices featuring a pool configuration representing the pool as a set of spaces manifested by the pool; a set of storage devices sharing the pool; and a set of extents that map physical areas of the storage devices to logical areas of the spaces. The flexibility of these pooling techniques may enable such features as flexible capacity allocation, delayed binding, thin provisioning, and the participation of a storage device in two or more distinct pools shared with different sets of storage devices.

Abstract: A storage set (e.g., an array of hard disk drives) may experience a failure, such as a loss of power, a software crash, or a disconnection of a storage device, while writes to the storage set are in progress. Recover from the failure may involve scanning the storage set to detect and correct inconsistencies (e.g., comparing mirrors of a data set or testing checksums). However, lacking information about the locations of pending writes to the storage set during the failure, this “cleaning” process may involve scanning the entire storage set, resulting in protracted recovery processes. Presented herein are techniques for tracking writes to the storage set by apportioning the storage set into regions of a region size (e.g., one gigabyte), and storing on the nonvolatile storage medium descriptors of “dirty” regions comprising in-progress writes. The post-failure recovery process may then be limited to the regions identified as dirty.

Abstract: The storage devices of a storage device set (e.g., a RAID array) may generate a nonvolatile representation of the configuration of the storage device set, including logical disks, spaces, storage pools, and layout and provisioning plans, on the physical media of the storage devices. A computer accessing the storage device set may also generate a volatile memory representation of the storage device set to use while accessing the storage devices; however, the nonvolatile representation may not be performant due to its different usage and characteristics. Presented herein are techniques for accessing the storage device set according to a volatile memory representation comprising a hierarchy of logical disks, slabs, and extents, and an accessory comprising a provisioning component that handles slab accesses while applying provisioning plans, and that interfaces with a lower-level layout component that translates slab accesses into storage device accesses while applying layout plans to the storage device set.

Abstract: The re-encryption of data can be performed with independent cryptographic agents that can automatically encrypt and decrypt data in accordance with cryptographic regions, such that data within a single cryptographic region is encrypted and decrypted with the same cryptographic key. An “in-place” re-encryption can be performed by reading data from a chunk in an existing cryptographic region, shrinking the existing cryptographic region past the chunk, expanding a replacement cryptographic region over the chunk, and then writing the data back to the same location, which is now part of the replacement cryptographic region. An “out-of-place” re-encryption can be performed by reading data from a chunk in an existing cryptographic region and then writing the data back to a location immediately adjacent that is part of a replacement cryptographic region. After the re-encrypted data is “shifted”, the cryptographic regions can be expanded and contracted appropriately, and another chunk can be selected.

Abstract: Upon receiving a request to allocate a storage region, a storage device may initialize the contents of the storage device to default values (e.g., zero) in order to avoid problems arising from unknown data stored in the locations of the storage region (e.g., upon writing a data set to a location involved in a mirroring relationship, uninitialized data in the corresponding mirror location may result in a mismatch that jeopardizes the written data). However, initializing the storage device may be time-consuming and inefficient. Instead, a usage bitmap may be generated that, for respective location sets of the storage region, indicates whether values exist in the location. A read request may be fulfilled by examining the usage bitmap to determine whether values exist in the specified location, and if not, the default value may be returned without accessing the storage device. Other efficiencies may also be achieved using the usage bitmap.

Abstract: Upon receiving a request to allocate a storage region, a storage device may initialize the contents of the storage device to default values (e.g., zero) in order to avoid problems arising from unknown data stored in the locations of the storage region (e.g., upon writing a data set to a location involved in a mirroring relationship, uninitialized data in the corresponding mirror location may result in a mismatch that jeopardizes the written data). However, initializing the storage device may be time-consuming and inefficient. Instead, a usage bitmap may be generated that, for respective location sets of the storage region, indicates whether values exist in the location. A read request may be fulfilled by examining the usage bitmap to determine whether values exist in the specified location, and if not, the default value may be returned without accessing the storage device. Other efficiencies may also be achieved using the usage bitmap.

Abstract: A method for managing memory traffic includes causing first data to be written to a data cache memory, where a first write request comprises a partial write and writes the first data to a first portion of the data cache memory, and further includes tracking the number of partial writes in the data cache memory. The method further includes issuing a fill request for one or more partial writes in the data cache memory if the number of partial writes in the data cache memory is greater than a predetermined first threshold.

Abstract: Some implementations may include a virtual storage system to which data is written. The virtual storage system may include a cache and multiple hard drives. Multiple queues may be associated with the multiple hard drives such that each hard drive of the multiple hard drives has a corresponding queue of the multiple queues. A set of candidate rows may be selected from the cache. For each candidate row in the set of candidate rows, destination hard drives may be identified. Each candidate row may be placed in queues corresponding to the destination hard drives. Two or more candidate rows from the multiple queues may be written substantially contemporaneously (e.g., in parallel) to two or more destination hard drives.

Abstract: The representation of storage devices on computers (e.g., as logical volumes) may be complicated by the pooling of multiple storage devices in order to apply redundancy plans such as mirroring and checksumming. Presented herein is a storage device driver configured to operate as a storage device interface generating representations of the storage regions of the storage devices; to claim those regions as a storage controller; and to expose pooled storage regions as logical disks. Additionally, the storage device driver may support the inclusion of storage devices in a cluster, comprising nodes that may be appointed as managers of the storage pool configuration; as managers of the storage devices; as owners having exclusive read/write access to the storage pool or cluster resources; and as cluster resource writers having exclusive write access to a cluster resource. The nodes of the cluster may interoperate to share the storage devices while avoiding write conflicts.

Abstract: Embodiments are directed to creating global, aggregated namespaces for storage management and to providing consistent namespaces in a distributed storage system. In one scenario, a computer system defines data storage objects for each data storage node. The data storage objects uniquely identify storage elements of the data storage nodes, where each data storage object includes various associated attributes. The computer system replicates the defined data storage objects and any associated attributes from a first data storage node to a second, different data storage node among the data storage nodes. As such, the defined data storage objects are visible from any node in the data storage nodes. The computer system also aggregates the defined data storage objects for each of the data storage nodes and creates a global, aggregated namespace that includes the aggregated data storage objects for each of the data storage nodes.

Abstract: One embodiment of the present invention sets forth a compression status cache configured to store compression information for blocks of memory stored within an external memory. A data cache unit is configured to request, in response to a cache miss, compressed data from the external memory based on compression information stored in the compression status bit cache. The compression status for active buffers is dynamically swapped into the compression status cache as needed. Different compression formats may be specified for one or more tiles within an active buffer. One advantage of the disclosed compression status cache is that a lame amount of attached memory may be allocated as compressible memory blocks, without incurring a corresponding die area cost because a portion of the compression status stored off chip in attached memory is cached in the compression status cache.

Abstract: The storage devices of a storage device set (e.g., a RAID array) may generate a nonvolatile representation of the configuration of the storage device set, including logical disks, spaces, storage pools, and layout and provisioning plans, on the physical media of the storage devices. A computer accessing the storage device set may also generate a volatile memory representation of the storage device set to use while accessing the storage devices; however, the nonvolatile representation may not be performant due to its different usage and characteristics. Presented herein are techniques for accessing the storage device set according to a volatile memory representation comprising a hierarchy of logical disks, slabs, and extents, and an accessor comprising a provisioning component that handles slab accesses while applying provisioning plans, and that interfaces with a lower-level layout component that translates slab accesses into storage device accesses while applying layout plans to the storage device set.

Abstract: A thinly provisioned storage system detects whether physical storage capacity is available when there is a request to allocate storage capacity, prior to data being written to the storage system. In particular, at the time when the file system allocates storage, such as when creating a file or performing an extending write (append) operation, allocating storage to an unallocated region of a sparse file, defragmenting a file, and the like, a storage system can verify that actual physical storage capacity is available. Thus, if there is insufficient actual physical capacity at the time when a storage allocation is attempted, then an error message can be sent and remedial action can be taken.

Abstract: The provisioning of a volume that has multiple tiers corresponding to different trait sets. The volume to be provisioned is identified along with multiple tiers that are to be in the volume. For each of the tiers that are to be provisioned within the volume, a corresponding trait set is identified as to be applied to each tier. This corresponding trait set may be based on underlying storage systems that are available at the time of provisioning, or which are anticipated to be available. The volume is then caused to be provisioned with the corresponding tiers having the corresponding trait sets. Also, the provisioning of a file, which is determined to have one or more storage traits. Based on these storage traits, the file is then caused to be assigned to an appropriate tier.

Abstract: Techniques for recovery and redistribution of data from a virtual disk storage system are described herein. In one or more implementations, a storage scheme derived for a virtual disk configuration is configured to implement various recovery and redistribution designed to improve recovery performance. The storage scheme implements one or more allocation techniques to produce substantially uniform or nearly uniform distributions of data across physical storage devices associated with a virtual disk. The allocation facilitates concurrent regeneration and rebalancing operations for recovery of data in the event of failures. Additionally, the storage scheme is configured to implements parallelization techniques to perform the concurrent operations including but not limited to controlling multiple parallel read/writes during recovery.

Abstract: Resiliency techniques for a virtual disk are described that enable user control over storage efficiency and recovery time. Configuration parameters for a virtual disk are obtained that indicate a number of available storage devices and a specified tolerance for storage device failures. A default configuration for the virtual disk that designates a default amount of redundancy data to store with client data to balance storage efficiency and recovery time is derived based on the configuration parameters. Options may then be provided to specify a custom configuration that changes the amount of redundancy data to customize the level of storage efficiency and recovery time. The virtual disk is configured and data is stored thereon in accordance with the default configuration or the custom configuration as directed by the user.

Abstract: A system in which a file system may operate on a volume in which the logical address extent of the volume is divided into multiple tiers, each tier providing storage having a distinct trait set by mapping the logical addresses of the volume to appropriate underlying storage systems. A volume system exposes the volume to the file system in a manner that the file system itself has awareness of the tiers, and is aware of the trait sets of each tier. The file system may thus store file system namespaces (such as directories and files) into the tiers as appropriate for the file system namespace. A provisioning system may also be provided and be configured to provision the volume to include such tiers, and if desired, to extend the tiers.

Abstract: A volume system that presents a volume having an extent of logical addresses to a file system. A volume exposure system exposes the volume to the file system in a manner that the volume has multiple tiers, each offering storage of different traits. This is performed using multiple heterogenic underlying storage systems, each having different storage system-specific traits. Each underlying storage system may be hardware, software, or a combination thereof that permits each storage system to expose storage having the particular storage system-specific traits to the file system. The volume system supports each tier by mapping logical addresses of the tier to portions of underling storage systems that are consistent with the tier traits.