Abstract: Aspects of the subject matter described herein relate to storage allocation. In aspects, a hierarchical data structure is used to track allocation data for storage managed by a file system. The hierarchical data structure may have multiple levels with each level having data regarding a different granularity of storage. Portions of the hierarchical data structure may be locked independently of other portions of the hierarchical data structure. The hierarchical data structure may indicate that one or more portions of storage are for exclusive use by a directory. Extra space may be reserved in allocated space in anticipation of subsequent operations. Allocation requestors may obtain storage allocation from regions associated with different levels of the hierarchical data structure.

Abstract: Embodiments are directed to efficiently managing data storage and efficiently storing data. In one scenario, a computer system receives a write request addressed to one portion of data storage within a data store. The computer system redirects the received write request to another portion of data storage which includes storage containers, each of which is a logically specified portion of the data store. The computer system determines which storage container the write request is to write to, and identifies a storage container that is to be de-staged from the second portion of data storage to the first portion of data storage. The computer system then de-stages the identified storage containers to the first portion of data storage, the data of the de-staged container being stored on the first portion of data storage in the same order in which the data was stored in the identified storage container prior to de-staging.

Abstract: Embodiments are directed to modifying storage capacity within a data store and to modifying resiliency for a data store. In one scenario, a computer system receives a request to move data. The computer system may determine that data is to be moved from an allocation on one data store to a new allocation on another data store. The computer system may create a new allocation on the other data store, where the new allocation is configured to receive data from the first data store. The computer system then moves the data to the new allocation on the second data store as data I/O requests are received at the first data store. Data store access requests are synchronized with the data movement by directing the data store access requests to the first data store, to the second data store or to both data stores depending on the type of access request.

Abstract: To provide an operation that can quickly copy or rearrange a file, a file system allows stored data to be shared among two or more files or within a file. The file system provides an operation that allows an extent from one file to be copied into an extent of another file, or into a different extent in the same file if the extents do not overlap. To support such an operation, the file system tracks the storage locations for stored data for extents of a file, and whether the stored data for an extent of a file is shared with any other file or by another portion of the same file. When stored data is shared, writing to an extent of a file that refers to the shared stored data invokes a write operation to a new extent on the storage device, and the tracked information for the affected extents is updated.

Abstract: A thinly provisioned storage system detects whether physical storage capacity is available when there is a request to allocate storage capacity, prior to data being written to the storage system. In particular, at the time when the file system allocates storage, such as when creating a file or performing an extending write (append) operation, allocating storage to an unallocated region of a sparse file, defragmenting a file, and the like, a storage system can verify that actual physical storage capacity is available. Thus, if there is insufficient actual physical capacity at the time when a storage allocation is attempted, then an error message can be sent and remedial action can be taken.

Abstract: The storage of data sets in a storage set (e.g., data sets written to hard disk drives comprising a RAID array) may diminish the performance of the storage set through non-sequential writes, particularly if the storage devices promptly write data sets that are followed by sequentially following data sets. Additionally, storage sets may exhibit inconsistencies due to non-atomic writes of data sets and verifiers (e.g., checksums) and an intervening failure, such as an occurrence of the RAID write hole. Instead, data sets and verifiers may first be written to a stored on the nonvolatile media of a storage device before being committed to the storage set. Such writes may be sequentially written to the journal, irrespective of the locations of the data sets in the storage set; and recovery of a failure may simply involve re-committing the consistent records in the journal to correct incomplete writes to the storage set.

Abstract: Aspects of the subject matter described herein relate to storage allocation. In aspects, a hierarchical data structure is used to track allocation data for storage managed by a file system. The hierarchical data structure may have multiple levels with each level having data regarding a different granularity of storage. Portions of the hierarchical data structure may be locked independently of other portions of the hierarchical data structure. The hierarchical data structure may indicate that one or more portions of storage are for exclusive use by a directory. Extra space may be reserved in allocated space in anticipation of subsequent operations. Allocation requestors may obtain storage allocation from regions associated with different levels of the hierarchical data structure.

Abstract: A file system uses data integrity techniques that are a selectable attribute of a file system object. Some file system objects have integrity on for various accesses, while other file system objects do not. Different default settings can be provided for different file system objects. Such a setting for a file system object can be changed to and from having integrity on. Given a file system object with an attribute, the file system provides file system operations for which the data integrity operation used on the file system object depends on this attribute. Such operations include, but are not limited to, operations for changing the attribute, creating file system objects with such attributes, providing and changing default settings for such attributes, and writing data to and reading data from files, which use different data integrity techniques based on this attribute.

Abstract: Resiliency techniques for a virtual disk are described that enable user control over storage efficiency and recovery time. Configuration parameters for a virtual disk are obtained that indicate a number of available storage devices and a specified tolerance for storage device failures. A default configuration for the virtual disk that designates a default amount of redundancy data to store with client data to balance storage efficiency and recovery time is derived based on the configuration parameters. Options may then be provided to specify a custom configuration that changes the amount of redundancy data to customize the level of storage efficiency and recovery time. The virtual disk is configured and data is stored thereon in accordance with the default configuration or the custom configuration as directed by the user.

Abstract: A thinly provisioned storage system detects whether physical storage capacity is available when there is a request to allocate storage capacity, prior to data being written to the storage system. In particular, at the time when the file system allocates storage, such as when creating a file or performing an extending write (append) operation, allocating storage to an unallocated region of a sparse file, defragmenting a file, and the like, a storage system can verify that actual physical storage capacity is available. Thus, if there is insufficient actual physical capacity at the time when a storage allocation is attempted, then an error message can be sent and remedial action can be taken.

Abstract: Providing for a paradigm shift in block-level abstraction for storage devices is described herein. At a block-level, storage is characterized as a variable size data record, rather than a fixed size sector. In some aspects, the variable size data record can comprise a variable binary key-data pair, for addressing and identifying a variable size block of data, and for dynamically specifying the size of such block in terms of data storage. By changing the key or data values, the location, identity or size of block-level storage can be modified. Data records can be passed to and from the storage device to facilitate operational commands over ranges of such records. Block-level data compression, space management and transactional operations are provided, mitigating a need of higher level systems to characterize underlying data storage for implementation of such operations.

Abstract: Aspects of the subject matter described herein relate to maintaining consistency in a storage system. In aspects, one or more objects may be updated in the context of a transaction. In conjunction with updating the objects, logical copies of the objects may be obtained and modified. A request to write the updated logical copies is sent to a storage controller. The logical copies do not overwrite the original copies. In conjunction with sending the request, a data structure is provided for the storage controller to store on the disk. The data structure indicates the one or more objects that were supposed to be written to disk and may include verification data to indicate the content that was supposed to be written to disk. During recovery, this data structure may be used to determine whether all of the object(s) were correctly written to disk.

Abstract: Techniques for recovery and redistribution of data from a virtual disk storage system are described herein. In one or more implementations, a storage scheme derived for a virtual disk configuration is configured to implement various recovery and redistribution designed to improve recovery performance. The storage scheme implements one or more allocation techniques to produce substantially uniform or nearly uniform distributions of data across physical storage devices associated with a virtual disk. The allocation facilitates concurrent regeneration and rebalancing operations for recovery of data in the event of failures. Additionally, the storage scheme is configured to implements parallelization techniques to perform the concurrent operations including but not limited to controlling multiple parallel read/writes during recovery.

Abstract: Aspects of the subject matter described herein relate to maintaining consistency in a storage system. In aspects, one or more objects may be updated in the context of a transaction. In conjunction with updating the objects, logical copies of the objects may be obtained and modified. A request to write the updated logical copies is sent to a storage controller. The logical copies do not overwrite the original copies. In conjunction with sending the request, a data structure is provided for the storage controller to store on the disk. The data structure indicates the one or more objects that were supposed to be written to disk and may include verification data to indicate the content that was supposed to be written to disk. During recovery, this data structure may be used to determine whether all of the object(s) were correctly written to disk.

Abstract: Aspects of the subject matter described herein relate to error detection for files. In aspects, before allowing updates to a clean file, a flag marking the file as dirty is written to non-volatile storage. Thereafter, the file may be updated as long as desired. Periodically or at some other time, the file may be marked as clean after all outstanding updates to the file and error codes associated with the file are written to storage. While waiting for outstanding updates and error codes to be written to storage, if additional requests to update the file are received, the file may be marked as dirty again prior to allowing the additional requests to update the file. The request to write a clean flag regarding the file may be done lazily.

Abstract: Upon receiving a request to allocate a storage region, a storage device may initialize the contents of the storage device to default values (e.g., zero) in order to avoid problems arising from unknown data stored in the locations of the storage region (e.g., upon writing a data set to a location involved in a mirroring relationship, uninitialized data in the corresponding mirror location may result in a mismatch that jeopardizes the written data). However, initializing the storage device may be time-consuming and inefficient. Instead, a usage bitmap may be generated that, for respective location sets of the storage region, indicates whether values exist in the location. A read request may be fulfilled by examining the usage bitmap to determine whether values exist in the specified location, and if not, the default value may be returned without accessing the storage device. Other efficiencies may also be achieved using the usage bitmap.

Abstract: Upon receiving a request to allocate a storage region, a storage device may initialize the contents of the storage device to default values (e.g., zero) in order to avoid problems arising from unknown data stored in the locations of the storage region (e.g., upon writing a data set to a location involved in a mirroring relationship, uninitialized data in the corresponding mirror location may result in a mismatch that jeopardizes the written data). However, initializing the storage device may be time-consuming and inefficient. Instead, a usage bitmap may be generated that, for respective location sets of the storage region, indicates whether values exist in the location. A read request may be fulfilled by examining the usage bitmap to determine whether values exist in the specified location, and if not, the default value may be returned without accessing the storage device. Other efficiencies may also be achieved using the usage bitmap.

Abstract: A set of application programming interfaces (“APIs”) is provided that enables an application to perform operations on multiple system resources as a single logical unit of work through a transaction. The application can then commit or roll back the entire group of changes as a single unit in a coordinated manner. The APIs expose functions and methods that take a reference to a transaction context, such as a handle, name, or pointer, as one of their parameters so that the application can manipulate the resource as a transacted operation. The transaction is bound to all created handles so that all operations on the resource using those handles are also transacted. In an illustrative example, the set of APIs are transacted name-based WIN32 APIs that take a transaction handle. The transacted APIs expose transacted operations to the application for durable system resources in the OS kernel, including the NTFS file system (New Technology File System) and registry.

Abstract: Aspects of the subject matter described herein relate to storage systems and aliased memory. In aspects, a file system driver or other component may send a request to a memory controller to create an alias between two blocks of memory. One of the blocks of memory may be used for main memory while the other of the blocks of memory may be used for a storage system. In response, the memory controller may create an alias between the blocks of memory. Until the alias is severed, when the memory controller receives a request for data from the block in main memory, the memory controller may respond with data from the memory block used for the storage system. The memory controller may also implement other actions as described herein.

Abstract: Aspects of the subject matter described herein relate to error detection for files. In aspects, before allowing updates to a clean file, a flag marking the file as dirty is written to non-volatile storage. Thereafter, the file may be updated as long as desired. Periodically or at some other time, the file may be marked as clean after all outstanding updates to the file and error codes associated with the file are written to storage. While waiting for outstanding updates and error codes to be written to storage, if additional requests to update the file are received, the file may be marked as dirty again prior to allowing the additional requests to update the file. The request to write a clean flag regarding the file may be done lazily.