Abstract: Example methods and systems to process input/output (I/O) requests in a distributed storage system in a virtualized computing environment are disclosed. One example method includes executing a first thread to destage one or more data writes, wherein the one or more data writes correspond to a first bucket; executing a second thread to destage the one or more data deletes, wherein the one or more data deletes correspond to a second bucket; in response to executing the first thread, buffering write I/Os associated with the one or more data writes in a logical queue; in response to executing the second thread, buffering delete I/Os associated with the one or more data deletes in the logical queue; and adjusting a number of slots in the logical queue dedicated to buffer the delete I/Os based on a relationship between the first bucket and the second bucket.

Abstract: A method of synchronously executing input/output operations (IOs) for a plurality of applications using a storage device with a file system includes the steps of: receiving a first write IO including an instruction to write first data at a first address of the file system; determining that, within a first range of the file system comprising the first address, there are no pending unmap IOs for deallocating storage space of the storage device from files of the plurality of applications; after determining that there are no pending unmap IOs within the first range, locking the first range to prevent incoming unmap IOs from deallocating storage space within the first range from the files of the plurality of applications; after locking the first range, writing the first data to the storage device at the first address; and after writing the first data, unlocking the first range.

Abstract: A method of synchronously executing input/output operations (IOs) for a plurality of applications using a storage device with a file system includes the steps of: receiving a first write IO including an instruction to write first data at a first address of the file system; determining that, within a first range of the file system comprising the first address, there are no pending unmap IOs for deallocating storage space of the storage device from files of the plurality of applications; after determining that there are no pending unmap IOs within the first range, locking the first range to prevent incoming unmap IOs from deallocating storage space within the first range from the files of the plurality of applications; after locking the first range, writing the first data to the storage device at the first address; and after writing the first data, unlocking the first range.

Abstract: A method of deleting a first pointer block of a plurality of pointer blocks of a file system from a storage device used by a plurality of applications, wherein the plurality of pointer blocks are each subdivided into sub-blocks, includes the steps of: determining that a first sub-block of the first pointer block is marked as being empty of any addresses of the file system at which storage space is allocated to files of the applications; determining that a second sub-block of the first pointer block has not been marked as being empty; in response to the determining that the second sub-block has not been marked as being empty, determining that the second sub-block does not contain any addresses of the file system at which storage space is allocated to the files of the applications; and deleting the first pointer block from the storage device.

Abstract: An example method of handling, at a hypervisor on a host in a virtualized computing system, a write input/output (IO) operation to a file on a storage device having a virtual machine file system (VMFS) is described. The method includes: sorting, at the hypervisor, a scatter-gather array for the write IO operation into sets of scatter-gather elements, each of the sets including at least one scatter-gather element targeting a common file block address; resolving offsets of the sets of scatter-gather elements to identify a first scatter-gather array of transaction-dependent scatter-gather elements; generating logical transactions for the first scatter-gather array having updates to metadata of the VMFS for the file; batching the logical transactions into a physical transaction; and executing the physical transaction to commit the updates to the metadata of the VMFS on the storage device for the file.

Abstract: A method of deleting a first pointer block of a plurality of pointer blocks of a file system from a storage device used by a plurality of applications, wherein the plurality of pointer blocks are each subdivided into sub-blocks, includes the steps of: determining that a first sub-block of the first pointer block is marked as being empty of any addresses of the file system at which storage space is allocated to files of the applications; determining that a second sub-block of the first pointer block has not been marked as being empty; in response to the determining that the second sub-block has not been marked as being empty, determining that the second sub-block does not contain any addresses of the file system at which storage space is allocated to the files of the applications; and deleting the first pointer block from the storage device.

Abstract: A method of synchronously executing input/output operations (IOs) for a plurality of applications using a storage device with a file system includes the steps of: receiving a first write IO including an instruction to write first data at a first address of the file system; determining that, within a first range of the file system comprising the first address, there are no pending unmap IOs for deallocating storage space of the storage device from files of the plurality of applications; after determining that there are no pending unmap IOs within the first range, locking the first range to prevent incoming unmap IOs from deallocating storage space within the first range from the files of the plurality of applications; after locking the first range, writing the first data to the storage device at the first address; and after writing the first data, unlocking the first range.

Abstract: System and method for journaling metadata update transactions of file system operations use logical metadata update transactions to record metadata updates for a target file in response to file system operation requests at a file system of the system. A single physical metadata update transaction is generated by consolidating multiple logical metadata update transactions for the target file. The physical metadata update transaction is then written to a journal area of a physical storage.

Abstract: A method of deleting a first pointer block of a plurality of pointer blocks of a file system from a storage device used by a plurality of applications, wherein the plurality of pointer blocks are each subdivided into sub-blocks, includes the steps of: determining that a first sub-block of the first pointer block is marked as being empty of any addresses of the file system at which storage space is allocated to files of the applications, determining that a second sub-block of the first pointer block has not been marked as being empty; in response to the determining that the second sub-block has not been marked as being empty, determining that the second sub-block does not contain any addresses of the file system at which storage space is allocated to the files of the applications; and deleting the first pointer block from the storage device.

Abstract: An example method of handling, at a hypervisor on a host in a virtualized computing system, a write input/output (IO) operation to a file on a storage device having a virtual machine file system (VMFS) is described. The method includes: generating logical transactions for the write IO operation having updates to metadata of the VMFS for the file; estimating, for the logical transactions, common space reservations for those of the updates to common fields in the metadata for the file; estimating, for the logical transactions, exclusive space reservations for those of the updates to exclusive fields in the metadata for the file; batching the logical transactions into a physical transaction, which includes a single reservation of space in a journal of the VMFS based on the common space reservations and a reservations of space in the journal for each of the exclusive space reservations, respectively.

Abstract: An example method of handling, at a hypervisor on a host in a virtualized computing system, a write input/output (IO) operation to a file on a storage device having a virtual machine file system (VMFS) is described. The method includes: sorting, at the hypervisor, a scatter-gather array for the write IO operation into sets of scatter-gather elements, each of the sets including at least one scatter-gather element targeting a common file block address; resolving offsets of the sets of scatter-gather elements to identify a first scatter-gather array of transaction-dependent scatter-gather elements; generating logical transactions for the first scatter-gather array having updates to metadata of the VMFS for the file; batching the logical transactions into a physical transaction; and executing the physical transaction to commit the updates to the metadata of the VMFS on the storage device for the file.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for reclaiming one or more portions of storage resources in a computer system serving one or more virtual computing instances, where the storage resources in the computer system are organized in clusters of storage blocks. In one aspect, a method includes maintaining a respective block tracking value for each storage block that indicates whether a call to reclaim the storage block is outstanding; determining, from the block tracking values, a respective cluster priority value for each of the clusters based on a count of storage blocks in the respective cluster for which a call to reclaim is outstanding; and reclaiming a first portion of storage resources in the computer system in accordance with the cluster priority values.

Abstract: The disclosure herein describes managing a rate of processing unmap requests for a data storage volume. Unmap requests are received from a cluster of active hosts that are associated with the data storage volume. Latency data values of each active host are then accessed. A long-term cluster latency average value is calculated based on the accessed latency data values of all active hosts over a long-term time period and a short-term cluster latency average value is calculated based on the accessed latency data values of all active hosts over a short-term time period. An unmap rate adjustment value is calculated based on a difference between the long-term cluster latency average value and the short-term cluster latency average value. The rate of processing unmap requests for the data storage volume is adjusted based on the unmap rate adjustment value and the unmap requests are performed based on the adjusted rate.

Abstract: The systems described herein are configured to enhance the efficiency of memory usage and access in a VM file system data store with respect to allocating memory in large and small file block clusters using affinity metadata and propagating and maintaining the affinity metadata in support of the described allocation. In order to maintain affinity metadata of the large file block cluster, affinity generation values stored on the large file block cluster are read and cached affinity generation values for each small file block cluster are read from an in-memory cache associated with the large file block cluster. When the stored affinity generation values and the cached affinity generation values do not match, affinity metadata from all the small file block clusters associated with the large file block cluster is used to update the affinity metadata of the large file block cluster and the associated cache.

Abstract: The disclosure herein describes managing a rate of processing unmap requests for a data storage volume. Unmap requests are received from a cluster of active hosts that are associated with the data storage volume. Latency data values of each active host are then accessed. A long-term cluster latency average value is calculated based on the accessed latency data values of all active hosts over a long-term time period and a short-term cluster latency average value is calculated based on the accessed latency data values of all active hosts over a short-term time period. An unmap rate adjustment value is calculated based on a difference between the long-term cluster latency average value and the short-term cluster latency average value. The rate of processing unmap requests for the data storage volume is adjusted based on the unmap rate adjustment value and the unmap requests are performed based on the adjusted rate.

Abstract: A method is provided for a computer to allocate a resource from a clustered file system (CFS) volume stored on one or more physical storage devices to a file. The CFS volume includes resources organized into resource clusters and the resource clusters make up regions. The method includes, for each region of resource clusters, determining a first count of resources allocated to the host computer and a second count of resources allocated to all other host computers, and calculating a region weight based on the first count and the second count. The method further includes sorting a list of the regions based on their region weights, selecting a region at or near the start of the list, and allocating the resource from a resource cluster in the selected region to the file.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for reclaiming one or more portions of storage resources in a computer system serving one or more virtual computing instances, where the storage resources in the computer system are organized in clusters of storage blocks. In one aspect, a method includes maintaining a respective block tracking value for each storage block that indicates whether a call to reclaim the storage block is outstanding; determining, from the block tracking values, a respective cluster priority value for each of the clusters based on a count of storage blocks in the respective cluster for which a call to reclaim is outstanding; and reclaiming a first portion of storage resources in the computer system in accordance with the cluster priority values.

Abstract: The disclosure herein describes managing a rate of processing unmap requests for a data storage volume. Unmap requests are received from a cluster of active hosts that are associated with the data storage volume. Latency data values of each active host are then accessed. A long-term cluster latency average value is calculated based on the accessed latency data values of all active hosts over a long-term time period and a short-term cluster latency average value is calculated based on the accessed latency data values of all active hosts over a short-term time period. An unmap rate adjustment value is calculated based on a difference between the long-term cluster latency average value and the short-term cluster latency average value. The rate of processing unmap requests for the data storage volume is adjusted based on the unmap rate adjustment value and the unmap requests are performed based on the adjusted rate.

Abstract: The disclosure herein describes managing a rate of processing unmap requests for a data storage volume. Unmap requests are received from a cluster of active hosts that are associated with the data storage volume. Latency data values of each active host are then accessed. A long-term cluster latency average value is calculated based on the accessed latency data values of all active hosts over a long-term time period and a short-term cluster latency average value is calculated based on the accessed latency data values of all active hosts over a short-term time period. An unmap rate adjustment value is calculated based on a difference between the long-term cluster latency average value and the short-term cluster latency average value. The rate of processing unmap requests for the data storage volume is adjusted based on the unmap rate adjustment value and the unmap requests are performed based on the adjusted rate.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for reclaiming one or more portions of storage resources in a computer system serving one or more virtual computing instances, where the storage resources in the computer system are organized in clusters of storage blocks. In one aspect, a method includes maintaining a respective block tracking value for each storage block that indicates whether a call to reclaim the storage block is outstanding; determining, from the block tracking values, a respective cluster priority value for each of the clusters based on a count of storage blocks in the respective cluster for which a call to reclaim is outstanding; and reclaiming a first portion of storage resources in the computer system in accordance with the cluster priority values.