Abstract: Techniques are provided for implementing a unified object format. The unified object format is used to format data in a performance tier (e.g., infrequently accessed data, snapshot data, etc.) into objects that are stored into an object store for low cost, scalable, long term storage compared to storage of the performance tier. With the unified object format, compression of the data may be retained when the data is stored as the objects into the object store. Additional compression may also be provided for the data in the objects. The unified object format includes slot header metadata used to track the location of the data within the object notwithstanding the data being compressed and/or stored at non-fixed boundaries. The slot header metadata may be cached at the performance tier for improved read performance and may be repaired by a repair subsystem (a slot header repair subsystem).

Abstract: A method performed by one or more processing resources of one or more computer systems is disclosed. The method comprises receiving an object at a first of a plurality of nodes from a second of the plurality of storage nodes within a cluster switch fabric, examining a value associated included within the received object, wherein the value is associated with a clock value of the second node and updating a clock operating at the first node with the received value.

Abstract: According to one embodiment, a computer implemented method comprises providing multiple channels between a first storage node and a second storage node with each channel having a separate network connection for packets of a transport layer session, assigning packets from each channel to a group of receive queues of the second storage node, continuously monitoring whether two or more channels of the multiple channels share a same receive queue of the second storage node, and sending a communication via a channel to the first storage node to indicate a dynamic change in a hash input field (e.g., a source port, a destination port, a source internet protocol (IP) address, and a destination IP address) when two or more channels of the multiple channels share a same receive queue of the second storage node.

Abstract: A method performed by one or more processing resources of one or more computer systems is disclosed. The method comprises receiving an object at a first of a plurality of nodes from a second of the plurality of storage nodes within a cluster switch fabric, examining a value associated included within the received object, wherein the value is associated with a clock value of the second node and updating a clock operating at the first node with the received value.

Abstract: Techniques are provided for combining data block and checksum block I/O into a single I/O operation. Many storage systems utilize checksums to verify the integrity of data blocks stored within storage devices managed by a storage stack. However, when a storage system reads a data block from a storage device, a corresponding checksum must also be read to verify integrity of the data in the data block. This results in increased latency because two read operations are being processed through the storage stack and are being executed upon the storage device. To reduce this latency and improve I/O operations per second, a single combined I/O operation corresponding to a contiguous range of blocks including the data block and the checksum block is processed through the storage stack instead of two separate I/O operations. Additionally, I/O operation may be combined into a single request that is executed upon the storage device.

Abstract: The instant disclosure provides a data structure store system and a method of managing data in the store. The method includes receiving, by a data structure store management system, a request for storing data from a client. In the created data structure, each data element includes a portion of the data. On receiving a read request for at least part of the data, the data structure store management system provides at least part of the data to a recipient device. The data elements are stored in persistent memory in the form of one or more non-volatile random access devices, wherein during a time interval between receiving the storing request and providing the at least part of the data by the data structure store management system to the recipient device, the data structure store management system provides no portion of the data for writing to a hard disk drive.

Abstract: Systems and methods for performing single I/O writes are provided. According to one embodiment, responsive to receipt of a write operation from a client by a file system layer of a node of a distributed storage system and a data payload of the operation having been determined to meet a compressibility threshold, an intermediate storage layer of the node logically interposed between the file system layer and a block storage media is caused to perform a single input/output (I/O) write operation that persists the compressed data payload and corresponding metadata to support asynchronous journaling of the write operation. The single I/O write operation coupled with the use of a new pool file that maintains a list of available blocks for single I/O write operations and a modified node crash recovery approach allows the write operation to be acknowledged to the client while the journaling is performed asynchronously.

Abstract: Techniques are provided for maintaining and recomputing reference counts in a persistent memory file system of a node. Primary reference counts are maintained for pages within persistent memory of the node. In response to receiving a first operation to link a page into a persistent memory file system of the persistent memory, a primary reference count of the page is incremented before linking the page into the persistent memory file system. In response to receiving a second operation to unlink the page from the persistent memory file system, the page is unlinked from the persistent memory file system before the primary reference count is decremented. Upon the node recovering from a crash, the persistent memory file system is traversed in order to update shadow reference counts for the pages with correct reference count values, which are used to overwrite the primary reference counts with the correct reference count values.

Abstract: In one embodiment, distributed data storage systems and methods integrate a change tracking manager with scalable databases. According to one embodiment, a computer implemented method comprises integrating change tracking of storage objects into the distributed object storage database that includes a first database of a first type and one or more chapter databases of a second type with the distributed object storage database supporting a primary lookup index and a secondary lookup index in order to locate a storage object. The method includes recording in a header of a chapter database a network topology for connecting a bucket having the chapter database to a first peer bucket when a new mirror to the first peer bucket is being established, and recording a first directive into the header of the chapter database to express a type of content to be mirrored from the bucket to the first peer bucket.

Abstract: Systems and methods are provided for sharing ephemeral storage of a virtual machine (VM) for use as victim caches for virtual storage appliances running on the VM. According to one embodiment, a central service may run within the VM and be responsible for managing allocation and reclamation of ephemeral storage space of the VM to/from the virtual storage appliances. Responsive to startup of a new virtual storage appliance on the VM, the new virtual storage appliance may request space from the central service to inform creation of its victim cache. In connection with servicing the request, the central service may take into consideration various factors including one or more of the total aggregate size of multiple local ephemeral drives associated with the VM, remaining available ephemeral storage space, the number of active virtual storage appliances, and the SLO of the virtual storage appliance seeking to establish its victim cache.

Abstract: Techniques are provided for implementing a persistent key-value store for caching client data, journaling, and/or crash recovery. The persistent key-value store may be hosted as a primary cache that provides read and write access to key-value record pairs stored within the persistent key-value store. The key-value record pairs are stored within multiple chains in the persistent key-value store. Journaling is provided for the persistent key-value store such that incoming key-value record pairs are stored within active chains, and data within frozen chains is written in a distributed manner across distributed storage of a distributed cluster of nodes. If there is a failure within the distributed cluster of nodes, then the persistent key-value store may be reconstructed and used for crash recovery.

Abstract: A system, method, and machine-readable storage medium for retrieving data are provided. In some embodiments, a cache may receive a request for data from a client. The cache may determine that a first subset of the data is stored on a storage device and that a second subset of the data is stored at a cloud address located at a cloud storage endpoint. The cache may also receive from the storage device the first subset of data. The cache further receives from the cloud storage endpoint the second subset of data in response to transmitting a request for the second subset of data stored at the cloud address to the cloud storage endpoint. The cache then transmits to the client the first and second subsets of data from the various sources in response to the data request.

Abstract: Methods, non-transitory machine readable media, and computing devices that provide more efficient hierarchical propagation in tree structures are disclosed. With this technology, a first delta record for a first interior node is created optionally in an atomic transaction along with updating a first tally record for a leaf node based on a first value. The transaction is in response to an action associated with the leaf node and the first interior node is a parent of the leaf node in a hierarchical tree. A timer associated with the first delta record is then set. A second value is updated in a second tally record for the first interior node based on the first value, when the timer has expired. Accordingly, this technology advantageously maintains recursive properties or values throughout a hierarchical tree continually, with reduced cost, even in a distributed network and in hierarchical trees with large numbers of nodes.

Abstract: Methods, non-transitory machine readable media, and computing devices that ensure correctness of file system analytics are disclosed. With this technology, a first generation number for a volume is incremented in response to a modification of a rule set that defines properties of objects of a file system associated with the volume. A determination is made when a second generation number in a first inode for a first one of the objects matches the first generation number. The first inode is identified based on a traversal of a directory tree associated with the file system. The modified rule set is applied to the properties for the first one of the objects to obtain values, when the second generation number fails to match the first generation number. Analytics data is output after the traversal has completed. The analytics data is generated in response to a query and is based on the values.

Abstract: Techniques are provided for caching data during an on-demand restore using a cloud block map. A client may be provided with access to an on-demand volume during a restore process that copies backup data from a snapshot within a remote object store to the on-demand volume stored within local storage. In response to receiving a request from the client for a block of the backup data not yet restored from the snapshot to the on-demand volume, the block may be retrieved from the snapshot in the remote object store. The block may be cached within a cloud block map stored within the local storage as a cached block. The client may be provided with access to the cached block.

Abstract: Systems and methods for making a cross-site storage solution resilient towards mediator unavailability are provided. According to one embodiment, a stretched storage system is operable to bring a mediator associated with a primary and secondary distributed storage system back into the role of an arbitrator for peered consistency groups (CGs). A mediator reseed status indicator is maintained for multiple CGs to identify when the mediator's status information for a CG is stale. When the mediator becomes available and a local CG is identified as the subject of a mediator reseed process, the master node of the primary that hosts a master copy of a dataset for the local CG performs the reseed process, including: (i) causing relationship status information for the local CG to be updated on the mediator to the current state maintained by the primary; and (ii) resetting the mediator reseed status indicator.

Abstract: Techniques are provided for implementing garbage collection and bin synchronization for a distributed storage architecture of worker nodes managing distributed storage composed of bins of blocks. As the distributed storage architecture scales out to accommodate more storage and worker nodes, garbage collection used to free unused blocks becomes unmanageable and slow. Accordingly garbage collection is improved by utilizing heuristics to dynamically speed up or down garbage collection and set sizes for subsets of a bin to process instead of the entire bin. This ensures that garbage collection does not use stale information about what blocks are in-use, and ensures garbage collection does not unduly impact client I/O processing or conversely falls behind on garbage collection. Garbage collection can be incorporated into a bin sync process to improve the efficiency of the bin sync process so that unused blocks are not needlessly copied by the bin sync process.

Abstract: Systems and methods are described for efficiently performing various operations at the granularity of a consistency group (CG) within a cross-site storage solution. An example of one of the various operations includes an independent and parallel resynchronization approach that independently brings individual volumes of a CG to a steady state of in-synchronization (InSync), thereby contributing to scalability of CGs by supporting CGs having a large number of member volumes without requiring a change to the resynchronization process. Another example includes preserving dependent write-order consistency when a remote mirror copy goes out-of-synchronization (OOS) for any reason by driving all member volumes OOS responsive to any member volume becoming OOS. Yet another example includes independent creation of snapshots by member volumes to support efficient and on-demand creation by an application of a common snapshots of all or a subset of peered member volumes of a CG with which the application is associated.

Abstract: Techniques are provided for remote object store error handling. A storage system may store data within one or more tiers of storage, such as a local storage tier (e.g., solid state storage and disks maintained by the storage system), a remote object store (e.g., storage provided by a third party storage provider), and/or other storage tiers. Because the remote object store may not provide the same data consistency and guarantees that the storage system provides for clients such as through the local storage tier, additional validation is provided by the storage system for the remote object store. For example, when data is put into an object of the remote object store, a verification get operation is performed to read and validate information within a header of the object. Other verifications and checks are performed such as using a locally stored metafile to detect corrupt or lost metadata and/or objects.

Abstract: Techniques are provided for implementing garbage collection and bin synchronization for a distributed storage architecture of worker nodes managing distributed storage composed of bins of blocks. As the distributed storage architecture scales out to accommodate more storage and worker nodes, garbage collection used to free unused blocks becomes unmanageable and slow. Accordingly garbage collection is improved by utilizing heuristics to dynamically speed up or down garbage collection and set sizes for subsets of a bin to process instead of the entire bin. This ensures that garbage collection does not use stale information about what blocks are in-use, and ensures garbage collection does not unduly impact client I/O processing or conversely falls behind on garbage collection. Garbage collection can be incorporated into a bin sync process to improve the efficiency of the bin sync process so that unused blocks are not needlessly copied by the bin sync process.