Abstract: A system and method for improving storage system performance by maintaining data integrity during bulk export to a cloud system is provided. A backup host reads a selected volume from the storage system via an I/O channel. The storage system remains online during bulk export and tracks I/O to the selected volume in a tracking log. The backup host compresses, encrypts, and calculates a checksum for each data block of the volume before writing a corresponding data object to export devices and sending a checksum data object to the cloud system. The devices are shipped to the cloud system, which imports the data objects and calculates a checksum for each. The storage system compares the imported checksums with the checksums in the checksum data object, and adds data blocks to the tracking log when errors are detected. An incremental backup is performed based on the contents of the tracking log.

Abstract: A method, non-transitory computer readable medium, and device that assists with reducing initialization duration and performance impact during configuration of storage drives includes identifying a plurality of new storage drives in a storage system. Next, one or more zeroed out storage drives is identified from the identified plurality of new storage drives based on information present in a data portion of each the identified plurality of new storage drives. A volume group comprising the identified one or more zeroed out drives is created and this created volume group is provided for data operation.

Abstract: Systems and techniques for recovering a storage array are disclosed. These systems and techniques include determining a size corresponding to a storage stripe of the storage array. Pieces assigned to the storage stripe are identified. A storage configuration corresponding to the pieces assigned to the storage stripe is detected. Ordinal information and parity information are determined corresponding to the pieces assigned to the storage stripe. The size determined corresponding to the storage stripe, identification of the pieces assigned to the storage stripe, the storage configuration, the ordinal information, and the parity information is stored in a data store to reconstruct lost or corrupted metadata corresponding to the storage array.

Abstract: A system shares I/O load between controllers in a high availability system. For writes, a controller determines based on one or more factors which controller will flush batches of data from write-back cache to better distribute the I/O burden. The determination occurs after the local storage controller caches the data, mirrors it, and confirms write complete to the host. Once it is determined which storage controller will flush the cache, the flush occurs and the corresponding metadata at a second layer of indirection is updated by that determined storage controller (whether or not it is identified as the owner of the corresponding volume to the host, while the volume owner updates metadata at a first layer of indirection). For a host read, the controller that owns the volume accesses the metadata from whichever controller(s) flushed the data previously and reads the data, regardless of which controller had performed the flush.

Abstract: A system and method for improving storage system performance by maintaining data integrity during bulk export to a cloud system is provided. A backup host reads a selected volume from the storage system via an I/O channel. The storage system remains online during bulk export and tracks I/O to the selected volume in a tracking log. The backup host compresses, encrypts, and calculates a checksum for each data block of the volume before writing a corresponding data object to export devices and sending a checksum data object to the cloud system. The devices are shipped to the cloud system, which imports the data objects and calculates a checksum for each. The storage system compares the imported checksums with the checksums in the checksum data object, and adds data blocks to the tracking log when errors are detected. An incremental backup is performed based on the contents of the tracking log.

Abstract: A method, a computing device, and a non-transitory machine-readable medium for allocating data compression activities in a storage system are provided. A method includes tracking, by a storage controller, computing resources corresponding to a storage server. The storage controller processes one or more host read requests to access data requested by one or more hosts, the processing of the one or more host read requests including decompressing the data requested by the one or more hosts from the storage server and providing the decompressed data to the one or more hosts. The storage controller determines an amount of available computing resources after processing the one or more host read requests. Based on the amount of available computing resources, the storage controller performs inline compression of a first portion of host write requests and background compression of a second portion of the host write requests.

Abstract: A system and method for improving storage system performance by maintaining data integrity during bulk export to a cloud system is provided. A backup host reads a selected volume from the storage system via an I/O channel. The storage system remains online during bulk export and tracks I/O to the selected volume in a tracking log. The backup host compresses, encrypts, and calculates a checksum for each data block of the volume before writing a corresponding data object to export devices and sending a checksum data object to the cloud system. The devices are shipped to the cloud system, which imports the data objects and calculates a checksum for each. The storage system compares the imported checksums with the checksums in the checksum data object, and adds data blocks to the tracking log when errors are detected. An incremental backup is performed based on the contents of the tracking log.

Abstract: A method, a computing device, and a non-transitory machine-readable medium for allocating memory to data structures that map a first address space to a second is provided. In some embodiments, the method includes identifying, by a storage system, a pool of memory resources to allocate among a plurality of address maps. Each of the plurality of address maps includes at least one entry that maps an address in a first address space to an address in a second address space. An activity metric is determined for each of the plurality of address maps, and a portion of the pool of memory is allocated to each of the plurality of address maps based on the respective activity metric. The allocating of the portion of the memory pool to a first map may be performed in response to a merge operation being performed on the first map.

Abstract: A method for mapping a first address space to a second address space is provided. In some embodiments, the method includes creating a first array of lookup entries and one or more second arrays of metadata entries for maintaining an ordering among the lookup entries using a tree structure. Each of the metadata entries includes one or more data index values identifying a corresponding one of the lookup entries by its position in the first array and one or more metadata index values identifying a corresponding one of the metadata entries by its position in one of the one or more second arrays. The method further includes receiving a request including a lookup value, traversing the tree structure to locate a lookup entry corresponding to the lookup value, and when the lookup value is located among the lookup entries, using the located lookup entry to process the request.

Abstract: Systems and techniques for recovering a storage array are disclosed. These systems and techniques include determining a size corresponding to a storage stripe of the storage array. Pieces assigned to the storage stripe are identified. A storage configuration corresponding to the pieces assigned to the storage stripe is detected. Ordinal information and parity information are determined corresponding to the pieces assigned to the storage stripe. The size determined corresponding to the storage stripe, identification of the pieces assigned to the storage stripe, the storage configuration, the ordinal information, and the parity information is stored in a data store to reconstruct lost or corrupted metadata corresponding to the storage array.

Abstract: A method includes: storing a first data extent on a physical medium, wherein the physical medium is divided into a plurality of storage blocks, wherein each of the storage blocks has a size that is different than a size of the first data extent, further wherein the first data extent is stored to a first block of the plurality of storage blocks; generating a descriptor for the first data extent, wherein the descriptor indicates that the first data extent starts within the first block of the plurality of blocks and indicates an offset from the beginning of the first block at which the first data extent starts; and storing the descriptor within the first block.

Abstract: A system for tracking metadata changes and recovering from system interruptions is provided. With host I/O, corresponding metadata incremental changes are aggregated and stored in a write-ahead log before being performed to their in-memory buffers. As those buffers are flushed, checkpoints are created and stored in the log. As the log wraps to the start, older entries are overwritten after they are freed from any remaining dependencies by newer checkpoints. If metadata entities have not created new checkpoints, they are instructed to in order to free up space for new aggregated batches and checkpoints. After an interruption, the wrap point is located in the log. From the wrap point, the log is scanned backwards to provide checkpoints to metadata entities. The log is then scanned forwards to perform changes specified by aggregated batches. The metadata entities' volatile memory states are recovered to what they were before the interruption.

Abstract: A system and method for improving storage system performance by maintaining data integrity during bulk export to a cloud system is provided. A backup host reads a selected volume from the storage system via an I/O channel. The storage system remains online during bulk export and tracks I/O to the selected volume in a tracking log. The backup host compresses, encrypts, and calculates a checksum for each data block of the volume before writing a corresponding data object to export devices and sending a checksum data object to the cloud system. The devices are shipped to the cloud system, which imports the data objects and calculates a checksum for each. The storage system compares the imported checksums with the checksums in the checksum data object, and adds data blocks to the tracking log when errors are detected. An incremental backup is performed based on the contents of the tracking log.

Abstract: A system and method for recovering a dataset is provided that analyzes the dataset as it currently exists in order to determine those portions that do not need to be recovered. In some embodiments, the method includes identifying a dataset stored on a set of storage devices and corresponding to a first point in time. A request to restore the dataset to a second point in time is received, and a subset of the dataset is identified that is different between the first point in time and the second point in time. Data associated with the subset is selectively retrieved that corresponds to the second point in time, and the retrieved data is merged with the dataset stored on the set of storage devices. The two points in time may have any relationship, and in various examples, the method performs a roll-back or a roll-forward of the dataset.

Abstract: A system and method are provided for backing up and recovering data that allows the data to be modified and backed up even while recovery is still in progress. In some embodiments, the method includes performing a data recovery procedure on a computing system. The data recovery procedure includes identifying a set of data objects stored on a recovery system; retrieving the set of data objects; and storing data of the retrieved set of data objects to at least one storage device. Data objects may be prioritized so that data that is in demand is retrieved first. Data that is modified during the data recovery procedure is tracked and backed up to an object-storage system during the data recovery procedure. In some embodiments, backing up the modified data is part of an incremental backup procedure that excludes data objects that contains only unmodified data.

Abstract: A system, method, and computer program product for the provision of a snapshot repository that grows and shrinks over time. The snapshot repository stores snapshot images. When the used capacity of the repository reaches an upper bound of the repository, the image may be wrapped to the start of the repository if the image stored there previously has been deleted. Otherwise another stripe volume is added from a pool of available volumes and the image continues storage there. As older images are deleted according to a retention policy, stripe volumes that become empty may be dis-associated from their associated LBA range and released to the pool or re-mapped to a lower LBA range of the repository. When the empty stripe volume is at the end of the repository, the ending LBA may be adjusted to the end of the last allocated stripe volume as the volume is released.

Abstract: A system, method, and computer program product for the provision of an elastic snapshot repository is disclosed. A snapshot repository with a particular size stores snapshot images. As the used capacity of the snapshot repository exceeds a predetermined threshold, another volume is added from a pool of available volumes. When the used capacity of the snapshot repository is at, or falls below, a lower threshold, a second snapshot repository is created. The schedule associated with the first snapshot repository is transferred to the second snapshot repository. The snapshot images in the first snapshot repository remain available to meet a minimum history requirement. New snapshot images are stored to the second snapshot repository until there are enough snapshot images in the second snapshot repository, alone, to meet the minimum history requirement. The first snapshot repository is deleted in response and the associated volumes released to the pool.

Abstract: An I/O processing stack includes a proxy that can provide processing services for access requests to initialized and uninitialized storage regions. For a write request, the proxy stores write information in a write metadata repository. If the write is requested for an address in an initialized storage region of the storage system, the proxy performs a write to the initialized region based on region information in the write I/O access request. If the write is requested for an address in an uninitialized storage region of the storage system, the proxy performs an on-demand initialization of the storage region and then performs a write to the storage region based on region information provided by the proxy.

Abstract: An I/O processing stack includes a proxy that can provide processing services for access requests to initialized and uninitialized storage regions. For a write request, the proxy stores write information in a write metadata repository. If the write is requested for an address in an initialized storage region of the storage system, the proxy performs a write to the initialized region based on region information in the write I/O access request. If the write is requested for an address in an uninitialized storage region of the storage system, the proxy performs an on-demand initialization of the storage region and then performs a write to the storage region based on region information provided by the proxy.

Abstract: A method for controlling write order in a remote data storage system used for asynchronous replication. The data backup method includes receiving writes issued concurrently by a host prior to any being completed. An async write manager, such as a software application on a storage controller or on a host, operates to group each of the writes into a concurrency group with a replica write corresponding to each of the host writes being queued for transmittal over a link to a remote data storage system. The replica writes each include a concurrency group identifier, such as a group number or last completed write number, that identifies which concurrency group the replica has been assigned. The method includes transferring the replica writes to the remote data storage system and processing the concurrency group identifiers with a write order manager to determine which subsets of the replica writes to issue concurrently.