Abstract: A technique performs stream-based storage of data. The technique involves receiving, by processing circuitry of data storage equipment, an incoming flow of data. The technique further involves detecting, by the processing circuitry, different data streams within the incoming flow of data. The technique further involves performing, by the processing circuitry, data placement operations based on the different data streams detected within the incoming flow of data. The data placement operations are configured and operative to place data of each data stream of the different data streams in a different segment of storage provided by a data storage array of the data storage equipment. With data of each data stream being placed in a different segment, the resulting operation is more efficient, e.g., optimized sequential reads and writes, more effective data prefetching, more effective auto-tiering of data, and so on.

Abstract: A method is used in managing deduplication of data in storage systems. A candidate data object is identified for deduplicating a data object by evaluating digests stored in a current digest segment to determine whether another digest matching a digest associated with the data block is stored in the current digest segment. The current digest segment includes a set of digests associated with a set of data blocks previously received for deduplication. Based on the evaluation, a deduplicating technique is applied to the data object. The current digest segment is stored in an index table. A previous digest segment associated with a digest stored in the index table matches the digest associated with the data block is replaced by the current digest segment.

Abstract: Techniques for creating snapshots of data storage objects that can perform certain operations (e.g., flushing dirty data, setting up extent pointers, allocating block storage space, etc.) during background (or deferred) processing. The disclosed techniques employ one or more extent copy trackers that can be created during processing of a transaction, while I/O request from host computers are suspended. The extent copy trackers are configured to perform some or all of the certain operations in the background, after the transaction has been committed and/or the processing of the transaction has been completed. By performing such operations during background processing, a processing time required to complete the snapshot transaction is reduced, thereby reducing latency in the resumption of the I/O requests from the host computers.

Abstract: Techniques for amortizing metadata updates due to data delete operations in data storage systems that implement log structured storage of data from virtual volumes. The techniques employ a segment database (DB) and a deleted chunk DB. The segment DB is implemented as a key-value store. The deleted chunk DB is likewise implemented as a key-value store, but configured as a log structured merge (LSM) tree. By configuring the deleted chunk DB as an LSM-tree, more efficient use of memory and improved reduction of metadata updates can be achieved. Stored segments of log structured data can also be effectively “cleaned” in a background process that involves ordered traversals of the segment DB and the deleted chunk DB, allowing for more efficient recovery of storage space consumed by the deleted data chunks.

Abstract: A method is used in managing deduplication of data in storage systems. A candidate data object is identified for deduplicating a data object by evaluating digests stored in a current digest segment to determine whether another digest matching a digest associated with the data block is stored in the current digest segment. The current digest segment includes a set of digests associated with a set of data blocks previously received for deduplication. Based on the evaluation, a deduplicating technique is applied to the data object. The current digest segment is stored in an index table. A previous digest segment associated with a digest stored in the index table matches the digest associated with the data block is replaced by the current digest segment. A plurality of digest segments are organized into a segment group and a reference counter is associated with the segment group, wherein if the reference counter reaches zero, storage space consumed by the digest group is reclaimed.

Abstract: A method is used in managing deduplication of data in storage systems. A digest is determined for a data object received for deduplication. A candidate data object is identified for deduplicating the data object. A digest associated with the candidate data object matches the digest associated with the data object. The digest in a digest segment is maintained based on identification of the candidate data object. The digest segment includes a set of digests associated with a set of data blocks identified for deduplication in an ordered arrangement.

Abstract: Techniques for handling pattern identifiers in a data storage system. By replacing a block pointer with a pattern identifier, the techniques can identify a data block (or an indirect data block) as a bad block, without resorting to the use of a separate flag or bad block (BB) bit in per-block metadata (e.g., a mapping pointer) of the data block. The techniques can also avoid waste of valuable metadata space by using pattern identifiers at various levels of a mapping tree, leveraging pointer granularity at lower levels, mid-levels, and progressively higher levels of the mapping tree.

Abstract: Techniques for handling multiple data streams in stream-aware data storage systems. The data storage systems can detect multiple sub-streams in an incoming stream of data, form a group of data blocks corresponding to each respective sub-stream, and associate, bind, and/or assign a stream ID to each data block in the respective sub-stream. The data storage systems can write each group of data blocks having the same stream ID to the same segment of a data log in one or more non-volatile storage devices, and manage and/or maintain, in persistent data storage, attribute information pertaining to the groups of data blocks in the respective sub-streams relative to time periods during which the respective groups of data blocks were written and/or received. The techniques can improve the detection of multiple sub-streams in an incoming stream of data, and improve the management of attribute information pertaining to data blocks in the respective sub-streams.

Abstract: Techniques for handling multiple data streams in stream-aware data storage systems. The data storage systems can detect multiple sub-streams in an incoming stream of data, form a group of data blocks corresponding to each respective sub-stream, and associate, bind, and/or assign a stream ID to each data block in the respective sub-stream. The data storage systems can write each group of data blocks having the same stream ID to the same segment of a data log in one or more non-volatile storage devices, and manage and/or maintain, in persistent data storage, attribute information pertaining to the groups of data blocks in the respective sub-streams relative to time periods during which the respective groups of data blocks were written and/or received. The techniques can improve the detection of multiple sub-streams in an incoming stream of data, and improve the management of attribute information pertaining to data blocks in the respective sub-streams.

Abstract: Techniques for handling pattern identifiers in a data storage system. By replacing a block pointer with a pattern identifier, the techniques can identify a data block (or an indirect data block) as a bad block, without resorting to the use of a separate flag or bad block (BB) bit in per-block metadata (e.g., a mapping pointer) of the data block. The techniques can also avoid waste of valuable metadata space by using pattern identifiers at various levels of a mapping tree, leveraging pointer granularity at lower levels, mid-levels, and progressively higher levels of the mapping tree.

Abstract: A technique involves, from an incoming flow of data that includes a first stream from a first source and another stream from another source, placing data of the first stream into first storage segments and data of the other stream into other storage segments that are different from the first storage segments. The technique further involves, while some of the data of the first stream becomes invalidated over time and while a garbage collection service consolidates remaining valid data of the first stream together within the first segments, tracking the number of times the remaining valid data of the first stream is consolidated together within the first segments by the garbage collection service. The technique further involves comingling (i) remaining valid data of the first stream which has been consolidated together a predefined number of times within the first segments with (ii) the data of the other stream.

Abstract: A technique for managing metadata in a data storage system designates block pointers as either sources or copies, where sources contribute to reference counts of pointed-to structures but copies do not. The technique maintains parent-child relationships between parent BPSs (block pointer sets) and child BPSs, where each BPS includes an array of block pointers. Each child BPS is created as a copy of a parent BPS and has block pointers initially designated as copies. The technique performs a metadata-merge operation to merge the block pointers of the parent BPS into those of a child BPS by promoting attributes of block pointers in the child BPS from copy to source, avoiding any need to perform reference count updates on structures pointed to by promoted block pointers.

Abstract: A computer-implemented technique performs block deduplication. The technique involves generating a list of candidate blocks and adding classifications to the list of candidate blocks. The classifications classify the candidate blocks on the list into different categories based on a set of block sequence matching criteria. The technique further involves performing, based on the classifications added to the list of candidate blocks, further block deduplication processing on first candidate blocks having a first classification while not performing further block deduplication processing on second candidate blocks having a second classification. Examples of the categories include an “exact long match” category, a “disrupted long match” category, a “loosely long match” category, and a “short match” category.

Abstract: A technique is directed to cleaning a log structure. The technique involves identifying extents (e.g., a contiguous segment of 8 MB) to reclaim from a first storage tier of a set of storage tiers containing the log structure. The technique further involves performing a tier selection operation to select a target storage tier from the set of storage tiers based on a utilization measure of the log structure. The technique further involves, after identifying the extents to reclaim and performing the tier selection operation, storing data from the identified extents into a new extent of the target storage tier and freeing the identified extents. Such a technique combines log cleaning and tiering into a single operation thus placing less stress on storage devices (e.g., less wear on flash memory, etc.), consuming fewer system resources, and providing better performance.

Abstract: A method is used in deduplicating sets of data blocks. A candidate data object is identified for deduplicating a data object. A digest associated with the candidate data object matches a digest associated with the data object. Digest information of a set of data objects is evaluated. The set of data objects are selected for evaluation based on an association between location of the set of data objects and location of the candidate data object. Based on the evaluation, a deduplicating technique is applied for deduplicating the data object.

Abstract: A method is used in managing data deduplication in storage systems. A data deduplication process is performed by applying a deduplicating technique to data of a deduplication domain. The data deduplication process is scheduled based on a priority. Characteristics of data deduplication performed by the data deduplication process are evaluated. Based on the evaluation, execution of the data deduplication process is effected.

Abstract: A method is used in deduplicating sets of data blocks. A collection of data is selected from a set of storage extents. The collection of data is selected based on evaluation of digest information for first and second data blocks of the collection of data. A deduplicating technique is applied to the collection of data.

Abstract: A method is used in managing deduplication density. A deduplication density of a data object is determined. The deduplication density is indicated by a number of deduplicated data objects represented by the data object. Based on the deduplication density of the data object, the data object is skipped for applying a deduplicating technique to the data object. An association is created between the data object and another data object. The other data object is used for applying the deduplicating technique where contents of the other data object are identical to contents of the data object.

Abstract: A method is used in managing deduplication density. A deduplication density of a data object is determined. The deduplication density is indicated by a number of deduplicated data objects represented by the data object. Based on the deduplication density of the data object, a deduplicating technique is applied to the data object.

Abstract: A method is used in deduplicating ranges of data blocks. A collection of data is selected from a set of storage extents. The collection of data is selected based on evaluation of digest information for a subset of the collection of data. A deduplicating technique is applied to the collection of data.