Abstract: Techniques for managing data blocks involve: generating, based on a first hash algorithm, a first fingerprint for a first block. The techniques further involve: if it is determined that there is a second fingerprint, in a fingerprint database, that is generated for a second block based on the first hash algorithm and matches the first fingerprint, determining whether there is a third fingerprint, in the fingerprint database, that is generated for the second block based on a second hash algorithm. The techniques further involve: if it is determined that the third fingerprint exists in the fingerprint database, generating a fourth fingerprint for the first block based on the second hash algorithm; and determining whether the first block and the second block are duplicate by comparing the third fingerprint and the fourth fingerprint. Such techniques can effectively reduce the overhead of identifying duplicate data blocks in data deduplication.

Abstract: A memory sub-system can determine a block granularity for an input/output (I/O) data stream received from a host system. The memory sub-system can determine that the block granularity is different than a memory block granularity of a first memory region in a first namespace of the one or more memory devices, where the first memory region is to store the I/O data stream. The memory sub-system can accumulate blocks from the I/O data stream in a second memory region in a second namespace of the one or more memory devices. Responsive to a capacity of the accumulated blocks in the second memory region satisfying a threshold criterion, the memory sub-system can migrate the accumulated plurality of blocks from the second memory region to the first memory region.

Abstract: A memory system includes a delay time determining unit and a recovering unit. The delay time determining unit determines an amount of a delay time for delaying a recovery operation upon an occurrence of an abnormal power event in the memory system. The recovering unit is in communication with the delay time determining unit to receive information related to the delay time and performs the recovery operation after lapse of an amount of time equal to the delay time.

Abstract: A method, computer system, and a computer program product for providing backend deduplication awareness at a virtualizing layer is disclosed. The present invention may include receiving a deduplication information from a backend storage controller associated with performing an input/output (IO) operation to a physical address of a disk. The present invention may include translating the physical address to a logical address to apply to a plurality of storage extents in a virtualizing layer. The present invention may include constructing a graph including corresponding nodes representing the plurality of storage extents and including corresponding deduplication edge weights representing a plurality of deduplications between the plurality of storage extents. The present invention may include identifying at least one subgraph within the constructed graph, wherein the identified at least one subgraph represents a storage extent cluster that is suitable for garbage collection as a cluster.

Abstract: There is a need for more effective and efficient constrained-optimization-based operational load balancing. In one example, a method comprises determining constraint-satisfying operator-unit mapping arrangements that satisfy an operator unity constraint and an operator capacity constraint; for each constraint-satisfying operator-unit mapping arrangement, determining an arrangement utility measure; processing each arrangement utility measure using an optimization-based ensemble machine learning model that is configured to determine an optimal operator-unit mapping arrangement of the plurality of constraint-satisfying operator-unit mapping arrangements; and initiating the performance of one or more operational load balancing operations based on the optimal operator-unit mapping arrangement.

Abstract: An embodiment of an electronic apparatus may include one or more substrates, and logic coupled to the one or more substrates, the logic to control access to two or more persistent storage devices, wherein at least one of the two or more persistent storage devices is to be logically organized with a namespace divided into two or more zones, determine two or more different zone size dependent parameters associated with the two or more persistent storage devices, determine a smallest aligned boundary based on each of the two or more different zone size dependent parameters, and set a zone group size for access to the two or more persistent storage devices based on the determined smallest aligned boundary. Other embodiments are disclosed and claimed.

Abstract: Systems and methods for processing copy commands are disclosed. A first controller of the storage device receives a copy command from a host via a first queue. The storage device generates, based on the copy command, a read command and a write command, and submits the read and write commands to a second controller of the storage device via a second queue. The second controller retrieves and processes the read and write commands from the second queue. The storage device reads, based on the processing of the read command, data stored in a first location of a storage media associated with a source address, and writes the data to a second location of the storage media associated with a destination address. The first controller transmits a signal to the host for indicating completion of the copy command.

Abstract: In a storage system including a first tier and a second tier a method includes: storing access statistics per object; obtaining a request to perform a write operation; calculating a recency factor to the first object based on the access statistics; and writing the first object to one of the first tier and the second tier, depending on the recency factor. Performing garbage collection process on the second tier may include: reading metadata of an object stored in the second tier; determining whether the object is valid based on the metadata; if the object is invalid, discarding the object; and if the second object is valid: calculating a recency factor for the object based on the access statistics of the object; and moving the object to the first tier or leaving the object in the second tier, depending on the recency factor of the second object.

Abstract: A memory system may include: a nonvolatile memory device; a volatile memory suitable for storing write data; and a controller suitable for: allocating a normal write buffer in the volatile memory when normal write data are inputted, allocating a first write buffer in the volatile memory when first write data, which are grouped into a first transaction and first total size information on a total size of the first transaction, are inputted, allocating a second write buffer in the volatile memory when second write data, which are grouped into a second transaction and second total size information on a total size of the second transaction, are inputted, managing sizes of the first and second write buffers to change them in response to the first and second total size information, respectively, and managing a size of the normal write buffer to fix it to a set size.

Abstract: A system and method for fast communication between processors on complex chips using encoding, wherein a training data set is used to find patterns and associated smaller indices, or codewords, which are stored in a reference codebook library, and where reconstruction and deconstruction algorithms are used to encode and decode data as it is received. The codebook and algorithms may be stored in the firmware of a semiconductor which enable reduced resources and cost when transmitting data between or among devices that utilize such semiconductors.

Abstract: A method of operating a storage system having solid-state storage memory with segment level heterogeneity is provided. The method includes mapping data into data segments, and mapping the data segments into data stripes. The method includes writing a first data stripe from a first data segment across a first plurality of blades of the storage system comprising heterogeneous total amounts of the solid-state storage memory per blade, and writing a second data stripe from the first data segment across a second plurality of blades of the storage system.

Abstract: Embodiments of the present disclosure relate to a memory system and an operating method of the memory system. According to embodiments of the present disclosure, the memory system may include a memory device having a plurality of memory blocks configured to store data; and a memory controller configured to: store the data in a first memory block group including N memory blocks among the plurality of memory blocks, set the data stored in the first memory block group to a read-only state, and migrate, after a threshold time has elapsed from a reference time, a target data which is all or part of the data stored in the first memory block group, to a second memory block group including M memory blocks among the plurality of memory blocks.

Abstract: A computer system includes a NoSQL/SQL cluster and a distributed storage. In order to make storage target data redundant, a compute node in the NoSQL/SQL cluster instructs other compute nodes in the NoSQL/SQL cluster to write the storage target data into the distributed storage. As regards a file containing the storage target data, the compute node in the NoSQL/SQL cluster deduplicates data in storage apparatuses of a plurality of storage nodes in the distributed storage. The distributed storage performs erasure coding to store a file of the storage target data newly found to be duplicate in deduplication and to store a file of the storage target data not deduplicated in deduplication.

Abstract: Systems and methods are disclosed comprising receiving L2P table information from a storage system over a communication interface, maintaining a host L2P table at a physical address using the received L2P table information, and providing a read command to the storage system for first data associated with a first LBA and a host L2P entry associated with the first data. The host L2P entry can include a physical address of the first LBA on the storage system according to the host L2P table and a physical address of a portion of the L2P table on the storage system associated with the first LBA. Control circuitry of the storage system can validate the physical address of the first LBA from the host L2P entry using the physical address of the portion of the host L2P table associated with the first LBA.

Abstract: A system and method for encoding anonymized dataset. A dataset may be pre-processed by dividing into a plurality of sourceblocks at all reasonable sourceblock lengths, and then counting how many times each sourceblock occurs in the dataset, resulting in a tally record of tokens and their count value. This tally record may then be anonymized and transmitted as an anonymized tally record to a data deconstruction engine which combined with a library manager creates a codebook and performs optimization techniques on the codebook. The received anonymized tally record may be parsed into individual tokens by identifying the tokens with the highest count value. The tokens may then be sent, in descending order of count value, to the library manger where each token may be assigned a codeword. Then a half-backed codebook is created using the tokens and each token's unique codeword, before sending the half-backed codebook to a system user.

Abstract: Host computers running applications that store data on a block-based storage system such as a SAN provide hints that differentiate IO data based on which application generated the IO. The hints may include tags that are associated with IO commands sent to the block-based storage system. Each host application is associated with a unique identifier that is placed in the tag. Application name-to-identifier mappings may be sent from the hosts to the block-based storage system. Per-identifier/application deduplication statistics are maintained by the block-based storage system and shared with other block-based storage system. Deduplication is disabled or de-emphasized for IO data generated by applications with statistically low deduplication ratios.

Abstract: A technique of performing data reduction includes, upon detecting a match between similarity hashes of a candidate dataset and a target dataset, evaluating an adjacent candidate dataset and an adjacent target dataset for similarity with each other and, in response to confirming such similarity, performing data reduction of the adjacent candidate dataset with reference to the adjacent target dataset.

Abstract: A system and method for error-resilient data reduction, utilizing a phase detector, a data requestor, a multi-phase trainer, a reconstruction engine, a deconstruction engine, and one or more reference codebooks. A multi-phase trainer may be used to train the reconstruction and deconstruction engines on various phase sourceblocks in order recover quickly from corrupted data files that cause the phase alignment of the sourceblocks to become out of phase. A phase detector may determine when the sourceblocks get out of phase and when the return to in-phase by checking if a predetermined threshold probability of correct encoding is met. Data requestor may request for retransmission only the data that was received out of phase.

Abstract: A system and methods for secure storage for data deduplication comprising a data deconstruction engine, a data reconstruction engine, a library manager, a reference codebook, and a codeword storage which performs simultaneous compaction and deduplication of data sets. A data set may be comprised of one or more sourcepackets which may be optimally deconstructed into a plurality of sourceblocks and wherein each sourceblock may be compared against a reference codebook that contains key-value pairs of a sourceblock and its associated reference code in order to determine if a received sourceblock is a duplicate of data already stored within the reference codebook. Non-duplicate sourceblocks can have a reference code algorithmically created and stored in the reference codebook, thereby ensuring that when a duplicate sourceblock is received, it will not be stored as duplicated data.

Abstract: The inventor has conceived, and reduced to practice, a system and method for data compaction using that applies delta encoding methods to entropy encoding methods to improve data compaction of entropy encoding methods under certain conditions and when compacting data having certain characteristics. Delta encoding may be applied to entropy encoding methods to further compact data sets by reducing the number of sourceblocks included in a codebook to those most commonly encountered in data to be encoded and, where mismatches occur during encoding, using delta encoding of bit differences with existing sourceblocks in the codebook rather than adding new sourceblocks to the codebook.