Abstract: Technology is provided for updating a data set in a data storage system. In an example data storage system, the system stores a separate copy of an initial data set on each one of a plurality of storage devices, one of which is designated as a leader storage device. The system receives update data and transmits it to each other one of the plurality of replica storage devices. The system updates the copy of the initial data set stored on a replica storage device based on the updated data, resulting in an updated data set and adds a provisional marker to the updated data set. The system transmits an update notification to each of the other replica storage devices. Responsive to determining that update notifications have been received from a threshold number of replica storage, the system removes the provisional marker from the updated data set.

Abstract: Various aspects for managing input/output operations in a storage network are described. For instance, a method may include applying a hash function on a target data object to calculate a hash key for the target data object and identifying a target storage bucket for the target data object based on the hash key and a hash table map. The method may further include reading a data object key for a data object stored in the target storage bucket and comparing the data object key and the hash key to determine a match. The method may also include determining that the data object is the target data object if the data object key and the hash key match and reading the target data object from the target storage bucket when there is a match. Some methods can be performed using a single remote direct access request.

Abstract: A system that implements a near-optimal, reduced-dependency erasure code construction to redundantly distribute computer data across multiple storage nodes includes a memory that stores machine instructions and a processor that executes the machine instructions to group storage segments into discrete groups, each of which corresponds to an individual storage node. The processor further executes the machine instructions to represent regeneration constraints and associate the constraints with storage segments in multiple storage nodes. The processor also executes the machine instructions to generate a parity check matrix based on the regeneration constraints, the associations and the storage segments. The processor additionally executes the machine instructions to construct a generator matrix based on the parity check matrix.

Abstract: Small objects are efficiently stored with erasure codes by combining a small object with other small objects and/or large objects to form a single large object for chunking, and providing early notification of permanent storage to the sources of the objects to prevent small objects from becoming stale while waiting for additional objects to be combined.

Abstract: An approach for fingerprinting large data objects at the wire speed has been disclosed. The techniques include Fresh/Shift pipelining, split Fresh, optimization, online channel sampling, and pipelined selection. The architecture can also be replicated to work in parallel for higher system throughput. Fingerprinting may provide an efficient mechanism for identifying duplication in a data stream, and deduplication based on the identified fingerprints may provide reduced storage costs, reduced network bandwidth consumption, reduced processing time and other benefits. In some embodiments, fingerprinting may be used to ensure or verify data integrity and may facilitate detection of corruption or tampering. An efficient manner of generating fingerprints (either via hardware, software, or a combination) may reduce a computation load and/or time required to generate fingerprints.

Abstract: An approach for fingerprinting large data objects at the wire speed has been disclosed. The techniques include Fresh/Shift pipelining, split Fresh, optimization, online channel sampling, and pipelined selection. The architecture can also be replicated to work in parallel for higher system throughput. Fingerprinting may provide an efficient mechanism for identifying duplication in a data stream, and deduplication based on the identified fingerprints may provide reduced storage costs, reduced network bandwidth consumption, reduced processing time and other benefits. In some embodiments, fingerprinting may be used to ensure or verify data integrity and may facilitate detection of corruption or tampering. An efficient manner of generating fingerprints (either via hardware, software, or a combination) may reduce a computation load and/or time required to generate fingerprints.

Abstract: The present disclosure relates to a system and methods of controlling a system of storage devices. In particular, the present disclosure relates to methods of controlling peak power and energy consumption in storage systems due to storage devices while maintaining data availability. The system implements a method for maintaining data availability in a storage subsystem by determining a plurality of storage devices to include in a fixed set of storage devices based on a fault tolerance system. The storage devices included in the fixed set are prevented from transitioning between RPM spin modes. The method further involves controlling peak power and energy consumed by the storage subsystem which may include transitioning the storage devices not included in the fixed set from a high RPM operational mode to a low RPM operational mode to reduce peak power and energy consumption.

Abstract: A journaling approach is used to distribute data of different sizes between areas of a segment's log on a physical NAND flash erase block. The Main area contains large, contiguous extents of data, and the Journal area contains logical blocks of small data. An Updates area also contains updates that are pending. One disclosed embodiment includes storing a first file fragment associated with a file in a journal area of a log, where a size of the file fragment is less than a physical NAND flash page size limit, receiving a second file fragment associated with the file, combining the first file fragment and the second file fragment when a combined size of the fragments is equal to the physical NAND flash page size limit, storing the combined fragments in a main area of a second log, receiving an update associated with the combined fragments, and storing the update in an updates area of a third log.

Abstract: An approach for generating updated error detecting code for a partial update of data is disclosed. The techniques include receiving data representing a change to a portion of a data object, the data object having a first error detecting code, and the portion of the data object having an offset from the beginning of the data object; generating a combination term by combining the data and the portion of the data object; and computing a second error detecting code based on the combination term. The techniques may further include computing a third error detecting code by combining the first error detecting code and the second error detecting code, the third error detecting code being configured to detect an error in the data object as changed by the data, and storing the data and the third error detecting code.

Abstract: Techniques for generating parities and repairing data erasures using repair-optimal parities are disclosed. The system includes an encoding module, which receives a request to recreate data for a subset of a plurality of content stores. The encoding module generates a new first parity and a new second parity using a subset of remaining content from the plurality of content stores. The encoding module generates a first portion of the requested data using the new first parity and a first subset of an original first parity for the plurality of content stores and a second portion of the requested data using the new second parity and a second subset of an original second parity for the plurality of content stores. The encoding module may recreate the data for the content store using the first portion of the requested data and the second portion of requested data.

Abstract: A storage device may include a data storage portion including a set of blocks designated to store metadata and a controller. The controller may be configured to write first metadata at a first location designated by a first pointer. The first location may reference a block that does not contain any valid metadata. The controller may be configured to determine a number of valid blocks of previously written metadata in a subset of the set of blocks. A first block of the subset may be at a second location designated by a second pointer. The controller may be configured to, if the number of valid blocks is greater than zero, rewrite the valid previously written metadata to a group of one or more sequential blocks.

Abstract: Techniques for generating parities and repairing data erasures using repair-optimal parities are disclosed. The system includes an encoding module, which receives a request to recreate data for a subset of a plurality of content stores. The encoding module generates a new first parity and a new second parity using a subset of remaining content from the plurality of content stores. The encoding module generates a first portion of the requested data using the new first parity and a first subset of an original first parity for the plurality of content stores and a second portion of the requested data using the new second parity and a second subset of an original second parity for the plurality of content stores. The encoding module may recreate the data for the content store using the first portion of the requested data and the second portion of requested data.

Abstract: The present disclosure relates to methods and systems for performing operations in a communications protocol. An example method can include submitting, from a device, a request for a queue entry representing a command from a host comprising a request for data stored at a device memory location; receiving the command from the host; and executing the command. An example method can also include selecting a bit string representing whether a requested data stream has been received, and storing the bit string into a memory buffer portion to mark the buffer portion. The method can include receiving, into the memory buffer, the stream. The method can include retrieving contents of the buffer portion, and determining whether the contents contain the bit string. If so, the method can include determining that portions of the stream have not been received. Otherwise, the method can include determining that the stream has been received.

Abstract: Small objects are efficiently stored with erasure codes by combining a small object with other small objects and/or large objects to form a single large object for chunking, and providing early notification of permanent storage to the sources of the objects to prevent small objects from becoming stale while waiting for additional objects to be combined.

Abstract: A method and apparatus are provided for implementing enhanced performance with read before write to phase-change-memory. Each write to PCM is preceded by a read and a calculation to discover a location of any bad bits. The write data is converted to a format that can be corrected for a given number of previously undiscovered bit errors, and the writes are unverified.

Abstract: The present disclosure relates to a system and methods of controlling a system of storage devices. In particular, the present disclosure relates to methods of controlling peak power and energy consumption in storage systems due to storage devices while maintaining data availability. The system implements a method for maintaining data availability in a storage subsystem by determining a plurality of storage devices to include in a fixed set of storage devices based on a fault tolerance system. The storage devices included in the fixed set are prevented from transitioning between RPM spin modes. The method further involves controlling peak power and energy consumed by the storage subsystem which may include transitioning the storage devices not included in the fixed set from a high RPM operational mode to a low RPM operational mode to reduce peak power and energy consumption.

Abstract: A system that implements a near-optimal, reduced-dependency erasure code construction to redundantly distribute computer data across multiple storage nodes includes a memory that stores machine instructions and a processor that executes the machine instructions to group storage segments into discrete groups, each of which corresponds to an individual storage node. The processor further executes the machine instructions to represent regeneration constraints and associate the constraints with storage segments in multiple storage nodes. The processor also executes the machine instructions to generate a parity check matrix based on the regeneration constraints, the associations and the storage segments. The processor additionally executes the machine instructions to construct a generator matrix based on the parity check matrix.

Abstract: Methods and systems for reducing write amplification and over-provisioning of flash memory devices are disclosed herein. The approaches described herein are especially useful for decreasing write amplification and over-provisioning in the case of small files or files with gradually growing content. In the case of smaller files, small portions or small updates to files are stored in a NAND flash page of a Journal area. This approach both decreases write amplification and over-provisioning, and also improves garbage collection efficiency. For gradually growing file content, different updates to the file can be stored in Journal areas of different logs. Every diff/update will share space in a NAND flash page with updates of other files. Subsequently, all available updates in the Journal Area for the same logical block of the growing file will be joined and saved if the total amount of updates will be equal to NAND flash page size.

Abstract: A journaling approach is used to distribute data of different sizes between areas of a segment's log on a physical NAND flash erase block. The Main area contains large, contiguous extents of data, and the Journal area contains logical blocks of small data. An Updates area also contains updates that are pending. One disclosed embodiment includes storing a first file fragment associated with a file in a journal area of a log, where a size of the file fragment is less than a physical NAND flash page size limit, receiving a second file fragment associated with the file, combining the first file fragment and the second file fragment when a combined size of the fragments is equal to the physical NAND flash page size limit, storing the combined fragments in a main area of a second log, receiving an update associated with the combined fragments, and storing the update in an updates area of a third log.

Abstract: A journaling approach is used to distribute cold and hot data between different areas of a segment's log on a physical erase block. The Main area of the log is used for cold data, and the Journal area is used for hot data. The Main area contains large, contiguous extents of rarely changed data (e.g., read-only data), and the Journal contains logical blocks of small and frequently updated data. An Updates area also contains updates that are pending. Data from the Main and Updates areas are accumulated and written to a Main area of a different segment's log during a garbage collection operation. The physical erase block is erased and added to a pool of clean physical erase blocks. Using a Journaling approach significantly simplifies the garbage collection process.