Abstract: A method for optimizing data storage includes obtaining a data object for storage at memory hardware in communication with data processing hardware. The memory hardware includes a plurality of storage devices, each storage device of the plurality of storage devices including storage parameters different from each other storage device of the plurality of storage devices. The method also includes determining one or more data object parameters associated with the data object and predicting, using a model and the data object parameters and the storage parameters, an object temperature representative of a frequency of access for the data object and an object lifetime representative of an amount of time the data object is to be stored. The method further includes selecting, using the predicted object temperature and object lifetime, one of the storage devices, and storing the data object at the selected one of the storage devices.

Abstract: A predictive method for scheduling of the operations is described. The predictive method utilizes data generated from computing an expected lifetime of the individual files or objects within the container. The expected lifetime of individual files or objects can be generated based on machine learning techniques. Operations such as garbage collection are scheduled at an epoch where computational efficiencies are realized for performing the operation.

Abstract: A predictive method for scheduling of the operations is described. The predictive method utilizes data generated from computing an expected lifetime of the individual files or objects within the container. The expected lifetime of individual files or objects can be generated based on machine learning techniques. Operations such as garbage collection are scheduled at an epoch where computational efficiencies are realized for performing the operation.

Abstract: Technology is provided for updating a data set in a data storage system. In an example data storage system, the system stores a separate copy of an initial data set on each one of a plurality of storage devices, one of which is designated as a leader storage device. The system receives update data and transmits it to each other one of the plurality of replica storage devices. The system updates the copy of the initial data set stored on a replica storage device based on the updated data, resulting in an updated data set and adds a provisional marker to the updated data set. The system transmits an update notification to each of the other replica storage devices. Responsive to determining that update notifications have been received from a threshold number of replica storage, the system removes the provisional marker from the updated data set.

Abstract: The disclosed technology can advantageously provide an efficient data recovery system including a plurality of storage nodes including a first storage node and a second storage node, and a storage logic that is coupled to the storage nodes and that manages storage of data on the storage nodes. The storage logic is executable to: receive a data set including data elements including a first set of data elements associated with the first storage node and a second set of data elements associated with the second storage node; generate a first parity of the data set, the first parity including a horizontal parity including a set of horizontal parity entries; and combine the data elements from the data set to produce a skipper parity including a set of skipper parity entries. Combining the data elements includes transforming a subset of the data elements from the data set using an invertible operation, the set of horizontal parity entries being different from the set of skipper parity entries.

Abstract: Technology is provided for updating a data set at a data storage system. In an example storage system, the system stores a data set in a plurality of data storage devices. The system stores parity data at a plurality of parity devices. The system receives update data from a client system for a first section of the data set. The system generates updated parity data based on an original version of the first section of the data set and the update data. The system transmits update parity data to the plurality of parity devices. The system receives update notifications from a plurality of parity devices. The system determines that update notifications have been received from at least a threshold number of parity devices in the plurality of parity devices. In response, the system updates the first section of the data set at the leader data storage device.

Abstract: Technology that detects computation errors is disclosed, in which a system may include one or more processors and storage logic. The storage logic may be executed by the one or more processors to perform operations comprising: receiving a data vector, the data vector including a plurality of ordered blocks; transposing the data vector into a set of sub vectors, each of the sub vectors including a corresponding data element from each of the ordered blocks; generating a set of discrete cyclic redundancy checks (CRCs) based on the set of sub vectors; transposing the set of discrete CRCs into a set of mixed CRCs, each of the mixed CRCs including a CRC data element from each of the discrete CRCs; and compacting the set of mixed CRCs into a reduced CRC.

Abstract: Technology is provided for updating a data set at a data storage system. In an example storage system, the system stores a data set in a plurality of data storage devices. The system stores parity data at a plurality of parity devices. The system receives update data from a client system for a first section of the data set. The system generates updated parity data based on an original version of the first section of the data set and the update data. The system transmits update parity data to the plurality of parity devices. The system receives update notifications from a plurality of parity devices. The system determines that update notifications have been received from at least a threshold number of parity devices in the plurality of parity devices. In response, the system updates the first section of the data set at the leader data storage device.

Abstract: An approach for generating updated error detecting code for a partial update of data is disclosed. The techniques include receiving data representing a change to a portion of a data object, the data object having a first error detecting code, and the portion of the data object having an offset from the beginning of the data object; generating a combination term by combining the data and the portion of the data object; and computing a second error detecting code based on the combination term. The techniques may further include computing a third error detecting code by combining the first error detecting code and the second error detecting code, the third error detecting code being configured to detect an error in the data object as changed by the data, and storing the data and the third error detecting code.

Abstract: Techniques for generating parities and repairing data erasures using a cache oblivious encoding algorithm are disclosed. The system includes an encoding module which receives a request to recreate data for a subset of a plurality of content stores from a storage manager. The encoding module generates a new first parity and a new second parity using the remaining content in the plurality of content stores. The encoding module generates a first portion of the requested data using the new first parity and a first parity for the plurality of content stores and a second portion of the requested data using the new second parity and a second parity for the plurality of content stores. The encoding module may recreate the data for the plurality of content stores using the first portion of the requested data and the second portion of the requested data.

Abstract: Technology that detects computation errors is disclosed, in which a system may include one or more processors and storage logic. The storage logic may be executed by the one or more processors to perform operations comprising: receiving a data vector, the data vector including a plurality of ordered blocks; transposing the data vector into a set of sub vectors, each of the sub vectors including a corresponding data element from each of the ordered blocks; generating a set of discrete cyclic redundancy checks (CRCs) based on the set of sub vectors; transposing the set of discrete CRCs into a set of mixed CRCs, each of the mixed CRCs including a CRC data element from each of the discrete CRCs; and compacting the set of mixed CRCs into a reduced CRC.

Abstract: Technology is provided for updating a data set in a data storage system. In an example data storage system, the system stores a separate copy of an initial data set on each one of a plurality of storage devices, one of which is designated as a leader storage device. The system receives update data and transmits it to each other one of the plurality of replica storage devices. The system updates the copy of the initial data set stored on a replica storage device based on the updated data, resulting in an updated data set and adds a provisional marker to the updated data set. The system transmits an update notification to each of the other replica storage devices. Responsive to determining that update notifications have been received from a threshold number of replica storage, the system removes the provisional marker from the updated data set.

Abstract: The disclosed technology can advantageously provide an efficient data recovery system including a plurality of storage nodes including a first storage node and a second storage node, and a storage logic that is coupled to the storage nodes and that manages storage of data on the storage nodes. The storage logic is executable to: receive a data set including data elements including a first set of data elements associated with the first storage node and a second set of data elements associated with the second storage node; generate a first parity of the data set, the first parity including a horizontal parity including a set of horizontal parity entries; and combine the data elements from the data set to produce a skipper parity including a set of skipper parity entries. Combining the data elements includes transforming a subset of the data elements from the data set using an invertible operation, the set of horizontal parity entries being different from the set of skipper parity entries.

Abstract: A system that implements a near-optimal, reduced-dependency erasure code construction to redundantly distribute computer data across multiple storage nodes includes a memory that stores machine instructions and a processor that executes the machine instructions to group storage segments into discrete groups, each of which corresponds to an individual storage node. The processor further executes the machine instructions to represent regeneration constraints and associate the constraints with storage segments in multiple storage nodes. The processor also executes the machine instructions to generate a parity check matrix based on the regeneration constraints, the associations and the storage segments. The processor additionally executes the machine instructions to construct a generator matrix based on the parity check matrix.

Abstract: Small objects are efficiently stored with erasure codes by combining a small object with other small objects and/or large objects to form a single large object for chunking, and providing early notification of permanent storage to the sources of the objects to prevent small objects from becoming stale while waiting for additional objects to be combined.

Abstract: The present disclosure relates to a system and methods of controlling a system of storage devices. In particular, the present disclosure relates to methods of controlling peak power and energy consumption in storage systems due to storage devices while maintaining data availability. The system implements a method for maintaining data availability in a storage subsystem by determining a plurality of storage devices to include in a fixed set of storage devices based on a fault tolerance system. The storage devices included in the fixed set are prevented from transitioning between RPM spin modes. The method further involves controlling peak power and energy consumed by the storage subsystem which may include transitioning the storage devices not included in the fixed set from a high RPM operational mode to a low RPM operational mode to reduce peak power and energy consumption.

Abstract: An approach for generating updated error detecting code for a partial update of data is disclosed. The techniques include receiving data representing a change to a portion of a data object, the data object having a first error detecting code, and the portion of the data object having an offset from the beginning of the data object; generating a combination term by combining the data and the portion of the data object; and computing a second error detecting code based on the combination term. The techniques may further include computing a third error detecting code by combining the first error detecting code and the second error detecting code, the third error detecting code being configured to detect an error in the data object as changed by the data, and storing the data and the third error detecting code.

Abstract: Techniques for generating parities and repairing data erasures using repair-optimal parities are disclosed. The system includes an encoding module, which receives a request to recreate data for a subset of a plurality of content stores. The encoding module generates a new first parity and a new second parity using a subset of remaining content from the plurality of content stores. The encoding module generates a first portion of the requested data using the new first parity and a first subset of an original first parity for the plurality of content stores and a second portion of the requested data using the new second parity and a second subset of an original second parity for the plurality of content stores. The encoding module may recreate the data for the content store using the first portion of the requested data and the second portion of requested data.

Abstract: Techniques for generating parities and repairing data erasures using repair-optimal parities are disclosed. The system includes an encoding module, which receives a request to recreate data for a subset of a plurality of content stores. The encoding module generates a new first parity and a new second parity using a subset of remaining content from the plurality of content stores. The encoding module generates a first portion of the requested data using the new first parity and a first subset of an original first parity for the plurality of content stores and a second portion of the requested data using the new second parity and a second subset of an original second parity for the plurality of content stores. The encoding module may recreate the data for the content store using the first portion of the requested data and the second portion of requested data.

Abstract: Small objects are efficiently stored with erasure codes by combining a small object with other small objects and/or large objects to form a single large object for chunking, and providing early notification of permanent storage to the sources of the objects to prevent small objects from becoming stale while waiting for additional objects to be combined.