Abstract: The instant disclosure provides a data structure store system and a method of managing data in the store. The method includes receiving, by a data structure store management system, a request for storing data from a client. In the created data structure, each data element includes a portion of the data. On receiving a read request for at least part of the data, the data structure store management system provides at least part of the data to a recipient device. The data elements are stored in persistent memory in the form of one or more non-volatile random access devices, wherein during a time interval between receiving the storing request and providing the at least part of the data by the data structure store management system to the recipient device, the data structure store management system provides no portion of the data for writing to a hard disk drive.

Abstract: The instant disclosure provides a data structure store system and a method of managing data in the store. The method includes receiving, by a data structure store management system, a request for storing data from a client. In the created data structure, each data element includes a portion of the data. On receiving a read request for at least part of the data, the data structure store management system provides at least part of the data to a recipient device. The data elements are stored in persistent memory in the form of one or more non-volatile random access devices, wherein during a time interval between receiving the storing request and providing the at least part of the data by the data structure store management system to the recipient device, the data structure store management system provides no portion of the data for writing to a hard disk drive.

Abstract: This invention makes capacity fluid among multiple kubernetes clusters maintained by an organization by introducing a system and method named capacity middleware to shrink and grow clusters based on their resource requirements. Capacity Middleware, run on the Management Cluster alongside an API controlling Clusters and assigns annotations related to priority on objects of Cluster resource, annotation for no preemption Quota to objects of MachineDeployment specifying the number of resources for each cluster and annotation of valid capacity (capacityValidated) by default set to false on objects of Machine resource which is used by the Capacity Middleware as a signal to respond to these objects. The capacity middleware iteratively checks and frees or assigns resources based needs of different clusters based on difference between required capacity and available capacity. A difference of negative suggests need for preempting resource whereas a difference in positive number suggest additionally required resources.

Abstract: The instant disclosure provides a data structure store system and a method of managing data in the store. The method includes receiving, by a data structure store management system, a request for storing data from a client. In the created data structure, each data element includes a portion of the data. On receiving a read request for at least part of the data, the data structure store management system provides at least part of the data to a recipient device. The data elements are stored in persistent memory in the form of one or more non-volatile random access devices, wherein during a time interval between receiving the storing request and providing the at least part of the data by the data structure store management system to the recipient device, the data structure store management system provides no portion of the data for writing to a hard disk drive.

Abstract: The instant disclosure provides a data structure store system and a method of managing data in the store. The method includes receiving, by a data structure store management system, a request for storing data from a client. In the created data structure, each data element includes a portion of the data. On receiving a read request for at least part of the data, the data structure store management system provides at least part of the data to a recipient device. The data elements are stored in persistent memory in the form of one or more non-volatile random access devices, wherein during a time interval between receiving the storing request and providing the at least part of the data by the data structure store management system to the recipient device, the data structure store management system provides no portion of the data for writing to a hard disk drive.

Abstract: A rebuild node of a storage system can assess risk of the storage system not being able to provide a data object. The rebuild node(s) uses information about data object fragments to determine health of a data object, which relates to the risk assessment. The rebuild node obtains object fragment information from nodes throughout the storage system. With the object fragment information, the rebuild node(s) can assess object risk based, at least in part, on the object fragments indicated as existing by the nodes. To assess object risk, the rebuild node(s) treats absent object fragments (i.e., those for which an indication was not received) as lost. When too many object fragments are lost, an object cannot be rebuilt. The erasure coding technique dictates the threshold number of fragments for rebuilding an object. The risk assessment per object influences rebuild of the objects.

Abstract: The instant disclosure provides a data structure store system and a method of managing data in the store. The method includes receiving, by a data structure store management system, a request for storing data from a client. In the created data structure, each data element includes a portion of the data. On receiving a read request for at least part of the data, the data structure store management system provides at least part of the data to a recipient device. The data elements are stored in persistent memory in the form of one or more non-volatile random access devices, wherein during a time interval between receiving the storing request and providing the at least part of the data by the data structure store management system to the recipient device, the data structure store management system provides no portion of the data for writing to a hard disk drive.

Abstract: A method, device and non-transitory computer readable medium that manages read access includes organizing a plurality of requests for objects on one or more storage media, such as tapes or spin-down disks, based on at least a deadline for each of the plurality of requests. One of one or more replicas for each of the objects on the one or more storage media is selected based on one or more factors. An initial schedule for read access is generated based at least on the deadline for each of the plurality of requests, the selected one of the replicas for each of the objects, and availability of one or more drives. The initial schedule for read access on the one or more of the drives for each of the plurality of requests for the objects is provided.

Abstract: A rebuild node of a storage system can assess risk of the storage system not being able to provide a data object. The rebuild node(s) uses information about data object fragments to determine health of a data object, which relates to the risk assessment. The rebuild node obtains object fragment information from nodes throughout the storage system. With the object fragment information, the rebuild node(s) can assess object risk based, at least in part, on the object fragments indicated as existing by the nodes. To assess object risk, the rebuild node(s) treats absent object fragments (i.e., those for which an indication was not received) as lost. When too many object fragments are lost, an object cannot be rebuilt. The erasure coding technique dictates the threshold number of fragments for rebuilding an object. The risk assessment per object influences rebuild of the objects.

Abstract: A method, non-transitory computer readable medium, and archive node computing device that receives an indication of each of a plurality of archived files required to service a job from one of a plurality of compute node computing devices of an analytics tier. An optimized schedule for retrieving the archived files from one or more archive storage devices of an archive tier is generated. The optimized schedule is provided to the one of the plurality of compute node computing devices. Requests for the archived files received from the one of the plurality of compute node computing device and at least one other of the plurality of compute node computing devices, wherein the requests are sent according to the optimized schedule.

Abstract: The instant disclosure provides a data structure store system and a method of managing data in the store. The method includes receiving, by a data structure store management system, a request for storing data from a client. In the created data structure, each data element includes a portion of the data. On receiving a read request for at least part of the data, the data structure store management system provides at least part of the data to a recipient device. The data elements are stored in persistent memory in the form of one or more non-volatile random access devices, wherein during a time interval between receiving the storing request and providing the at least part of the data by the data structure store management system to the recipient device, the data structure store management system provides no portion of the data for writing to a hard disk drive.

Abstract: The technique introduced here includes a system and method for identification of duplicate data directly at a data-object level. The technique illustratively utilizes a hierarchical tree of fingerprints for each data object to compare data objects and identify duplicate data blocks referenced by the data objects. The hierarchical fingerprint trees are constructed in such a manner that a top-level fingerprint (or object-level fingerprint) of the hierarchical tree is representative of all data blocks referenced by a storage system. In embodiments, inline techniques are utilized to generate hierarchical fingerprints for new data objects as they are created, and an object-level fingerprint of the new data object is compared against preexisting object-level fingerprints in the storage system to identify exact or close matches.

Abstract: A method, non-transitory computer readable medium, and archive node computing device that receives an indication of each of a plurality of archived files required to service a job from one of a plurality of compute node computing devices of an analytics tier. An optimized schedule for retrieving the archived files from one or more archive storage devices of an archive tier is generated. The optimized schedule is provided to the one of the plurality of compute node computing devices. Requests for the archived files received from the one of the plurality of compute node computing device and at least one other of the plurality of compute node computing devices, wherein the requests are sent according to the optimized schedule.

Abstract: A method, device and non-transitory computer readable medium that manages read access includes organizing a plurality of requests for objects on one or more storage media, such as tapes or spin-down disks, based on at least a deadline for each of the plurality of requests. One of one or more replicas for each of the objects on the one or more storage media is selected based on one or more factors. An initial schedule for read access is generated based at least on the deadline for each of the plurality of requests, the selected one of the replicas for each of the objects, and availabilty of one or more drives. The initial schedule for read access on the one or more of the drives for each of the plurality of requests for the objects is provided.

Abstract: The technique introduced here includes a system and method for identifying and mapping duplicate data objects referenced by data objects. The technique illustratively utilizes a hierarchical tree of fingerprints for each data object to compare the data objects and identify duplicate data blocks referenced by the data objects. A progressive comparison of the hierarchical trees starts from a top layer of the hierarchical trees and proceeds toward a base layer. Between the compared data objects (i.e., the compared hierarchical trees), the technique maps matching fingerprints only at the top-most layer of the hierarchical trees at which the fingerprints match. Lower layer matching fingerprints are neither compared nor mapped. Data blocks corresponding to the matching fingerprints are then deleted. Such an identification and mapping technique substantially reduces the amount of mapping metadata stored in data objects that have been subject to deduplication.

Abstract: Technology is described for responding to service level objectives during deduplication. In various embodiments, the technology receives a service level objective (SLO); receives data to be stored at the data storage system; computes an amount of deduplication to apply to the received data responsive to the SLO; deduplicates the data to the computed amount; and stores the deduplicated data. The deduplicated data may be stored in such a manner that the data can be read in a manner that meets the SLO.

Abstract: The technique introduced here includes a system and method for identifying and mapping duplicate data objects referenced by data objects. The technique illustratively utilizes a hierarchical tree of fingerprints for each data object to compare the data objects and identify duplicate data blocks referenced by the data objects. A progressive comparison of the hierarchical trees starts from a top layer of the hierarchical trees and proceeds toward a base layer. Between the compared data objects (i.e., the compared hierarchical trees), the technique maps matching fingerprints only at the top-most layer of the hierarchical trees at which the fingerprints match. Lower layer matching fingerprints are neither compared nor mapped. Data blocks corresponding to the matching fingerprints are then deleted. Such an identification and mapping technique substantially reduces the amount of mapping metadata stored in data objects that have been subject to deduplication.

Abstract: The technique introduced here includes a system and method for identification of duplicate data directly at a data-object level. The technique illustratively utilizes a hierarchical tree of fingerprints for each data object to compare data objects and identify duplicate data blocks referenced by the data objects. The hierarchical fingerprint trees are constructed in such a manner that a top-level fingerprint (or object-level fingerprint) of the hierarchical tree is representative of all data blocks referenced by a storage system. In embodiments, inline techniques are utilized to generate hierarchical fingerprints for new data objects as they are created, and an object-level fingerprint of the new data object is compared against preexisting object-level fingerprints in the storage system to identify exact or close matches.

Abstract: The technique introduced here includes a system and method for preprocessing a data set to improve deduplication, and more specifically for reducing latency. The technique illustratively utilizes one or more preprocessing steps, including a “skipping” step and a “folding” step, which can be applied to a data set prior to deduplication to reduce the time consumed by deduplication. The folding step is applied to segments of the data set to reduce the length of the segments. The skipping step can be applied to the data set prior to the folding step to remove particular segments of the data set, to further improve deduplication performance in certain circumstances. The overall effect of the skipping and folding steps of this technique is to produce a data set of reduced total length for consideration in identifying duplicate data, which aids in reducing the time required for deduplication.

Abstract: A system and method for compression of partially ordered data sets is provided. A first record of the data set is compressed by encoding the record using a Fibonacci encoding technique. Thereafter, for each subsequent record N, the N?1st record is subtracted from the Nth record before encoding the result, thereby allowing each subsequent record to store the difference (or delta) from the previous record.