Abstract: A method, article of manufacture, and apparatus for creating a fingerprint to container id index is discussed. The index may be stored in-memory, on disk, and on a solid-state device. The index may be used to quickly locate a container identifier given a data segment fingerprint.

Abstract: According to some embodiments, a backup storage system receives a request from a client to access a data segment. The system looks up an in-memory index for a first entry based on a fingerprint of the data segment, such that the in-memory index includes a number of entries, each entry mapping a fingerprint to a storage location of a solid state device (SSD) storing a corresponding data segment. In the event a first entry is found, the system retrieves the data segment from the SSD device based on the first entry. In the event of a read failure while accessing the SSD device, the system determines failed data blocks of the SSD device associated with the failure event and invalidates the failed data blocks of the SSD device such that subsequent access to the failed data blocks are denied without having to access a corresponding SSD device.

Abstract: According to some embodiment, a backup storage system receives a request from a client to read a data segment associated with a file object stored in a storage system. In response to the request, the system performs a lookup operation in a first index stored in a memory to identify a first index entry based on a fingerprint of the requested data segment to obtain a first write-evict unit (WEU) identifier (ID) identifying a first WEU storing the requested data segment. The system accesses a solid state device (SSD) operating as a cache memory device to retrieve the data segment from the first WEU. The system extracts and decompresses a compressed data segment retrieved from the first WEU and returns the decompressed data segment to the client without accessing a storage unit for retrieving the same data segment.

Abstract: According to some embodiments, a backup storage system receives a request from a client at a storage system for accessing data segments. For each of a first groups of the data segments requested that are stored in a solid state device (SSD) cache, the system requests a first batch job for each of the first groups to retrieve the first groups of the data segments from the SSD cache via a first set of input/output (IO) threads. For each of a second groups of the data segments requested that are not stored in the SSD cache, the system requests a second batch job for each of the second groups to retrieve the second groups of the data segments from storage units of the storage system via a second set of input/output (IO) threads. The system assembles received segments and returns them to the client altogether.

Abstract: According to some embodiments, a backup storage system receives a request from a client for writing a data segment associated with a file object stored to a storage system. In response to the request, the system writes the data segment to one of many storage units of the storage system. The system determines whether the data segment is associated with a file region of the file object that is frequently accessed. The system writes the data segment in a first of many of write-evict units (WEUs) stored in a solid state device (SSD) operating as a cache memory device for caching data, after it is determined that the data segment is associated with the frequently accessed file region.

Abstract: A method, article of manufacture, and apparatus for locating data inside a deduplicated storage system is discussed. An index on a solid state device may contain a finger print to container identifier mapping. The container identifier may identify a logical container storing data related to the fingerprint. This data may therefore be located using the index given the fingerprint.

Abstract: According to some embodiment, a backup storage system receives a request from a client for writing a data segment associated with a file object stored to a storage system. In response to the request, the system writes the data segment to one of a plurality of storage units of the storage system. The system determines whether an amount of access to a solid state device (SSD) operating as a cache memory device exceeds an access threshold based on a time window distribution. The system throttles access to the SSD for caching data in response to determining that the amount of access to the SSD has exceeded the access threshold.

Abstract: According to some embodiment, a backup storage system determines whether one or more data segments associated with a file object have been updated, where the data segment(s) are stored in a first segment level in a multi-level hierarchy. For each updated data segment, the system applies a reference-based hierarchical data structure to track metadata associated with the updated data segment, where the metadata is stored in a second segment level in the multi-level hierarchy. The system determines whether the reference-based hierarchical data structure has reached a specific size. The system writes the metadata associated with the updated data segment(s) to a solid state device (SSD) operating as a memory cache device responsive to determining that the reference-based hierarchical data structure has reached the specific size.

Abstract: A data storage device includes a cache for a data storage and a processor. The data storage includes an object storage. The processor obtains cache hardware heuristics data for a first time period; makes a first determination that the cache hardware heuristics data for the first time period does not meet a goal associated with the first time period; and populates the cache using a reduced size index cache in response to the first determination during a second time period.

Abstract: According to some embodiment, a backup storage system receives a request from a client for reading a data segment associated with a file object stored in a storage system. In response to the request, the system determines whether a cache hit counter associated with the data segment exceeds a cache hit threshold. The system further determines whether the data segment is associated with a file region of the file object that is frequently accessed. The system writes the data segment into a memory responsive to determining that the cache hit counter exceeds the cache hit threshold and the data segment is associated with the frequently accessed file region. Otherwise, the system writes the data segment into a solid state device (SSD) operating as a cache device.

Abstract: A method, article of manufacture, and apparatus for maintaining a cache index inside a deduplicated storage system is discussed. A block allocation is requested from a persistent storage device, wherein the block allocation comprises a logical block for storing data. The logical block is determined to be unavailable. A next sequential block is requested from the persistent storage until an available logical block is located, the available logical block satisfying the block allocation. Data comprising a plurality of buckets is written to the available logical block.

Abstract: A method, article of manufacture, and apparatus for scaling an index on a solid state drive in a deduplicated storage system is discussed. The index may be scaled as index hash buckets approach capacity, or as new solid state drives are added to the system. The hash buckets may be mapped to HDD hash buckets, which are stored on a hard drive.

Abstract: A method, article of manufacture, and apparatus for maintaining a cache index inside a deduplicated storage system is discussed. A determination is made to flush a cache buffer to the cache index, wherein both the cache buffer and the cache index comprise a plurality of cache buckets, wherein the cache buckets are stored on a plurality of logical data blocks. A write is initiated to at least one of the plurality of logical data blocks, wherein the write comprises at least one of the cache buckets. An error is received in response to the write request. The error is stored in a memory on the deduplicated storage system.

Abstract: A data storage device includes a cache for an object storage and a processor. The processor suspends processing of files for storage in the object storage. While the processing of files is suspended the processor generates a rebuilt index using the object storage, generates a rebuilt index cache using the object storage, stores the rebuilt index in the object storage, and stores the rebuilt index cache in the cache.

Abstract: A method, article of manufacture, and apparatus for populating an index cache on a deduplicated storage system is discussed. A determination to flush an in-memory index to a hard drive (“HDD”) on the deduplicated storage system is made, wherein the in-memory index comprises in-memory hash buckets containing fingerprint identifiers and container identifiers. A first HDD index is loaded from the HDD into a memory, wherein the first index includes a plurality of HDD buckets. The fingerprint identifiers and the container identifiers are merged from the in-memory hash buckets into the HDD buckets. The HDD buckets are mapped to a plurality of solid state drive (“SSD”) buckets, the SSD buckets together comprising a SSD index. The fingerprint identifiers and container identifiers are inserted into the plurality of SSD buckets.

Abstract: A method, article of manufacture, and apparatus for recovering an index on a deduplicated storage device is discussed. A system index is merged into both a first persistent index and a second persistent index, wherein the system index comprises a plurality of buckets. A disrupting event is encountered during the merge. The first persistent index and the second persistent index are determined not to be synchronized as a result of the disrupting event. The merge is continued from a last synchronized location, wherein the last synchronized location comprises a bucket from the plurality of buckets that is successfully merged into both the first persistent index and the second persistent index.