Abstract: Embodiments relating to garbage collection for a deduplicated and compressed storage device are described. One embodiment provides for a computer implemented method including creating a multiple sets of Bloom filters distributed across a set of multiple computing device nodes. One set of Bloom filters stores differing ranges of fingerprints for data stored on deduplicated storage containers, while a second set of Bloom filters includes a set of fingerprints correlated with files in a file system directory structure. A set of live fingerprints is determined for live fingerprints and storage segments associated with those fingerprints are copied to new storage containers.

Abstract: Techniques for balancing data compression and read performance of data chunks of a storage system are described herein. According to one embodiment, similar data chunks are identified based on sketches of a plurality of data chunks stored in the storage system. A first portion of the similar data chunks as a first group is associated with a first storage area. The first storage area is associated with one or more data chunks that are dissimilar to the first group but are likely accessed together. The first group of the similar data chunks and its associated dissimilar data chunks are compressed and stored in the first storage area.

Abstract: Androgynous modular connectors (e.g., modular spacecraft connectors) are disclosed herein. The modular connectors can include prismatic-shaped body with side panels, a base, and a face panel. The face panel can include three or more sides that define an opening. A claw assembly can be housed within the prismatic-shaped body. The claw assembly can include two or more extendable arms. A motor can be housed within the prismatic-shaped body. The motor can be operatively connected to the claw assembly. The motor can be configured to extend and retract the claw assembly.

Abstract: Use of a survival queue to manage a container-based flash cache is disclosed. In various embodiments, a corresponding survival time is associated with each of a plurality of containers stored in a flash cache, each container comprising a plurality of data blocks. The survival time may be determined based at least in part on a calculated proportion of relatively recently accessed data blocks associated with the container is associated with the container. A container to evict from the flash cache is selected based at least in part on a determination that the corresponding survival time of the selected container has expired.

Abstract: Embodiments relating to garbage collection for a deduplicated and compressed storage device are described. One embodiment provides for a method comprising creating a first set of temporary files associated with a range of fingerprints for data within data files associated with a directory tree structure; creating a second set of temporary files associated with a range of fingerprints of storage segments stored on one or more deduplicated storage containers; sorting the fingerprints in each temporary file using distributed out of core sorting across each node in the set of multiple computing device nodes to generate a first set of sorted files and a second set of sorted files; determining an intersection of the fingerprints in the first set of sorted files and the second set of sorted files; and generating a garbage collection recipe for each of the one or more deduplicated storage containers.

Abstract: Techniques to manage usage of a flash-based storage are disclosed. In various embodiments, the execution time of the flash-based storage is divided into quanta. Within each quantum comprising at least a subset of quanta, flash erasures are allowed without restriction up to a prescribed erasure quota. Erasures are throttled within a slack range bound at a lower end by the erasure quota and at an upper end by an upper bound, including by dividing the slack range into two or more intervals and within each interval applying a corresponding erasure control policy, wherein the respective corresponding erasure control policies applied to successive intervals in the slack range become increasingly strict in a stepwise manner.

Abstract: Techniques for improving data compression of a storage system using coarse and fine grained similarity are described herein. According to one embodiment, region sketches for a plurality of regions of the set of data are generated, each region storing a plurality of data chunks. A region sketch index having a plurality of entries is maintained, each corresponding to one of the region sketches of the regions. The entries of the region sketch index are sorted based on the sketches of the regions, such that regions with an identical region sketch are positioned adjacent to each other within the region sketch index, representing similar regions. The data chunks of the similar regions that are identified based on the sorted entries of the region sketch index are reorganized to improve data compression of the data chunks of the similar regions.

Abstract: One embodiment is related to a method for redistributing cache space, comprising: determining utility values associated with all of a plurality of clients, each client being associated with a respective utility value, the utility value being indicative of an efficiency of cache space usage of the associated client; and redistributing cache space among the plurality of clients based on the utility values.

Abstract: One embodiment is related to a method for redistributing cache space, comprising: determining a request by a first client of a plurality of clients for additional cache space, each of the plurality of clients being associated with a guaranteed minimum amount (MIN) and a maximum amount (MAX) of cache space; and fulfilling or denying the request based on an amount of cache space the first client currently occupies, an amount of cache space requested by the first client, and the MIN and the MAX cache space associated with the first client.

Abstract: Embodiments relating to garbage collection for a deduplicated and compressed storage device are described. One embodiment provides for a networked system comprising a set of multiple computing devices, each computing device to provide one or more deduplication and replication services to one or more deduplicated storage containers and a central computing device connected to each computing device in the set of multiple computing devices over a local network, the central computing device to provide deduplication garbage collection to each computing device in the set of computing devices.

Abstract: Managing data blocks stored in a data processing system comprises logging input/output (I/O) accesses to an in-memory buffer with each log entry recording at least identifiers of the data blocks accessed and when the data blocks were accessed. When the size of the log entries reaches a predetermined threshold, the system may append log entries of the in-memory buffer to the end of a history log file. The history log file is analyzed to determine patterns of accesses, and each pattern is stored in a record in an access heuristics database. While processing a request for access to a data block, the data processing system queries the access heuristics database to obtain prior access patterns associated with the data block. A data management action may be taken based on the prior access patterns.

Abstract: Embodiments relating to garbage collection for a deduplicated and compressed storage device are described. One embodiment provides for a deduplication data storage system comprising a set of storage devices including a set of storage containers; one or more computing device nodes coupled to the set of storage devices, the one or more computing device nodes to enable replication and deduplication of the set of storage containers on the set of storage devices; and a set of remote storage servers coupled to the one or more computing device nodes over a network, the set of remote storage servers to receive a replication of data on the set of storage containers from the one or more computing device nodes, and to generate a deduplication garbage collection recipe for the set of storage containers.

Abstract: Embodiments relating to garbage collection for a deduplicated and compressed storage device are described. One embodiment provides for a computer implemented method including creating a multiple sets of Bloom filters distributed across a set of multiple computing device nodes. One set of Bloom filters stores differing ranges of fingerprints for data stored on deduplicated storage containers, while a second set of Bloom filters includes a set of fingerprints correlated with files in a file system directory structure. A set of live fingerprints is determined for live fingerprints and storage segments associated with those fingerprints are copied to new storage containers.

Abstract: Techniques for balancing data compression and read performance of data chunks of a storage system are described herein. According to one embodiment, similar data chunks are identified based on sketches of a plurality of data chunks stored in the storage system. A first portion of the similar data chunks as a first group is associated with a first storage area. The first storage area is associated with one or more data chunks that are dissimilar to the first group but are likely accessed together. The first group of the similar data chunks and its associated dissimilar data chunks are compressed and stored in the first storage area.

Abstract: Embodiments are directed to maintaining cache status information for a flash-based cache after a system restart following a shutdown by periodically gathering the cache status information from one or more data structures in volatile memory, compiling the gathered cache status information into a checkpoint data structure stored in non-volatile memory, and upon the system restart, loading the checkpoint data element into the one or more data structures in volatile memory to restore the cache status information to a state just prior to the shutdown to maintain warmness of the cache across the restart. The restored cache status information is used by the system to make informed write eviction decisions to maintain correctness of the system after the restart relative to the state just prior to the shutdown.

Abstract: Embodiments relating to garbage collection for a deduplicated and compressed storage device are described. One embodiment provides for a computer implemented method including creating a multiple sets of Bloom filters distributed across a set of multiple computing device nodes. One set of Bloom filters stores differing ranges of fingerprints for data stored on deduplicated storage containers, while a second set of Bloom filters includes a set of fingerprints correlated with files in a file system directory structure. A set of live fingerprints is determined for live fingerprints and storage segments associated with those fingerprints are copied to new storage containers.

Abstract: Techniques for balancing data compression and read performance of data chunks of a storage system are described herein. According to one embodiment, similar data chunks are identified based on sketches of a plurality of data chunks stored in the storage system. A first portion of the similar data chunks as a first group is associated with a first storage area. The first storage area is associated with one or more data chunks that are dissimilar to the first group but are likely accessed together. The first group of the similar data chunks and its associated dissimilar data chunks are compressed and stored in the first storage area.

Abstract: A plurality of linked lists of elements is maintained corresponding to a plurality of threads accessing a plurality of cache entries, including a first linked list corresponding to a first thread and a second linked list corresponding to a second thread. Each element of each linked list corresponds to one of the plurality of cache entries. In response to the first thread accessing a cache entry corresponding to an element of the second linked list of elements, the element corresponding to the accessed cache entry is inserted to a head of the first linked list of elements. The element corresponding to the accessed cache entry is removed from the second linked list. One or more neighboring elements that were adjacent to the removed elements are re-linked on the second linked list.

Abstract: Embodiments relating to garbage collection for a deduplicated and compressed storage device are described. One embodiment provides for a data storage system comprising an array of redundant storage devices including a first set of storage devices to be configured as live storage devices and a second set of storage devices to be configured as spare storage devices, a spare storage device to be enabled in event of a failure of a live storage device; and a set of processing devices coupled to the array of redundant storage devices, the set of processing devices to execute logic to enable data replication and deduplication for the array of redundant storage devices and perform distributed deduplication garbage collection on the first set of storage devices using one or more devices in the second set of storage devices as temporary storage.

Abstract: One embodiment provides a document management system comprising a storage system to store one or more encrypted documents, at least a first portion of a first encrypted document encrypted using a first encryption key, and an encryption key manager to manage a set of encryption keys for the documents on the storage system, the encryption key manager further to discard the first encryption key to provide secure removal of the portion of the encrypted document.