Abstract: Embodiments are directed to maintaining cache status information for a flash-based cache after a system restart following a shutdown by periodically gathering the cache status information from one or more data structures in volatile memory, compiling the gathered cache status information into a checkpoint data structure stored in non-volatile memory, and upon the system restart, loading the checkpoint data element into the one or more data structures in volatile memory to restore the cache status information to a state just prior to the shutdown to maintain warmness of the cache across the restart. The restored cache status information is used by the system to make informed write eviction decisions to maintain correctness of the system after the restart relative to the state just prior to the shutdown.

Abstract: Embodiments relating to garbage collection for a deduplicated and compressed storage device are described. One embodiment provides for a computer implemented method including creating a multiple sets of Bloom filters distributed across a set of multiple computing device nodes. One set of Bloom filters stores differing ranges of fingerprints for data stored on deduplicated storage containers, while a second set of Bloom filters includes a set of fingerprints correlated with files in a file system directory structure. A set of live fingerprints is determined for live fingerprints and storage segments associated with those fingerprints are copied to new storage containers.

Abstract: Techniques for balancing data compression and read performance of data chunks of a storage system are described herein. According to one embodiment, similar data chunks are identified based on sketches of a plurality of data chunks stored in the storage system. A first portion of the similar data chunks as a first group is associated with a first storage area. The first storage area is associated with one or more data chunks that are dissimilar to the first group but are likely accessed together. The first group of the similar data chunks and its associated dissimilar data chunks are compressed and stored in the first storage area.

Abstract: A plurality of linked lists of elements is maintained corresponding to a plurality of threads accessing a plurality of cache entries, including a first linked list corresponding to a first thread and a second linked list corresponding to a second thread. Each element of each linked list corresponds to one of the plurality of cache entries. In response to the first thread accessing a cache entry corresponding to an element of the second linked list of elements, the element corresponding to the accessed cache entry is inserted to a head of the first linked list of elements. The element corresponding to the accessed cache entry is removed from the second linked list. One or more neighboring elements that were adjacent to the removed elements are re-linked on the second linked list.

Abstract: Embodiments relating to garbage collection for a deduplicated and compressed storage device are described. One embodiment provides for a data storage system comprising an array of redundant storage devices including a first set of storage devices to be configured as live storage devices and a second set of storage devices to be configured as spare storage devices, a spare storage device to be enabled in event of a failure of a live storage device; and a set of processing devices coupled to the array of redundant storage devices, the set of processing devices to execute logic to enable data replication and deduplication for the array of redundant storage devices and perform distributed deduplication garbage collection on the first set of storage devices using one or more devices in the second set of storage devices as temporary storage.

Abstract: One embodiment provides a document management system comprising a storage system to store one or more encrypted documents, at least a first portion of a first encrypted document encrypted using a first encryption key, and an encryption key manager to manage a set of encryption keys for the documents on the storage system, the encryption key manager further to discard the first encryption key to provide secure removal of the portion of the encrypted document.

Abstract: A target storage system receives a representative fingerprint and fingerprint representations from a source storage system. Each fingerprint representation contains only a portion of a corresponding fingerprint and the representative fingerprint is a full fingerprint. The fingerprints of the data chunks are missing at the target storage system are identified based on the fingerprint representative and the fingerprint representations. A bitmap is transmitted to the source storage system, each bit indicating whether one of the fingerprints is missing. One or more fingerprints are received from the source storage system that are missing at the target storage system based on the bitmask. One or more missing data chunks are identified based on at least the one or more fingerprints received from the source storage system. The missing data chunks are then received from the source storage system to be stored at the target storage system.

Abstract: A data processing system and methods for performing cache eviction are disclosed. An exemplary method includes maintaining a metadata set for each cache unit of a cache device, wherein the cache device comprises a plurality of cache units, each cache unit having a plurality of segments, calculating a score for each metadata set, and arranging the metadata sets in a list in ascending order from lowest score to highest score. The exemplary method further includes in response to determining that a cache eviction is to be performed, selecting a cache unit corresponding to the metadata set in the list having the lowest score, without recalculating a score for any of the metadata set, and evicting the selected cache unit. The metadata nay include, for example, segment count metadata, validity metadata, last access time (LAT) metadata, and hotness metadata.

Abstract: A data processing system and methods for performing cache eviction are disclosed. An exemplary method includes maintaining a metadata set for each cache unit of a cache device at a sub-cache unit granularity, wherein the cache device comprises a plurality of cache units, each cache unit having a plurality of segments, wherein the cache device is accessible by a cache client at a segment granularity. The exemplary method further includes in response to determining that a cache eviction is to be performed, selecting a predetermined number of cache units from the plurality of cache units, determining a score for each of the selected cache units based on the respective metadata set maintained at the sub-cache unit granularity, and evicting one or more of the selected predetermined number of cache units based on their scores. The metadata may include, for example, last access time (LAT) metadata, an access count, and hotness metadata, and metadata may be maintained at a segment or a segment group granularity.

Abstract: A data processing system and methods for performing cache eviction are disclosed. An exemplary method includes maintaining a metadata set for each cache unit of a cache device comprising a plurality of cache units, each cache unit having a plurality of segments. In response to determining that a cache eviction is to be performed, a cache unit is evicted based on its metadata set. The exemplary method includes selecting one or more segments of the evicted cache unit to copy to a second cache unit based on the metadata set of the evicted cache unit, copying the selected one or more segments to the second cache unit, and writing the second cache unit to a storage device. The metadata set may include deletion hints (DH) to indicate valid segments, last access time (LAT) or age based metadata, an access count, or a score for each segment based on the metadata set.

Abstract: In response to a request from a client to store a data block in a storage system, the data block is segmented into a plurality of subblocks. Each of the plurality of subblocks is individually compressed into a compressed subblock. The compressed subblocks are packed into a compressed data block. The compressed data block having the individually compressed subblocks therein is stored in a persistent storage device. Metadata of the compressed data block is stored in an index entry in an index of the storage system, including storing subblock locators indicating locations of the compressed subblocks. Each of the subblocks can be individually accessed based on a corresponding subblock locator without having to access remaining subblocks.

Abstract: An embodiment is described in which a memory device stores a record of I/O accesses to data blocks. And each access record indicates which data block was accessed and during which time period the access occurred. A memory-efficient data structure (MEDS) may be generated and stored in a cache or storage device and the access data moved from the memory device into the MEDS. The MEDS represents blocks that were accessed during a particular time period. When a second data block is accessed, a query function is applied to the second block's identifier to return a value based on data stored in the MEDS. The return value from the query function indicates whether the second data block was accessed during the particular time period associated with the MEDS. A storage management action is performed based on whether the second data block was accessed during the particular time period.

Abstract: Exemplary methods for predicting backup and restore failure include analyzing, at a management server, resource utilization statistics periodically collected during backup of data from a source storage system to a target storage system. In one embodiment, the methods include creating a predictive model based on the analysis of the collected resource utilization statistics. In one embodiment, the method includes predicting, using the predictive model, whether a backup time or a restore time of future backup will exceed a backup time threshold or restore time threshold, respectively.

Abstract: Techniques for improving data compression of a storage system in an online manner are described herein. According to one embodiment, in response to a sequence of data to be stored, the sequence of data is partitioned into a plurality of data chunks according to a predetermined chunking algorithm. A sketch for each of the data chunks is generated based on one or more features extracted from the data chunk. Each of the data chunks of the sequence of data is associated with one of a plurality of groups based on the sketch, wherein each group is represented by a sketch. The data chunks of each group are compressed and stored in a compression region of the storage systems, such that similar data chunks are compressed and stored in the same compression region.

Abstract: A method for storing data in a data storage system by partitioning the data into a plurality of data chunks and generating representative data for each of the plurality of chunks by applying a predetermined algorithm to each chunk of the plurality of chunks. Subsequently, the representative data is compared and sorted. Representative data for base data chunks and representative data for other data chunks that can be stored relative to the base data chunks are identified by evaluating the sorted set of representative data. Finally, each of the other data chunks identified as those that can be stored relative to a base data chunk are stored in the data storage system as the difference between the data chunk and a base data chunk.

Abstract: A data processing system and methods for performing cache eviction are disclosed. An exemplary method includes maintaining a metadata set for each cache unit of a cache device, wherein the cache device comprises a plurality of cache units, each cache unit having a plurality of segments, wherein each metadata set includes deletion hints (DH) metadata indicating whether the plurality of segments of a corresponding cache unit are valid. The exemplary method further includes in response to determining that a cache eviction is to be performed, selecting a predetermined number of cache units from the plurality of cache units, and determining a score for each of the selected cache units based on the DH metadata of the respective metadata set. The DH metadata may include, for example, a validation count for each segment group or cache unit. A deprecated segment can be changed back to being valid, and the score for each of the selected cache units may further be determined based on a determined probability.

Abstract: Stream locality delta compression is disclosed. A previous stream indicated locale of data segments is selected. A first data segment is then determined to be similar to a data segment in the stream indicated locale.

Abstract: Exemplary methods for caching data in a cache device include determining characteristics of a plurality of file extents associated with a plurality of files stored in a random access memory (RAM) device. In one embodiment, the methods include deferring caching of the stored plurality of file extents in a cache device until a predetermined condition has occurred. According to one embodiment, the methods include, in response to determining the predetermined condition has occurred, packing a first portion of the plurality of file extents into a first cache unit based on the characteristics of the file extents, wherein file extents of the first cache unit are likely to be accessed within a predetermined period of time and evicted together from the cache device. The methods further include caching the first cache unit in the cache device and removing the cached file extents from the RAM device.

Abstract: Techniques for reducing and discouraging sending large scale emails are described herein. According to one embodiment, in response to a first email received from a sender to be sent to a list of recipients, a distribution cost of the first email is determined based on content of the first email and the recipients. An email client application is to present a first graphical user interface (GUI) page to the sender prompting a confirmation from the sender, where the first GUI page includes information indicating a size of the first email and a number of recipients, if the distribution cost of the first email is above a first predetermined threshold. In response to a positive confirmation from the sender, the first email is sent to the intended recipient.

Abstract: Exemplary methods for managing cache based on approximate least recently used (LRU) cache entries include maintaining a distributed data structure (DDS) of data elements, each corresponding to a cache entry of a plurality of cache entries, wherein each data element can be atomically accessed by multiple threads. In response to a cache eviction request from a first thread, determining an approximately LRU cache entry among the cache entries based on values atomically accessed from a first subset of the DDS of data elements, wherein the first subset of the DDS is atomically accessed using an atomic instruction without acquiring a lock to prevent another thread from accessing the first subset of the DDS to determine other approximately LRU cache entries among the cache entries, while allowing a second thread accessing a second subset of the DDS substantially concurrently. Evicting the determined approximately LRU cache entry.