Abstract: Techniques for improving data compression of a storage system in an online manner are described herein. According to one embodiment, in response to a sequence of data to be stored, the sequence of data is partitioned into a plurality of data chunks according to a predetermined chunking algorithm. A sketch for each of the data chunks is generated based on one or more features extracted from the data chunk. Each of the data chunks of the sequence of data is associated with one of a plurality of groups based on the sketch, wherein each group is represented by a sketch. The data chunks of each group are compressed and stored in a compression region of the storage systems, such that similar data chunks are compressed and stored in the same compression region.

Abstract: A method for storing data in a data storage system by partitioning the data into a plurality of data chunks and generating representative data for each of the plurality of chunks by applying a predetermined algorithm to each chunk of the plurality of chunks. Subsequently, the representative data is compared and sorted. Representative data for base data chunks and representative data for other data chunks that can be stored relative to the base data chunks are identified by evaluating the sorted set of representative data. Finally, each of the other data chunks identified as those that can be stored relative to a base data chunk are stored in the data storage system as the difference between the data chunk and a base data chunk.

Abstract: A data processing system and methods for performing cache eviction are disclosed. An exemplary method includes maintaining a metadata set for each cache unit of a cache device, wherein the cache device comprises a plurality of cache units, each cache unit having a plurality of segments, wherein each metadata set includes deletion hints (DH) metadata indicating whether the plurality of segments of a corresponding cache unit are valid. The exemplary method further includes in response to determining that a cache eviction is to be performed, selecting a predetermined number of cache units from the plurality of cache units, and determining a score for each of the selected cache units based on the DH metadata of the respective metadata set. The DH metadata may include, for example, a validation count for each segment group or cache unit. A deprecated segment can be changed back to being valid, and the score for each of the selected cache units may further be determined based on a determined probability.

Abstract: Data objects of a file are cached in a cache memory of a storage system. An access sequence of the cached data objects is determined based on metadata of the file. In response to a request for cache space reclamation, a data object is evicted from the cache memory whose next access is a farthest amongst the cached data objects based on the access sequence of the data objects.

Abstract: A computer-implemented method for compressing a data set, the method comprising receiving a first data block of the data set, selecting automatically by a compression management module a compression module from a plurality of compression modules to apply to the first data block based on projected compression efficacy or resource utilization, and compressing the first data block with the selected compression module to generate a first compressed data block.

Abstract: Techniques for reducing and discouraging sending large scale emails are described herein. According to one embodiment, in response to a first email received from a sender to be sent to a list of recipients, a distribution cost of the first email is determined based on content of the first email and the recipients. An email client application is to present a first graphical user interface (GUI) page to the sender prompting a confirmation from the sender, where the first GUI page includes information indicating a size of the first email and a number of recipients, if the distribution cost of the first email is above a first predetermined threshold. In response to a positive confirmation from the sender, the first email is sent to the intended recipient.

Abstract: Techniques for determining vulnerability of disks are described herein. According to one embodiment, for each of a plurality of disks representing a redundant array of independent disks (RAID), a reallocated sector count associated with the disk is obtained, the reallocated sector count representing a number of sectors that have been reallocated due to an error of a storage transaction to the disk. A failure probability of the disk given the obtained reallocated sector count is determined using a predictive model, wherein the predictive model was generated based on history operating data of a set of known disks. Thereafter, a failure probability of at least two of the disks in the RAID is determined based on the failure probability of each of the disks to determine vulnerability of the RAID.

Abstract: Techniques for improving data compression of a storage system in an online manner are described herein. According to one embodiment, in response to a sequence of data to be stored, the sequence of data is partitioned into a plurality of data chunks according to a predetermined chunking algorithm. A sketch for each of the data chunks is generated based on one or more features extracted from the data chunk. Each of the data chunks of the sequence of data is associated with one of a plurality of groups based on the sketch, wherein each group is represented by a sketch. The data chunks of each group are compressed and stored in a compression region of the storage systems, such that similar data chunks are compressed and stored in the same compression region.

Abstract: A domain name server request is received from a local domain name server. Operational performance parameters of a web host and a first content distribution network are received. Aggregate operational performance parameters of a second content distribution network are received. A first record is sent to the local domain name server that indicates the domain name server request is to be handled by the web host on a condition that a performance value is below a predetermined operational performance level threshold. A second record is sent to the local domain name server that indicates the domain name server request is to be handled by the first or the second content distribution network on a condition that the performance value is equal to or above the predetermined operational performance level threshold.

Abstract: A cost function is determined for assigning first deduplicating storage units of a first storage system for replication onto second deduplicating storage units of a second storage system. One or more of the first storage units in the first storage system are assigned to one or more of the second storage units in the second storage system based on a minimized cost resulting from the cost function.

Abstract: In response to a request for stored data, retrieving an index entry, based on an identifier of the requested data, corresponding to an indexed block of storage containing the requested data. The index entry includes a start location of an indexed storage block and sub-block locators that identify the start of one or more sub-blocks within the indexed storage block. The sub-block containing the requested data is determined and the corresponding sub-block locator is read to find the starting location of the sub-block. Without reading the entire indexed storage block, the sub-block may be read from the starting location of the sub-block, decompressed, and the decompressed requested data read from the sub-block may be transmitted to the client. In this way, fewer I/O operations are needed that read the requested data, and memory needed for storing index information is minimized.

Abstract: Techniques for improving data compression of a deduplicated storage system are described herein. According to one embodiment, the similarity of a plurality of data chunks stored in one or more first storage areas of the storage system is determined based on a plurality of sketches, each describing characteristics of one of the data chunks. The data chunks are grouped into a plurality of groups of similar data chunks based on the similarity of the data chunks. The groups of similar data chunks are compressed, such that similar data chunks are compressed close to each other.

Abstract: A request is received to read a portion of a file from the storage system. In response to the request, future requests to the file are predicted based in part on the received request and metadata of the file. Access of a subsequent portion of the file is optimized based on the predicted future requests and the data contained in the portion of the file is then returned.

Abstract: A computer-implemented method for indexing content stored in a cache memory device is disclosed. The method starts with maintaining a fingerprint index having a plurality of fingerprint entries, each mapping a fingerprint to a storage location of a cache memory device, where the cache memory device caches some of data blocks stored in a persistent storage device of a storage system, and where the fingerprint index is a partial index indexing a portion of data stored in the cache memory device. In response to receiving a request to insert a new fingerprint, the method continues with evicting one of the fingerprint entries according to a predetermined eviction algorithm and inserting the new fingerprint into the evicted fingerprint entry.

Abstract: Operational performance parameters of a web host and a first content distribution network are monitored. Capacity and health statistics of a second content distribution network are received. A determination is made whether a customer request is to be served by the first content distribution network or the second content distribution network based on a performance threshold. Usage of a primary service provider, the web host, the first content distribution network, and the second content distribution network are tracked.

Abstract: Techniques for improving data compression of a storage system are described herein. According to one embodiment, a first sequence of data is partitioned into a plurality of data chunks in a first sequence order according to a predetermined chunking algorithm. The similarity of the data chunks is determined based on data patterns of the data chunks. The data chunks are reorganized into a second sequence order based on the similarity of the data chunks, the second sequence order being different from the first sequence order. The reorganized data chunks are compressed in the second sequence order into a second sequence of data, such that similar data chunks are stored and compressed together within the second sequence of data.

Abstract: A computer-implemented method for indexing content stored in a cache memory device is disclosed. The method starts with maintaining a file index having a plurality of extent entries, each extent entry corresponding to one of a plurality of file extents stored in a cache memory device that caches data stored in a persistent storage device of a storage system. The method continues with maintaining a fingerprint index having a plurality of fingerprint entries, each mapping a fingerprint to a data region of a file indexed in the file index, wherein each fingerprint indexed in the fingerprint index is retrieved from metadata stored in the persistent storage device of the storage system when one or more corresponding data chunks were accessed, and deduplicating and accessing the file extents stored in the cache memory device using the file index and the fingerprint index.

Abstract: A system and method for protecting streams in a mixed infrastructure includes determining processing elements that are to access a data stream in a stream processing environment and determining a security level for each processing element. Keys are generated per stream per processing element in accordance with the security level. The keys are associated with processing elements in an access control list in a location accessible by producing and consuming processing elements. The stream is decrypted for processing using keys released upon authenticating processing elements in accordance with the access control list. At security boundaries, the stream is re-encrypted in accordance with a next processing element.

Abstract: A computer-implemented method for caching content in a cache memory device is disclosed. The method starts with receiving a request for accessing a first data block associated with a first file, and a file manager provides access of the first data block in a persistent storage device of a storage system. The file manager then caches the first data block in a cache memory device including deduplicating the first data block, wherein at least some of data blocks stored in the cache memory device are deduplicated data blocks, and wherein at least one of the data blocks is referenced by different regions of an identical file or different files.

Abstract: A system and method for protecting streams in a mixed infrastructure includes determining processing elements that are to access a data stream in a stream processing environment and determining a security level for each processing element. Keys are generated per stream per processing element in accordance with the security level. The keys are associated with processing elements in an access control list in a location accessible by producing and consuming processing elements. The stream is decrypted for processing using keys released upon authenticating processing elements in accordance with the access control list. At security boundaries, the stream is re-encrypted in accordance with a next processing element.