Abstract: A system and method of offline annotation of future access are disclosed. According to one embodiment, a request is received at a storage system to read a portion of a file stored in the storage system. In response to the request, chunks of the file are cached in a cache memory of the storage system. In response to a request for cache space reclamation, the system then determines future request to the file based in part on a next access auxiliary table (NAAT) associated with the file, which was created prior to receiving the request to read and stored in a persistent storage location of the storage system. Based on the determination, the system evicts from the cache memory at least one chunk of a read unit (RU) whose next access is a furthest among the cached chunks.

Abstract: A computer-implemented method for indexing content stored in a cache memory device is disclosed. The method starts with maintaining a file index having a plurality of extent entries, each extent entry corresponding to one of a plurality of file extents stored in a cache memory device that caches data stored in a persistent storage device of a storage system. In response to receiving a request to read a first file region of a first file, the method continues with retrieving first data block from the persistent storage device that contains the first data block, caching the first data block at a first storage location of the cache memory device; and creating a first extent entry in the file index having at least a first node, where the first node includes an address of the first storage location and a first bitmap indicating which data blocks are valid.

Abstract: A computer-implemented method for caching content in a cache memory device is disclosed. The method starts with receiving, at a cache manager, one or more data chunks to be cached in a cache memory device, where the one or more data chunks are retrieved from a persistent storage disk of a storage system in response to a read request of a region of a file. Then the one or more data chunks of a file extent is compressed using a predetermined compression algorithm, and the file extent is packed into a write-evict unit (WEU) maintained in a random-access memory (RAM) that has been open to store a plurality of file extents. In response to determining that the WEU is full, the cache manager writes the WEU from the RAM into the cache memory device.

Abstract: Data objects of a file are cached in a cache memory of a storage system, where the data objects of the file are accessed sequentially via access windows in a chain. It is estimated whether a next access of a data object of the file likely occurs in which of the access windows using respective bloom filters associated with the access windows. In response to a request for cache space reclamation, a data object is evicted from the cache memory that will be likely accessed in a farthest access window in the chain from a current access window based on the estimation.

Abstract: Operational performance parameters of a web host and a first content distribution network are monitored. Capacity and health statistics of a second content distribution network are received. A determination is made whether a customer request is to be served by the first content distribution network or the second content distribution network based on a performance threshold. Usage of a primary service provider, the web host, the first content distribution network, and the second content distribution network are tracked.

Abstract: Methods, systems, and products generate web pages using elidable links to additional content. When a link is selected in a web page, elision is used to automatically remove a URL and its associated content from the web page, thus reducing previously visited material. When a user selects an elision-enabled link, the link is not displayed during subsequent displaying of the same web page.

Abstract: A service model, integrated system, and method for enabling a service provider to deliver an integrated web hosting and content distribution service offering, which affords assured operational performance service levels, regardless of whether the customer's web site is served by the service provider's hosting center, the service provider's content distribution network (CDN), by a third party web host, or by a third party CDN. A monitoring system of the primary service provider receives detailed capacity and health statistics from any CDN under the operational control of the primary service provider, receives aggregate capacity and health statistics from other CDNs not under the operational control of the primary service provider. A redirection system then decides to which web host, content distribution network, or combination thereof, user requests for content are directed in order that operational performance service levels are maintained.

Abstract: A computer-implemented method for indexing content stored in a cache memory device is disclosed. The method starts with in response to receiving a first request for caching a first file extent associated with a first file in a cache memory device, generating a first fingerprint based on content of the first file extent. Then the method continues with searching in a fingerprint index based on the first fingerprint to determine whether the first file extent has been stored in the cache memory. In response to determining that a fingerprint entry matching the first fingerprint is found, the method then continues with associating a first identifier identifying the first file extent and the first file with a storage location of the cache memory device obtained from the matching fingerprint entry, without storing the first file extent in the cache memory device.

Abstract: A mechanism is provided that aggregates data in a way that permits data to be deleted efficiently, while minimizing the overhead necessary to support bulk deletion of data. A request is received for automatic deletion of segments in a container and a waterline is determined for the container. A determination is made if at least one segment in the container falls below the waterline. Finally, in response to one segment falling below the waterline, the segment from the container is deleted. Each object has an associated creation time, initial retention value, and retention decay curve (also known as a retention curve). At any point, based on these values and the current time, the object's current retention value may be computed. The container system continually maintains a time-varying waterline: at any point, objects with a retention value below the waterline may be deleted.

Abstract: A method for storing data in a data storage system by partitioning the data into a plurality of data chunks and generating representative data for each of the plurality of chunks by applying a predetermined algorithm to each chunk of the plurality of chunks. Subsequently, the representative data is compared and sorted. Representative data for base data chunks and representative data for other data chunks that can be stored relative to the base data chunks are identified by evaluating the sorted set of representative data. Finally, each of the other data chunks identified as those that can be stored relative to a base data chunk are stored in the data storage system as the difference between the data chunk and a base data chunk.

Abstract: A cooperative data stream processing system is provided that utilizes a plurality of independent, autonomous and possibly heterogeneous sites in a cooperative arrangement to process user-defined job requests over dynamic, continuous streams of data. A method is provided to organize the distributed sites into a plurality of virtual organizations that can be further combined and virtualized into virtualized virtual organizations. These virtualized virtual organizations can also include additional distributed sites and existing virtualized virtual organizations and all members of a given virtualized virtual organization can share data and processing resources in order to process jobs on either a task-based or goal-based allocation mechanism. The virtualized virtual organization is created dynamically using ad-hoc collaborations among the members and is arranged in either a federated or cooperative architecture. Collaborations between members is either tightly-coupled or loosely coupled.

Abstract: A system for directing for storage comprises a processor and a memory. The processor is configured to determine a segment overlap for each of a plurality of nodes. The processor is further configured to determine a selected node of the plurality of nodes based at least in part on the segment overlap for each of the plurality of nodes and based at least in part on a selection criteria. The memory is coupled to the processor and configured to provide the processor with instructions.

Abstract: Techniques for detecting unwanted data are described herein. In one embodiment, a request is received for storing a data object in a storage system from a client over a network, where the request includes first representative data representing the data object without including actual content of the data object. It is detected whether the data object contains unwanted content by comparing the first representative data with second representative data without accessing the actual content of the data object, where the second representative data represents the unwanted content. A response is transmitted to the client over the network indicating whether the data object is likely to contain the unwanted object based on comparison of the first and second representative data.

Abstract: Techniques for searching data in a storage system are described herein. In one embodiment, in response to a request for searching target data in a storage system, first representative data for the target data being searched are generated by applying a predetermined algorithm to at least a portion of the target data. The first representative data are searched and compared with second representative data representing one or more data sets stored in the storage system. It is indicated a likelihood that the target data or similar content has been found in the storage system based on the search and comparison.

Abstract: A system for directing for storage includes a processor and a memory. The processor is configured to determine a segment overlap for each of a plurality of nodes. The processor is further configured to determine a selected node of the plurality of nodes based at least in part on the segment overlap for each of the plurality of nodes and based at least in part on a selection criteria. The memory is coupled to the processor and configured to provide the processor with instructions.

Abstract: Techniques for replicating data chunks in a storage system are described herein. In one embodiment, in response to a request for replicating data chunks of a source storage system having a first average chunk size to a target storage system having a second average chunk size, a new chunk size is determined based on metadata of the data chunks in view of an average chunk size of the target storage system. The data chunks are resized based on the new chunk size to generate resized data chunks. The resized data chunks are transmitted from the source storage system to the target storage system for replication.

Abstract: A cooperative data stream processing system is provided that utilizes a plurality of independent, autonomous and potentially heterogeneous sites in a cooperative arrangement to process user-defined inquiries over dynamic, continuous streams of data. The system derives jobs from the inquiries and these jobs are executed on the various distributed sites by executing applications containing processing elements on these sites. Therefore, components of a given job can be executed simultaneously and in parallel on a plurality of sites within in the system. The sites associated with a given job execution have the need to share data, both primal and derived. A tunnel mechanism is provided that establishes tunnels between pairs of sites within the system. Each tunnel includes either a sink processing element on an originating site and a source processing element on a destination site or a gateway processing element on each site and a network connection between the sink and source processing elements.

Abstract: A storage system includes a plurality of data vats, and a processor including an optimizing unit that optimizes a value of data stored in the storage system. The optimizing unit optimizes the value by computing and implementing an optimal decision for allocating new data to a first data vat of the plurality of data vats, moving existing data from at least a second data vat of the plurality of data vats to the first data vat, and deleting existing data from the first data vat, based on an amount of data in each of the plurality of data vats.

Abstract: The method and apparatus collect file recipes from deduplicated data storage systems, the file recipes consist of a list of fingerprints of data chunks of a file. Detailed meta-data for each unique data chunk is also collected. In an offline process, research and analysis can be performed on either the meta-data itself or on a reconstruction of a full trace of meta-data constructed by matching recipe fingerprints to the corresponding meta-data. The method and system can generate the full meta-data trace efficiently in an on-line or off-line process. Typical deduplicated storage systems achieve 10× or higher deduplication rates, and the meta-data collection is faster than processing all of the original files and produces compact meta-data that is smaller to store.

Abstract: Techniques are disclosed for optimizing schedules used in implementing plans for performing tasks in data processing systems. For example, an automated method of negotiating for resources in a data processing system, wherein the data processing system comprises multiple sites, comprises a negotiation management component of a computer system at a given one of the sites performing the following steps. One or more tasks from at least one source of one or more plans are obtained. Each plan is annotated with one or more needed resources and one or more potential resource providers at one or more sites in the data processing system. An optimized resource negotiation schedule based on the one or more obtained tasks is computed. The schedule comprises an order in which resources are negotiated.