Abstract: Techniques for evaluating deduplication effectiveness of data chunks in a storage system are described herein. In one embodiment, metadata of first data chunks associated with a deduplicated storage system is examined, where the first data chunks have been partitioned according to a first chunk size. A second chunk size is calculated based on the examination of the metadata of first data chunks. Metadata of the first data chunks is merged according to the second chunk size to represent second data chunks to which the first data chunks would have been merged. A deduplication rate of the second data chunks is determined based on the merged metadata.

Abstract: The method and apparatus collect file recipes from deduplicated data storage systems, the file recipes consist of a list of fingerprints of data chunks of a file. Detailed meta-data for each unique data chunk is also collected. In an offline process, research and analysis can be performed on either the meta-data itself or on a reconstruction of a full trace of meta-data constructed by matching recipe fingerprints to the corresponding meta-data. The method and system can generate the full meta-data trace efficiently in an on-line or off-line process. Typical deduplicated storage systems achieve 10× or higher deduplication rates, and the meta-data collection is faster than processing all of the original files and produces compact meta-data that is smaller to store.

Abstract: Backup requirements of a new client and one or more existing clients stored in a first set of first storage units are determined. Data of the one or more existing clients is then migrated from the first set of storage units to a first storage unit that is selected from a second set of storage units based on a cost calculated using a cost function based on at least backup requirements of the new and existing clients. Subsequent data of the new client and the existing clients is stored in the first storage unit.

Abstract: A request for allocating a storage unit of a storage system is received to back up data of one or more clients. The storage system includes multiple storage units and each storage unit storing data that is deduplicated within each storage unit. In response to the request, one or more of the storage units are selected based on an amount of deduplicated data that would be stored in each of the storage units after storing the data of the one or more clients. The selected one or more storage units are allocated to the one or more clients to back up the data of the one or more clients.

Abstract: A system and method for composing a stream servicing environment which considers all stakeholders includes identifying service component requirements needed for processing a data stream, and determining available service elements for processing the stream. Feasible service environments are constructed based upon the available service elements and the service component requirements. Efficiency measures are computed for each feasible service environment considering all stakeholders. A best service environment is determined based upon the efficiency measures.

Abstract: A cooperative data stream processing system utilizing a plurality of independent, autonomous and heterogeneous sites in a cooperative arrangement process user-defined job requests over dynamic, continuous streams of data. A distributed plan is created that identifies the processing elements that constitute a job that is derived from user-defined inquiries. These processing elements are arranged into subjobs that are mapped to various sites within the system for execution. The jobs are executed across the plurality of distributed sites in accordance with the distributed plan. The distributed plan also includes requirements for monitoring and back-up of the execution sites in the event of a failure on one of those sites. Execution of the jobs in accordance with the distributed plan is facilitated by the identification of an owner site to which the distributed plan is communicated and which is responsible for driving the execution of the distributed plan.

Abstract: An information processing system comprises first and second levels of a storage hierarchy, wherein accessing information in the first level consumes more energy than accessing information in the second level; and a processor for writing information to the second level of storage based on energy-conserving criteria. The energy-conserving criteria comprise a set of heuristics, including system state information and user preferences.

Abstract: A cooperative data stream processing system is provided that utilizes a plurality of independent, autonomous and possibly heterogeneous sites in a cooperative arrangement to process user-defined job requests over dynamic, continuous streams of data. The sites negotiate peering relationships to share data and processing resources to handle the submitted job requests. These peering relationships can be cooperative or federated and can be expressed using common interest policies. Each site within the system runs an instance of a system architecture for processing job requests and is therefore a self-contained, fully functional instance of the cooperative data stream processing system.

Abstract: A storage system includes a plurality of data vats, and a processor including an optimizing unit that optimizes a value of data stored in the storage system. The optimizing unit optimizes the value by computing and implementing an optimal decision for allocating new data to a first data vat of the plurality of data vats, moving existing data from at least a second data vat of the plurality of data vats to the first data vat, and deleting existing data from the first data vat, based on an amount of data in each of the plurality of data vats.

Abstract: A service model, integrated system, and method for enabling a service provider to deliver an integrated web hosting and content distribution service offering, which affords assured operational performance service levels, regardless of whether the customer's web site is served by the service provider's hosting center, the service provider's content distribution network (CDN), by a third party web host, or by a third party CDN. A monitoring system of the primary service provider receives detailed capacity and health statistics from any CDN under the operational control of the primary service provider, receives aggregate capacity and health statistics from other CDNs not under the operational control of the primary service provider. A redirection system then decides to which web host, content distribution network, or combination thereof, user requests for content are directed in order that operational performance service levels are maintained.

Abstract: A system and method for securely processing sensitive streams in a mixed infrastructure includes analyzing a stream to determine data sensitivity. A likelihood that processing elements employed to process the stream would result in a risk to sensitive information is determined. At least a portion of the data stream having sensitive information is transferred to a secure processing environment to ensure security of the data stream during processing.

Abstract: A method (and system) of storing data in a value-based storage system, includes optimizing a value of data stored in the value-based storage system.

Abstract: A failure recovery framework to be used in cooperative data stream processing is provided that can be used in a large-scale stream data analysis environment. Failure recovery supports a plurality of independent distributed sites, each having its own local administration and goals. The distributed sites cooperate in an inter-site back-up mechanism to provide for system recovery from a variety of failures within the system. Failure recovery is both automatic and timely through cooperation among sites. Back-up sites associated with a given primary site are identified. These sites are used to identify failures within the primary site including failures of applications running on the nodes of the primary site. The failed applications are reinstated on one or more nodes within the back-up sites using job management instances local to the back-up sites in combination with previously stored state information and data values for the failed applications.

Abstract: A failure recovery framework to be used in cooperative data stream processing is provided that can be used in a large-scale stream data analysis environment. Failure recovery supports a plurality of independent distributed sites, each having its own local administration and goals. The distributed sites cooperate in an inter-site back-up mechanism to provide for system recovery from a variety of failures within the system. Failure recovery is both automatic and timely through cooperation among sites. Back-up sites associated with a given primary site are identified. These sites are used to identify failures within the primary site including failures of applications running on the nodes of the primary site. The failed applications are reinstated on one or more nodes within the back-up sites using job management instances local to the back-up sites in combination with previously stored state information and data values for the failed applications.

Abstract: A service model, integrated system, and method for enabling a service provider to deliver an integrated web hosting and content distribution service offering, which affords assured operational performance service levels, regardless of whether the customer's web site is served by the service provider's hosting center, the service provider's content distribution network (CDN), by a third party web host, or by a third party CDN. A monitoring system of the primary service provider receives detailed capacity and health statistics from any CDN under the operational control of the primary service provider, receives aggregate capacity and health statistics from other CDNs not under the operational control of the primary service provider. A redirection system then decides to which web host, content distribution network, or combination thereof, user requests for content are directed in order that operational performance service levels are maintained.

Abstract: A redundancy elimination mechanism is provided, which applies aspects of duplicate block elimination and delta encoding at the block level. The redundancy elimination mechanism divides file objects into content-defined blocks or “chunks.” Identical chunks are suppressed. The redundancy elimination mechanism also performs resemblance detection on remaining chunks to identify chunks with sufficient redundancy to benefit from delta encoding of individual chunks. Any remaining chunks that do not benefit from delta encoding are compressed. Resemblance detection is optimized by merging groups of fingerprints into super fingerprints. This merging can be constructed to ensure that if two objects have a single super fingerprint in common, they are extremely likely to be substantially similar.

Abstract: A mechanism is provided that aggregates data in a way that permits data to be deleted efficiently, while minimizing the overhead necessary to support bulk deletion of data. A request is received for automatic deletion of segments in a container and a waterline is determined for the container. A determination is made if at least one segment in the container falls below the waterline. Finally, in response to one segment falling below the waterline, the segment from the container is deleted. Each object has an associated creation time, initial retention value, and retention decay curve (also known as a retention curve). At any point, based on these values and the current time, the object's current retention value may be computed. The container system continually maintains a time-varying waterline: at any point, objects with a retention value below the waterline may be deleted.

Abstract: Techniques for automatically detecting similar subsets (e.g., fragments) in electronic documents such as dynamic content-based data, e.g., web pages. The techniques of the invention may perform a systematic analysis of the web pages with respect to one or more of their information sharing behavior, their personalization characteristics, and their change pattern over time. Thus, the invention may be applied to discover fragments in web pages of web sites that are most beneficial for caching the contents of the web site. The present invention also comprises techniques for publishing electronic documents with automatic fragment detection.

Abstract: A system and method for optimizing a storage system to support short data object lifetimes and highly utilized storage space are provided. With the system and method, data objects are clustered based on when they are anticipated to be deleted. When an application stores data, the application provides an indicator of the expected lifetime of the data, which may be a retention value, a relative priority of the data object, or the like. Data objects having similar expected lifetimes are clustered together in common data structures so that clusters of objects may be deleted efficiently in a single operation. Expected lifetimes may be changed by applications automatically. The system automatically determines how to handle these changes in expected lifetime using one or more of copying the data object, reclassifying the container in which the data object is held, and ignoring the change in expected lifetime for a time to investigate further changes in expected lifetime of other data objects.

Abstract: Techniques for automatically detecting similar subsets (e.g., fragments) in electronic documents such as dynamic content-based data, e.g., web pages. The techniques of the invention may perform a systematic analysis of the web pages with respect to one or more of their information sharing behavior, their personalization characteristics, and their change pattern over time. Thus, the invention may be applied to discover fragments in web pages of web sites that are most beneficial for caching the contents of the web site. The present invention also comprises techniques for publishing electronic documents with automatic fragment detection.