Abstract: Among other things, a technique of data deduplication using multiple devices includes presenting, to host devices, a quantity of logical data storage greater than physical data storage available on a storage area network (SAN) device, by deduplicating at least some of the data received from the host devices for storage at the SAN device in a manner that is coordinated with over-provisioning capability of the SAN device.

Abstract: A method for storing data items in a data repository that allows a client storing a data item to lose the ability to either retrieve the stored data item or enable its retrieval, while preserving controlled access to the stored data item indefinitely. The client storing the data item encrypts it using an encryption key that it creates, which may be derived from the content of the data item. The encryption key is also stored in the data repository in an encrypted form, with its encryption making use of a public key. The method allows storage space to be shared for identical data items encrypted and stored by independent clients of the data repository.

Abstract: In one aspect, in general, a method for managing data includes receiving, at a data deduplication engine of a computer system, data to be stored in a data storage system, computing a first value corresponding to at least one portion of the received data, determining that the first value matches a second value stored in a collection of identifiers, each identifier corresponding to a particular portion of data stored in the data storage system, each identifier comprising a computed value corresponding to the particular portion of data and including metadata indicating a location where the particular portion of data is being stored in the data storage system, and based on the determination that the first value matches a second value stored in the collection of identifiers, reallocating one or more physical locations from corresponding one or more virtual locations to different respective one or more virtual locations.

Abstract: A method for organizing a storage system that is scalable and fault tolerant, the method including grouping together a number D of storage elements to form the storage system, where D is more than one, constructing a storage assignment table that includes table entries, computing, for each of the storage elements, an available capacity that depends on constraints on the placement of redundant data within the storage system, summing the available capacities to form a total available capacity for the storage system; and assigning the table entries in the storage assignment table to each identify one of the storage elements, wherein the available capacity of each of the storage elements is its effective storage capacity when used as part of the storage system, wherein not all of the D storage elements that form the storage system have the same available capacity, and wherein the fraction of all table entries that identify a one of the storage elements depends upon its fraction of the total available capacity.

Abstract: A method by which a disk-based distributed data storage system is organized for protecting historical records of stored data entities. The method comprises recording distinct states of an entity, corresponding to different moments of time, as separate entity versions coexisting within the distributed data storage system, and assigning expiration times to the entity versions independently within each of a plurality of storage sites according to a shared set of rules, before which times deletion is prohibited.

Abstract: A method by which a disk-based distributed data storage system is organized for protecting historical records of stored data entities. The method comprises recording distinct states of an entity, corresponding to different moments of time, as separate entity versions coexisting within the distributed data storage system, and assigning expiration times to the entity versions independently within each of a plurality of storage sites according to a shared set of rules, before which times deletion is prohibited.

Abstract: A method by which a disk-based distributed data storage system is organized for protecting historical records of stored data entities. The method comprises recording distinct states of an entity, corresponding to different moments of time, as separate entity versions coexisting within the distributed data storage system, and assigning expiration times to the entity versions independently within each of a plurality of storage sites according to a shared set of rules, before which times deletion is prohibited.

Abstract: A method for encoding a block of data to allow it to be stored or transmitted correctly in the face of accidental or deliberate modifications, the method including constructing a number n greater than one of original components, each of which is derived from the block and each of which is smaller than the block, and combining original components to construct a number m greater than one of new components, wherein each of the new components is smaller than the sum of the sizes of the original components combined to produce it, wherein the block can be reconstructed from any set of n different components selected from the original components and new components, and wherein a set of n different components selected from the original components and new components contains more redundant information about the block than the set of n original components.

Abstract: A method for keeping track, in an electronic data processing system, of when references made to an element that is stored in memory have all been removed, the method being designed to deal with failures in a manner that does not falsely conclude there are no references.

Abstract: A method by which a disk-based distributed data storage system is organized for protecting historical records of stored data entities. The method comprises recording distinct states of an entity, corresponding to different moments of time, as separate entity versions coexisting within the distributed data storage system, and assigning expiration times to the entity versions independently within each of a plurality of storage sites according to a shared set of rules, before which times deletion is prohibited.

Abstract: A method by which a disk-based distributed data storage system is organized for protecting historical records of stored data entities. The method comprises recording distinct states of an entity, corresponding to different moments of time, as separate entity versions coexisting within the distributed data storage system, and assigning expiration times to the entity versions independently within each of a plurality of storage sites according to a shared set of rules, before which times deletion is prohibited.

Abstract: A method by which a disk-based distributed data storage system is organized for protecting historical records of stored data entities. The method comprises recording distinct states of an entity, corresponding to different moments of time, as separate entity versions coexisting within the distributed data storage system, and assigning expiration times to the entity versions independently within each of a plurality of storage sites according to a shared set of rules, before which times deletion is prohibited.

Abstract: A method by which a disk-based distributed data storage system is organized for protecting historical records of stored data entities. The method comprises recording distinct states of an entity, corresponding to different moments of time, as separate entity versions coexisting within the distributed data storage system, and assigning expiration times to the entity versions independently within each of a plurality of storage sites according to a shared set of rules, before which times deletion is prohibited.

Abstract: In general, the invention features methods by which more than one client program connected to a network stores the same data item on a storage device of a data repository connected to the network. In one aspect, the method comprises encrypting the data item using a key derived from the content of the data item, determining a digital fingerprint of the data item, and storing the data item on the storage device at a location or locations associated with the digital fingerprint. In a second aspect, the method comprises determining a digital fingerprint of the data item, testing for whether the data item is already stored in the repository by comparing the digital fingerprint of the data item to the digital fingerprints of data items already in storage in the repository, and challenging a client that is attempting to deposit a data item already stored in the repository, to ascertain that the client has the full data item.

Abstract: A method for turning variable-length file path names into fixed-length object identifiers in a data repository, the method comprising, assigning a fixed-value identifier to a file system in the data repository, computing a first deterministic pseudo-random function that combines a first component of a plurality of file system pathname components associated with a file with the fixed-value identifier, computing a second deterministic pseudo-random function that combines a second component of the plurality of file system pathname components with the result of the first function, and using at least one of the computed deterministic pseudo-random functions to produce a fixed-length object identifier for the file.

Abstract: A method for constructing an index suitable for indexing a large set of records identified by long generally randomly distributed record names and for answering membership queries about the set, the method comprising assigning each different record a different record name, determining that a new record name is not already represented in the index by checking an entry in a first level index that is shorter than the complete new record name, and determining that a queried record name is already represented in the index by determining that the queried record name is represented in a second level index that contains enough information to reconstruct the complete queried record name.