Abstract: A cluster receives a request to store an object using replication or erasure coding. The cluster writes the object using erasure coding. A manifest is written that includes an indication of erasure coding and a unique identifier for each segment. The cluster returns a unique identifier of the manifest. The cluster receives a request from a client that includes a unique identifier. The cluster determines whether the object has been stored using replication or erasure coding. If using erasure coding, the method reads a manifest. The method identifies segments within the cluster using unique segment identifiers of the manifest. Using these unique segment identifiers, the method reconstructs the object. A persistent storage area of another disk is scanned to find a unique identifier of a failed disk. If using erasure coding, a missing segment previously stored on the disk is identified. The method locates other segments. Missing segments are regenerated.

Abstract: Digital objects within a fixed-content storage cluster use a page mapping table and a hash-to-UID table to store a representation of each object. For each object stored within the cluster, a record in the hash-to-UID table stores the object's hash value and its unique identifier (or portions thereof). To detect a duplicate of an object, a portion of its hash value is used as a key into the page mapping table. The page mapping table indicates a node holding a hash-to-UID table indicating currently stored objects in a particular page range. Finding the same hash value but with a different unique identifier in the table indicates that a duplicate of an object exists. Portions of the hash value and unique identifier may be used in the hash-to-UID table. Unneeded duplicate objects are deleted by copying their metadata to a manifest and then redirecting their unique identifiers to point at the manifest.

Abstract: Digital objects are stored and accessed within a fixed content storage cluster by using a page mapping table and a pages index. A stream is read from the cluster by using a portion of its unique identifier as a key into the page mapping table. The page mapping table indicates a node holding a pages index indicating where the stream is stored. A stream is written to the cluster by storing the stream on any suitable node and then updating a pages index stored within the cluster. The cluster recovers from a node failure by first replicating streams from the failed node and reallocating a page mapping table to create a new pages index. The remaining nodes send records of the unique identifiers corresponding to objects they hold to the new pages index. A node is added to the cluster by reallocating a page mapping table.

Abstract: Two or more numerical samples are read from a storage location (or many storage locations) and the samples are compared to generate a single bit of entropy. This method is repeated to populate two arrays which are also compared to generate a single bit. Comparison of the arrays is also repeated to generate a sequence of bits which are formed into computer words. The words are combined with a shift register to generate a seed block array of any length. The seed block array is used to seed a pseudo random number generator. The storage location is a register, accumulator, buffer, clock, address, memory location, etc., that changes periodically. Comparison may be performed by counting the number of certain types of bits in the two values and returning a one or zero depending upon the count.

Abstract: Each node and volume in a storage cluster makes a decision whether to reduce power consumption based on lack of requests from client applications and nodes over a time period. Node configuration parameters determine how long to wait until idling a node or volume, and how long to wait while idle before performing integrity checks. A bid value is calculated by each node and reflects how much it will cost for that node to write a file, read a file, or keep a copy. A node with the lowest bid wins, and nodes that are idle have a premium added to each bid to ensure that idle nodes are kept idle. In an archive mode, writes bids are reversed, nodes with less capacity submit lower bids, fuller nodes fill up faster and are then idled, while empty or near empty nodes may remain idle before winning a write bid.

Abstract: Each node and volume in a storage cluster makes a decision whether to reduce power consumption based on lack of requests from client applications and nodes over a time period. Node configuration parameters determine how long to wait until idling a node or volume, and how long to wait while idle before performing integrity checks. A bid value is calculated by each node and reflects how much it will cost for that node to write a file, read a file, or keep a copy. A node with the lowest bid wins, and nodes that are idle have a premium added to each bid to ensure that idle nodes are kept idle. In an archive mode, writes bids are reversed, nodes with less capacity submit lower bids, fuller nodes fill up faster and are then idled, while empty or near empty nodes may remain idle before winning a write bid.

Abstract: Any client application uses a namespace application to resolve its pathname in order to reference a computer file. Computer files are stored in a fixed-content storage cluster and are accessed by retrieving a unique identifier for the computer file using the namespace application. Any type of pathname scheme from any client application is supported by the namespace. The namespace application uses a bindings table to record bindings between objects including the start date and end date for each binding, and direction and separator data used in the pathname scheme. An attribute table in the namespace keeps track of each attribute and its value for each object of the namespace including a start date and an end date for each attribute. The namespace provides syntactic generality in that any pathname scheme of a client application can be resolved to identify a unique computer file in the storage cluster.

Abstract: Content-based addressing is used to navigate forward, backward and in a circular fashion through documents in a content space. To enable backward navigation, a descriptor file is created for a new version that contains not only a message digest for the new version, but also a message digest of the older document. A message digest is created for the descriptor file. A user navigates backward starting with the message digest of the descriptor file. To enable forward navigation, a mapping table maps a message digest of an older document into the message digest of the new version. A high-level descriptor file contains the message digest of the original document and the message digest of the mapping table. The message digest of the high-level descriptor file is returned to the user. A user navigates forward starting with the high-level descriptor file message digest.

Abstract: Each node and volume in a fixed-content storage cluster makes an independent decision whether to reduce power consumption based on lack of requests from client applications and nodes over a configurable time period. Node configuration parameters sleepAfter and wakeAfter respectively determine how long to wait until idling a node or volume, and how long to wait while idle before again performing integrity checks. A bid value is calculated by each node and reflects how much it will cost for that node to write a file, read a file, or keep a copy of the file. A node with the lowest bid wins, and nodes that are idle have a premium added to each bid to help ensure that idle nodes are kept idle. Normally, nodes with more capacity will submit a lower bid to write a file.

Abstract: Any client application uses a namespace application to resolve its pathname in order to reference a computer file. Computer files are stored in a fixed-content storage cluster and are accessed by retrieving a unique identifier for the computer file using the namespace application. Any type of pathname scheme from any client application is supported by the namespace. The namespace application uses a bindings table to record bindings between objects including the start date and end date for each binding, and direction and separator data used in the pathname scheme. An attribute table in the namespace keeps track of each attribute and its value for each object of the namespace including a start date and an end date for each attribute. The namespace provides syntactic generality in that any pathname scheme of a client application can be resolved to identify a unique computer file in the storage cluster.

Abstract: Content-based addressing is used to navigate forward, backward and in a circular fashion through documents in a content space. To enable backward navigation, a descriptor file is created for a new version that contains not only a message digest for the new version, but also a message digest of the older document. A message digest is created for the descriptor file. A user navigates backward starting with the message digest of the descriptor file. To enable forward navigation, a mapping table maps a message digest of an older document into the message digest of the new version. A high-level descriptor file contains the message digest of the original document and the message digest of the mapping table. The message digest of the high-level descriptor file is returned to the user. A user navigates forward starting with the high-level descriptor file message digest.

Abstract: A hash function used for content addressing is different from the hash function used for content verification. Adding a file to a database involves storing both hash function values in a table as pair. Verifying the integrity of a file believed to be a duplicate in a database, or when retrieving a file, makes use of the verification hash function. Files can be continuously checked. A multi-level database can be used. A second hash function can be added to an existing system. A verification hash function can be upgraded and more than one content verification hash function can be used. In a variation, a random number generator is used instead of a hash function for content addressing; the verification hash function is also used. Files addressed using a random number are added or retrieved from a database and their verification hash values are checked. Time stamps and digital signatures are used for security.

Abstract: Content-based addressing is used to navigate forward, backward and in a circular fashion through documents in a content space. To enable backward navigation, a descriptor file is created for a new version that contains not only a message digest for the new version, but also a message digest of the older document. A message digest is created for the descriptor file. A user navigates backward starting with the message digest of the descriptor file. To enable forward navigation, a mapping table maps a message digest of an older document into the message digest of the new version. A high-level descriptor file contains the message digest of the original document and the message digest of the mapping table. The message digest of the high-level descriptor file is returned to the user. A user navigates forward starting with the high-level descriptor file message digest.