Abstract: A client application is provided with a view of a subset of objects within an object store. A REST protocol, REST interface, or API may be used to access the objects. A coupling server may include a filter and a generator. The generator may automatically generate object metadata for newly created files that matches a filter. A client application may mount a drive on a computer based on metadata objects, such as providing a filter on object metadata to view files a user wants to access; or editing, reading, and updating object metadata as a file.

Abstract: Segments of a local range can appear multiple times in the sequence of erasure sets in a manifest without the data being duplicated. A subset of the content of an erasure set can be included in the logical object. It is not required that all erasure sets in an object use the same encoding. A manifest can indicate one or more portions of the logical object for which no data has been provided, and for which a “fill” pattern is specified. A portion of data can be inserted at any point inside the object by inserting a reference to one or more segment sets containing that data into the manifest. If the desired insertion point is within an existing segment set, then that manifest set can be “split” by incorporating two references to it, with the first specifying the range of data before the insertion point, and the second specifying the range of data after the insertion point. Many operations are thus enabled on an erasure-coded object in Object Storage such as “append”, “insert,” “overwrite” and “merge.

Abstract: Digital objects are stored and accessed within a fixed content storage cluster by using a page mapping table and a pages index. A stream is read from the cluster by using a portion of its unique identifier as a key into the page mapping table. The page mapping table indicates a node holding a pages index indicating where the stream is stored. A stream is written by storing the stream on any suitable node and then updating a pages index stored within the cluster responsible for knowing the location of digital objects having unique identifiers that fall within a particular address range. The cluster recovers from a node failure by first replicating streams from the failed node and reallocating a page mapping table to create a new pages index. The remaining nodes send records of the unique identifiers corresponding to objects they hold to the new pages index.

Abstract: A cluster receives a request to store an object using replication or erasure coding. The cluster writes the object using erasure coding. A manifest is written that includes an indication of erasure coding and a unique identifier for each segment. The cluster returns a unique identifier of the manifest. The cluster receives a request from a client that includes a unique identifier. The cluster determines whether the object has been stored using replication or erasure coding. If using erasure coding, the method reads a manifest. The method identifies segments within the cluster using unique segment identifiers of the manifest. Using these unique segment identifiers, the method reconstructs the object. A persistent storage area of another disk is scanned to find a unique identifier of a failed disk. If using erasure coding, a missing segment previously stored on the disk is identified. The method locates other segments. Missing segments are regenerated.

Abstract: A method writes a data block from a client application onto object storage. Object storage receives a write request with a data block over an API from a coupling server, the data block being an update to a target file stored within object storage. Next, object storage receives a completion request from the coupling server for the target file. Then, object storage receives a final manifest from the coupling server, the final manifest includes for each data block of the target file an identifier for said data block, a start location, an end location or a size, and a unique identifier for the data block; the manifest has the data blocks ordered by positions in the target file. Finally, object storage assembles of new version of the target file using the final manifest, the received data block and other data blocks of the target file, and writes a new version into object storage.

Abstract: Digital objects are stored and accessed within a fixed content storage cluster by using a page mapping table and a pages index. A stream is read from the cluster by using a portion of its unique identifier as a key into the page mapping table. The page mapping table indicates a node holding a pages index indicating where the stream is stored. A stream is written by storing the stream on any suitable node and then updating a pages index stored within the cluster responsible for knowing the location of digital objects having unique identifiers that fall within a particular address range. The cluster recovers from a node failure by first replicating streams from the failed node and reallocating a page mapping table to create a new pages index. The remaining nodes send records of the unique identifiers corresponding to objects they hold to the new pages index.

Abstract: A cluster receives a request to store an object using replication or erasure coding. The cluster writes the object using erasure coding. A manifest is written that includes an indication of erasure coding and a unique identifier for each segment. The cluster returns a unique identifier of the manifest. The cluster receives a request from a client that includes a unique identifier. The cluster determines whether the object has been stored using replication or erasure coding. If using erasure coding, the method reads a manifest. The method identifies segments within the cluster using unique segment identifiers of the manifest. Using these unique segment identifiers, the method reconstructs the object. A persistent storage area of another disk is scanned to find a unique identifier of a failed disk. If using erasure coding, a missing segment previously stored on the disk is identified. The method locates other segments. Missing segments are regenerated.

Abstract: Digital objects are stored and accessed within a fixed content storage cluster by using a page mapping table and a pages index. A stream is read from the cluster by using a portion of its unique identifier as a key into the page mapping table. The page mapping table indicates a node holding a pages index indicating where the stream is stored. A stream is written to the cluster by storing the stream on any suitable node and then updating a pages index stored within the cluster. The cluster recovers from a node failure by first replicating streams from the failed node and reallocating a page mapping table to create a new pages index. The remaining nodes send records of the unique identifiers corresponding to objects they hold to the new pages index. A node is added to the cluster by reallocating a page mapping table.

Abstract: Each node and volume in a storage cluster makes a decision whether to reduce power consumption based on lack of requests from client applications and nodes over a time period. Node configuration parameters determine how long to wait until idling a node or volume, and how long to wait while idle before performing integrity checks. A bid value is calculated by each node and reflects how much it will cost for that node to write a file, read a file, or keep a copy. A node with the lowest bid wins, and nodes that are idle have a premium added to each bid to ensure that idle nodes are kept idle. In an archive mode, writes bids are reversed, nodes with less capacity submit lower bids, fuller nodes fill up faster and are then idled, while empty or near empty nodes may remain idle before winning a write bid.

Abstract: A cluster receives a request to store an object using replication or erasure coding. The cluster writes the object using erasure coding. A manifest is written that includes an indication of erasure coding and a unique identifier for each segment. The cluster returns a unique identifier of the manifest. The cluster receives a request from a client that includes a unique identifier. The cluster determines whether the object has been stored using replication or erasure coding. If using erasure coding, the method reads a manifest. The method identifies segments within the cluster using unique segment identifiers of the manifest. Using these unique segment identifiers, the method reconstructs the object. A persistent storage area of another disk is scanned to find a unique identifier of a failed disk. If using erasure coding, a missing segment previously stored on the disk is identified. The method locates other segments. Missing segments are regenerated.

Abstract: Digital objects are stored and accessed within a fixed content storage cluster by using a page mapping table and a pages index. A stream is read from the cluster by using a portion of its unique identifier as a key into the page mapping table. The page mapping table indicates a node holding a pages index indicating where the stream is stored. A stream is written by storing the stream on any suitable node and then updating a pages index stored within the cluster responsible for knowing the location of digital objects having unique identifiers that fall within a particular address range. The cluster recovers from a node failure by first replicating streams from the failed node and reallocating a page mapping table to create a new pages index. The remaining nodes send records of the unique identifiers corresponding to objects they hold to the new pages index.

Abstract: Digital objects are stored and accessed within a fixed content storage cluster by using a page mapping table and a pages index. A stream is read from the cluster by using a portion of its unique identifier as a key into the page mapping table. The page mapping table indicates a node holding a pages index indicating where the stream is stored. A stream is written to the cluster by storing the stream on any suitable node and then updating a pages index stored within the cluster. The cluster recovers from a node failure by first replicating streams from the failed node and reallocating a page mapping table to create a new pages index. The remaining nodes send records of the unique identifiers corresponding to objects they hold to the new pages index. A node is added to the cluster by reallocating a page mapping table.

Abstract: Each node and volume in a storage cluster makes a decision whether to reduce power consumption based on lack of requests from client applications and nodes over a time period. Node configuration parameters determine how long to wait until idling a node or volume, and how long to wait while idle before performing integrity checks. A bid value is calculated by each node and reflects how much it will cost for that node to write a file, read a file, or keep a copy. A node with the lowest bid wins, and nodes that are idle have a premium added to each bid to ensure that idle nodes are kept idle. In an archive mode, writes bids are reversed, nodes with less capacity submit lower bids, fuller nodes fill up faster and are then idled, while empty or near empty nodes may remain idle before winning a write bid.

Abstract: Digital objects within a fixed-content storage cluster use a page mapping table and a hash-to-UID table to store a representation of each object. For each object stored within the cluster, a record in the hash-to-UID table stores the object's hash value and its unique identifier (or portions thereof). To detect a duplicate of an object, a portion of its hash value is used as a key into the page mapping table. The page mapping table indicates a node holding a hash-to-UID table indicating currently stored objects in a particular page range. Finding the same hash value but with a different unique identifier in the table indicates that a duplicate of an object exists. Portions of the hash value and unique identifier may be used in the hash-to-UID table. Unneeded duplicate objects are deleted by copying their metadata to a manifest and then redirecting unique identifiers to point at the manifest.

Abstract: Two or more numerical samples are read from a storage location (or many storage locations) and the samples are compared to generate a single bit of entropy. This method is repeated to populate two arrays which are also compared to generate a single bit. Comparison of the arrays is also repeated to generate a sequence of bits which are formed into computer words. The words are combined with a shift register to generate a seed block array of any length. The seed block array is used to seed a pseudo random number generator. The storage location is a register, accumulator, buffer, clock, address, memory location, etc., that changes periodically. Comparison may be performed by counting the number of certain types of bits in the two values and returning a one or zero depending upon the count.

Abstract: Digital objects within a fixed-content storage cluster use a page mapping table and a hash-to-UID table to store a representation of each object. For each object stored within the cluster, a record in the hash-to-UID table stores the object's hash value and its unique identifier (or portions thereof). To detect a duplicate of an object, a portion of its hash value is used as a key into the page mapping table. The page mapping table indicates a node holding a hash-to-UID table indicating currently stored objects in a particular page range. Finding the same hash value but with a different unique identifier in the table indicates that a duplicate of an object exists. Portions of the hash value and unique identifier may be used in the hash-to-UID table. Unneeded duplicate objects are deleted by copying their metadata to a manifest and then redirecting unique identifiers to point at the manifest.

Abstract: A cluster receives a request to store an object using replication or erasure coding. The cluster writes the object using erasure coding. A manifest is written that includes an indication of erasure coding and a unique identifier for each segment. The cluster returns a unique identifier of the manifest. The cluster receives a request from a client that includes a unique identifier. The cluster determines whether the object has been stored using replication or erasure coding. If using erasure coding, the method reads a manifest. The method identifies segments within the cluster using unique segment identifiers of the manifest. Using these unique segment identifiers, the method reconstructs the object. A persistent storage area of another disk is scanned to find a unique identifier of a failed disk. If using erasure coding, a missing segment previously stored on the disk is identified. The method locates other segments. Missing segments are regenerated.

Abstract: Digital objects within a fixed-content storage cluster use a page mapping table and a hash-to-UID table to store a representation of each object. For each object stored within the cluster, a record in the hash-to-UID table stores the object's hash value and its unique identifier (or portions thereof). To detect a duplicate of an object, a portion of its hash value is used as a key into the page mapping table. The page mapping table indicates a node holding a hash-to-UID table indicating currently stored objects in a particular page range. Finding the same hash value but with a different unique identifier in the table indicates that a duplicate of an object exists. Portions of the hash value and unique identifier may be used in the hash-to-UID table. Unneeded duplicate objects are deleted by copying their metadata to a manifest and then redirecting their unique identifiers to point at the manifest.

Abstract: Each node and volume in a fixed-content storage cluster makes an independent decision whether to reduce power consumption based on lack of requests from client applications and nodes over a configurable time period. Node configuration parameters sleepAfter and wakeAfter respectively determine how long to wait until idling a node or volume, and how long to wait while idle before again performing integrity checks. A bid value is calculated by each node and reflects how much it will cost for that node to write a file, read a file, or keep a copy of the file. A node with the lowest bid wins, and nodes that are idle have a premium added to each bid to help ensure that idle nodes are kept idle. Normally, nodes with more capacity will submit a lower bid to write a file.

Abstract: Each node and volume in a storage cluster makes a decision whether to reduce power consumption based on lack of requests from client applications and nodes over a time period. Node configuration parameters determine how long to wait until idling a node or volume, and how long to wait while idle before performing integrity checks. A bid value is calculated by each node and reflects how much it will cost for that node to write a file, read a file, or keep a copy. A node with the lowest bid wins, and nodes that are idle have a premium added to each bid to ensure that idle nodes are kept idle. In an archive mode, writes bids are reversed, nodes with less capacity submit lower bids, fuller nodes fill up faster and are then idled, while empty or near empty nodes may remain idle before winning a write bid.