Abstract: A massively scalable architecture for providing a highly reliable storage system that is capable of handling hundreds of millions of users and tens of billions of files. The system includes one or more clusters storing data elements that are received from a plurality of clients. Each cluster comprises a plurality of storage servers. Each storage server maintains one or more indexes for responding to simple read requests such as retrieving a designated element. According to the invention, the storage system includes one or more database clusters each database cluster having one or more database servers. The storage system accesses the storage clusters in order to reply to simple read requests; however, the database clusters are used to resolve more complex queries that are based on attributes of elements stored within the storage system. In one configuration, the storage system includes a database cluster for each class of applications accessing the storage system.

Abstract: A massively scalable architecture for providing a highly reliable storage system that is capable of handling hundreds of millions of users and tens of billions of files. The system includes one or more clusters storing data elements that are received from a plurality of clients. Each cluster comprises a plurality of storage servers. A partition master assigns each client a unique partition within one of the clusters such that each data element is stored in the partition assigned to the client. The storage system can be easily scaled as the number of reads increases by adding individual storage servers to each storage cluster. In addition, the storage system easily scales to handle an increase in the number of writes, or as the number of files per cluster exceeds a predefined limit, by adding new storage clusters to the storage system.

Abstract: A massively scalable architecture for providing a self-monitoring and self-correcting storage system that is capable of handling hundreds of millions of users and tens of billions of files. The system includes one or more clusters storing data elements that are received from a plurality of clients. Each cluster comprises a plurality of storage servers. The storage system facilitates the addition of new storage servers, and the fast recovery of failed storage servers, by logging system transactions in multiple journals of different lengths. When a storage server fails, a cluster backup determines the time of failure and replays one of the journals in order to bring the failed storage server up to date.

Abstract: A massively scalable architecture has been developed for providing an extensible storage system that is capable of handling hundreds of millions of users and tens of billions of files. The storage system includes to a plurality of storage clusters, each storage cluster having one or more storage servers. Each storage element has a corresponding schema object that is used to parse the data elements into data and attributes. Applications executing on the web servers are able to dynamically define a new type of element for storage within storage system. In one configuration the schemas are defined in Extensible Markup Language (XML).