Abstract: A wire protocol provides message formats for creating multiple network connections between a media server and a client. These multiple network connections may include a control link connection for passing control information and a data funnel connection for passing data of multiple media. The data funnel connection may be a multipoint-to-point connection that connects multiple data servers with the client. The protocol facilitates multiple requests being concurrently outstanding and asynchronous processing of requests. The protocol is designed to exist on top of a transport protocol layer.

Abstract: Efficient processing of time-bound messages is described herein. In an implementation, messages are classified either time-bounded messages or non-time-bounded messages. The time-bounded messages are then processed separately from the non-time-bound messages. Examples of time-bounded messages can include spam e-mails or other types or classes of e-mails subject to a retention policy, such that the are retained only for some set period of time. Metadata relating to the time-bound messages are compiled and stored separately from the messages, thereby enabling optimization of the metadata storage and processing apart from the messages. The time-bounded messages are accumulated before they are processed in bulk, thereby reducing the number if I/O operations consumed by the time-bound messages, and reducing the amount and cost of resources supporting the process of the time-bounded messages.

Abstract: A logical file system is described that distributes copies of files across various different physical storage resources yet provides a consistent view to the user of his or her data, regardless of which machine the user is accessing the files from, and even when the user's computer is offline. The distributed file system uses smart data redundancy to enable a virtually infinite amount of storage as long as additional storage resources are made available to the distributed file system. The result is a reliable storage system that does not necessarily tie the user's data to the user's particular computer. Instead, the user's data is associated with the user—for life—or for however long the user would like the data to be maintained, regardless of whether the user's computer or data storage components are replaced or destroyed.

Abstract: The safety of data stored across distributed storage resources may be dynamically monitored and indicated to the user. Monitoring may occur in real time and/or in response to changes in the stored data. The notification may be in the form of, e.g., a visual indicator that appears in the taskbar of a graphical user interface of an operating system. The indicator may be presented in a particular color or having some other visual property associated with the particular data safety state. In addition, the user may be provided with a convenient way of obtaining additional storage resources. Upon automatically detecting that storage is running low, the user may be presented with a user interface that offers the user a way to easily add to the existing storage. For example, the user may be directed to a predefined Internet web site that is known to offer such additional storage.

Abstract: A file that has been encrypted using a symmetric key and that has a corresponding access control entry with the symmetric key encrypted using the public key of a public/private key pair can be accessed. An encrypted key cache is also accessed to determine whether an access control entry to symmetric key mapping exists in the cache for the access control entry corresponding to the file. If such a mapping exists in the cache, then the mapped-to symmetric key is obtained form the cache, otherwise the encrypted symmetric key is decrypted using the private key of the public/private key pair. The encrypted key cache itself can also be encrypted and stored as an encrypted file.

Abstract: An architecture and implementation for losslessly restarting subsystems in a distributed file system is described. By partitioning functionality and logging appropriately across the kernel and user-level boundaries on a client, the user-level subsystem may be made losslessly restartable. A particular use of transactions achieves efficiency while retaining simplicity. Practical mechanisms for supporting state-based recovery in replicated state machines and like replica are described. In particular, a map assisted state transfer may include receiving one or more state updates, marshaling one or more active data-structures into a marshaled shadow, applying the received state updates to the marshaled shadow and re-instantiating the active data-structures by unmarshaling the marshaled shadow.

Abstract: An architecture and implementation for losslessly restarting subsystems in a distributed file system is described. By partitioning functionality and logging appropriately across the kernel and user-level boundaries on a client, the user-level subsystem may be made losslessly restartable. Practical mechanisms for supporting state-based recovery in replicated state machines and like replica are described. In particular, each client daemon may include an operations log and an applied log sequence number. Each client driver may include a potentially different operations log. Each client daemon may be configured to request logged operations associated with log sequence numbers in one or more ranges specified by a specification that includes the applied log sequence number. The requested logged operations may reside in the operations log maintained by a client driver.

Abstract: A file that has been encrypted using a symmetric key and that has a corresponding access control entry with the symmetric key encrypted using the public key of a public/private key pair can be accessed. An encrypted key cache is also accessed to determine whether an access control entry to symmetric key mapping exists in the cache for the access control entry corresponding to the file. If such a mapping exists in the cache, then the mapped-to symmetric key is obtained form the cache, otherwise the encrypted symmetric key is decrypted using the private key of the public/private key pair. The encrypted key cache itself can also be encrypted and stored as an encrypted file.

Abstract: Cryptographic protocols and methods of employing the same are described. The described protocols advantageously enable two or more identical encryptable objects that are coded for encryption with different keys to be identified as identical without access to either the unencrypted objects or the keys that are used in the encryption process. Additionally, the protocols enable two or more identical encryptable objects to be processed with different encryption keys, yet be stored in a manner so that the total required storage space is proportional to the space that is required to store a single encryptable object, plus a constant amount for each distinct encryption key. In various embodiments, the encryptable objects comprise files and the cryptographic protocols enable encrypted files to be used in connection with single instance store (SIS) systems.

Abstract: Cryptographic protocols and methods of employing the same are described. The described protocols advantageously enable two or more identical encryptable objects that are coded for encryption with different keys to be identified as identical without access to either the unencrypted objects or the keys that are used in the encryption process. Additionally, the protocols enable two or more identical encryptable objects to be processed with different encryption keys, yet be stored in a manner so that the total required storage space is proportional to the space that is required to store a single encryptable object, plus a constant amount for each distinct encryption key. In various embodiments, the encryptable objects comprise files and the cryptographic protocols enable encrypted files to be used in connection with single instance store (SIS) systems.

Abstract: Potentially identical objects (e.g., files) are located across multiple computers based on stochastic partitioning of workload. For each of a plurality of objects stored on a plurality of computers in a network, a portion of object information corresponding to the object is selected. The object information can be generated in a variety of manners (e.g., based on hashing the object, based on characteristics of the object, and so forth). Any of a variety of portions of the object information can be used (e.g., the least significant bits of the object information). A stochastic partitioning process is then used to identify which of the plurality of computers to communicate the object information to for identification of potentially identical objects on the plurality of computers.

Abstract: Potentially identical objects (e.g., files) are located across multiple computers based on stochastic partitioning of workload. For each of a plurality of objects stored on a plurality of computers in a network, a portion of object information corresponding to the object is selected. The object information can be generated in a variety of manners (e.g., based on hashing the object, based on characteristics of the object, and so forth). Any of a variety of portions of the object information can be used (e.g., the least significant bits of the object information). A stochastic partitioning process is then used to identify which of the plurality of computers to communicate the object information to for identification of potentially identical objects on the plurality of computers.

Abstract: A wire protocol provides message formats for creating multiple network connections between a media server and a client. These multiple network connections may include a control link connection for passing control information and a data funnel connection for passing data of multiple media. The data funnel connection may be a multipoint-to-point connection that connects multiple data servers with the client. The protocol facilitates multiple requests being concurrently outstanding and asynchronous processing of requests. The protocol is designed to exist on top of a transport protocol layer.

Abstract: A method and system for regulating tasks of background processes so as to reduce interference with foreground processes. The progress rate of a background task (e.g., amount of work performed per unit time) is measured and evaluated against a target amount. If the progress rate appears degraded, the background task is suspended for a computed time interval so as to back off from its interference with a foreground process. Each time the progress rate appears degraded, the time interval is exponentially increased from its previous value up to a maximum, however if the performance appears normal, the time interval is reset to a minimum. Evaluation of the work is statistically based so as to eliminate variations in measurements, and automatic calibration of the target amount is provided, as is a mechanism for prioritizing multiple background tasks.

Abstract: A wire protocol provides message formats for creating multiple network connections between a media server and a client. These multiple network connections may include a control link connection for passing control information and a data funnel connection for passing data of multiple media The data funnel connection may be a multipoint-to-point connection that connects multiple data servers with the client. The protocol facilitates multiple requests being concurrently outstanding and asynchronous processing of requests. The protocol is designed to exist on top of a transport protocol layer.

Abstract: A file format for a serverless distributed file system is composed of two parts: a primary data stream and a metadata stream. The data stream contains a file that is divided into multiple blocks. Each block is encrypted using a hash of the block as the encryption key. The metadata stream contains a header, a structure for indexing the encrypted blocks in the primary data stream, and some user information. The indexing structure defines leaf nodes for each of the blocks. Each leaf node consists of an access value used for decryption of the associated block and a verification value used to verify the encrypted block independently of other blocks. In one implementation, the access value is formed by hashing the file block and encrypting the resultant hash value using a randomly generated key. The key is then encrypted using the user's key as the encryption key. The verification value is formed by hashing the associated encrypted block using a one-way hash function.

Abstract: A method and system for regulating tasks of background processes so as to reduce interference with foreground processes. The progress rate of a background task (e.g., amount of work performed per unit time) is measured and evaluated against a target amount. If the progress rate appears degraded, the background task is suspended for a computed time interval so as to back off from its interference with a foreground process. Each time the progress rate appears degraded, the time interval is exponentially increased from its previous value up to a maximum, however if the performance appears normal, the time interval is reset to a minimum. Evaluation of the work is statistically based so as to eliminate variations in measurements, and automatic calibration of the target amount is provided, as is a mechanism for prioritizing multiple background tasks.

Abstract: A serverless distributed file system manages the storage of files and directories using one or more directory groups. The directories may be managed using Byzantine-fault-tolerant groups, whereas files are managed without using Byzantine-fault-tolerant groups. Additionally, the file system may employ a hierarchical namespace to store files. Furthermore, the directory group may employ a plurality of locks to control access to objects (e.g., files and directories) in each directory.

Abstract: A wire protocol provides message formats for creating multiple network connections between a media server and a client. These multiple network connections may include a control link connection for passing control information and a data funnel connection for passing data of multiple media. The data funnel connection may be a multipoint-to-point connection that connects multiple data servers with the client. The protocol facilitates multiple requests being concurrently outstanding and asynchronous processing of requests. The protocol is designed to exist on top of a transport protocol layer.

Abstract: Potentially identical objects (e.g., files) are located across multiple computers based on stochastic partitioning of workload. For each of a plurality of objects stored on a plurality of computers in a network, a portion of object information corresponding to the object is selected. The object information can be generated in a variety of manners (e.g., based on hashing the object, based on characteristics of the object, and so forth). Any of a variety of portions of the object information can be used (e.g., the least significant bits of the object information). A stochastic partitioning process is then used to identify which of the plurality of computers to communicate the object information to for identification of potentially identical objects on the plurality of computers.