Abstract: File availability in distributed file storage systems is improved by initially placing replicas of a file or other object on different ones of multiple devices using a first process. Subsequently, the placement of the replicas is improved by evaluating whether any replicas of a first file can be swapped with any replicas of a second file without a reduction in the combined file availability of the first and second files, and swapping a replica of the first file with a replica of the second file if the swapping results in no reduction in the combined file availability of the first and second files.

Abstract: A location history is a collection of locations over time for an object. A stay is a single instance of an object spending some time in one place, and a destination is any place where one or more objects have experienced a stay. Location histories are parsed using stays and destinations. In a described implementation, each location of a location history is recorded as a spatial position and a corresponding time at which the spatial position is acquired. Stays are extracted from a location history by analyzing locations thereof with regard to a temporal threshold and a spatial threshold. Specifically, two or more locations are considered a stay if they exceed a minimum stay duration and are within a maximum roaming distance. Each stay includes a location, a starting time, and an ending time. Destinations are produced from the extracted stays using a clustering operation and a predetermined scaling factor.

Abstract: A file that has been encrypted using a symmetric key and that has a corresponding access control entry with the symmetric key encrypted using the public key of a public/private key pair can be accessed. An encrypted key cache is also accessed to determine whether an access control entry to symmetric key mapping exists in the cache for the access control entry corresponding to the file. If such a mapping exists in the cache, then the mapped-to symmetric key is obtained form the cache, otherwise the encrypted symmetric key is decrypted using the private key of the public/private key pair. The encrypted key cache itself can also be encrypted and stored as an encrypted file.

Abstract: A distributed computing system can be operated in a fault tolerant manner using a set of computing devices. A set of computing devices can tolerate a number of failures by implementing identical replicas of a state machine and selecting proposals. The set of computing devices participating in the distributed computing system by hosting replicas can be modified by adding or removing a computing device from the set, or by specifying particular computing devices for participation. Changing the participating computing devices in the set increases fault tolerance by replacing defective devices with operational devices, or by increasing the amount of redundancy in the system.

Abstract: A file that has been encrypted using a symmetric key and that has a corresponding access control entry with the symmetric key encrypted using the public key of a public/private key pair can be accessed. An encrypted key cache is also accessed to determine whether an access control entry to symmetric key mapping exists in the cache for the access control entry corresponding to the file. If such a mapping exists in the cache, then the mapped-to symmetric key is obtained form the cache, otherwise the encrypted symmetric key is decrypted using the private key of the public/private key pair. The encrypted key cache itself can also be encrypted and stored as an encrypted file.

Abstract: Cryptographic protocols and methods of employing the same are described. The described protocols advantageously enable two or more identical encryptable objects that are coded for encryption with different keys to be identified as identical without access to either the unencrypted objects or the keys that are used in the encryption process. Additionally, the protocols enable two or more identical encryptable objects to be processed with different encryption keys, yet be stored in a manner so that the total required storage space is proportional to the space that is required to store a single encryptable object, plus a constant amount for each distinct encryption key. In various embodiments, the encryptable objects comprise files and the cryptographic protocols enable encrypted files to be used in connection with single instance store (SIS) systems.

Abstract: Cryptographic protocols and methods of employing the same are described. The described protocols advantageously enable two or more identical encryptable objects that are coded for encryption with different keys to be identified as identical without access to either the unencrypted objects or the keys that are used in the encryption process. Additionally, the protocols enable two or more identical encryptable objects to be processed with different encryption keys, yet be stored in a manner so that the total required storage space is proportional to the space that is required to store a single encryptable object, plus a constant amount for each distinct encryption key. In various embodiments, the encryptable objects comprise files and the cryptographic protocols enable encrypted files to be used in connection with single instance store (SIS) systems.

Abstract: Potentially identical objects (e.g., files) are located across multiple computers based on stochastic partitioning of workload. For each of a plurality of objects stored on a plurality of computers in a network, a portion of object information corresponding to the object is selected. The object information can be generated in a variety of manners (e.g., based on hashing the object, based on characteristics of the object, and so forth). Any of a variety of portions of the object information can be used (e.g., the least significant bits of the object information). A stochastic partitioning process is then used to identify which of the plurality of computers to communicate the object information to for identification of potentially identical objects on the plurality of computers.

Abstract: Potentially identical objects (e.g., files) are located across multiple computers based on stochastic partitioning of workload. For each of a plurality of objects stored on a plurality of computers in a network, a portion of object information corresponding to the object is selected. The object information can be generated in a variety of manners (e.g., based on hashing the object, based on characteristics of the object, and so forth). Any of a variety of portions of the object information can be used (e.g., the least significant bits of the object information). A stochastic partitioning process is then used to identify which of the plurality of computers to communicate the object information to for identification of potentially identical objects on the plurality of computers.

Abstract: A method and system for regulating tasks of background processes so as to reduce interference with foreground processes. The progress rate of a background task (e.g., amount of work performed per unit time) is measured and evaluated against a target amount. If the progress rate appears degraded, the background task is suspended for a computed time interval so as to back off from its interference with a foreground process. Each time the progress rate appears degraded, the time interval is exponentially increased from its previous value up to a maximum, however if the performance appears normal, the time interval is reset to a minimum. Evaluation of the work is statistically based so as to eliminate variations in measurements, and automatic calibration of the target amount is provided, as is a mechanism for prioritizing multiple background tasks.

Abstract: A method and system for regulating tasks of background processes so as to reduce interference with foreground processes. The progress rate of a background task (e.g., amount of work performed per unit time) is measured and evaluated against a target amount. If the progress rate appears degraded, the background task is suspended for a computed time interval so as to back off from its interference with a foreground process. Each time the progress rate appears degraded, the time interval is exponentially increased from its previous value up to a maximum, however if the performance appears normal, the time interval is reset to a minimum. Evaluation of the work is statistically based so as to eliminate variations in measurements, and automatic calibration of the target amount is provided, as is a mechanism for prioritizing multiple background tasks.

Abstract: A file format for a serverless distributed file system is composed of two parts: a primary data stream and a metadata stream. The data stream contains a file that is divided into multiple blocks. Each block is encrypted using a hash of the block as the encryption key. The metadata stream contains a header, a structure for indexing the encrypted blocks in the primary data stream, and some user information. The indexing structure defines leaf nodes for each of the blocks. Each leaf node consists of an access value used for decryption of the associated block and a verification value used to verify the encrypted block independently of other blocks. In one implementation, the access value is formed by hashing the file block and encrypting the resultant hash value using a randomly generated key. The key is then encrypted using the user's key as the encryption key. The verification value is formed by hashing the associated encrypted block using a one-way hash function.

Abstract: A serverless distributed file system manages the storage of files and directories using one or more directory groups. The directories may be managed using Byzantine-fault-tolerant groups, whereas files are managed without using Byzantine-fault-tolerant groups. Additionally, the file system may employ a hierarchical namespace to store files. Furthermore, the directory group may employ a plurality of locks to control access to objects (e.g., files and directories) in each directory.

Abstract: A file format for a serverless distributed file system is composed of two parts: a primary data stream and a metadata stream. The data stream contains a file that is divided into multiple blocks. Each block is encrypted using a hash of the block as the encryption key. The metadata stream contains a header, a structure for indexing the encrypted blocks in the primary data stream, and some user information. The indexing structure defines leaf nodes for each of the blocks. Each leaf node consists of an access value used for decryption of the associated block and a verification value used to verify the encrypted block independently of other blocks. In one implementation, the access value is formed by hashing the file block and encrypting the resultant hash value using a randomly generated key. The key is then encrypted using the user's key as the encryption key. The verification value is formed by hashing the associated encrypted block using a one-way hash function.

Abstract: A file format for a serverless distributed file system is composed of two parts: a primary data stream and a metadata stream. The data stream contains a file that is divided into multiple blocks. Each block is encrypted using a hash of the block as the encryption key. The metadata stream contains a header, a structure for indexing the encrypted blocks in the primary data stream, and some user information. The indexing structure defines leaf nodes for each of the blocks. Each leaf node consists of an access value used for decryption of the associated block and a verification value used to verify the encrypted block independently of other blocks. In one implementation, the access value is formed by hashing the file block and encrypting the resultant hash value using a randomly generated key. The key is then encrypted using the user's key as the encryption key. The verification value is formed by hashing the associated encrypted block using a one-way hash function.

Abstract: A file format for a serverless distributed file system is composed of two parts: a primary data stream and a metadata stream. The data stream contains a file that is divided into multiple blocks. Each block is encrypted using a hash of the block as the encryption key. The metadata stream contains a header, a structure for indexing the encrypted blocks in the primary data stream, and some user information. The indexing structure defines leaf nodes for each of the blocks. Each leaf node consists of an access value used for decryption of the associated block and a verification value used to verify the encrypted block independently of other blocks. In one implementation, the access value is formed by hashing the file block and encrypting the resultant hash value using a randomly generated key. The key is then encrypted using the user's key as the encryption key. The verification value is formed by hashing the associated encrypted block using a one-way hash function.

Abstract: Potentially identical objects (e.g., files) are located across multiple computers based on stochastic partitioning of workload. For each of a plurality of objects stored on a plurality of computers in a network, a portion of object information corresponding to the object is selected. The object information can be generated in a variety of manners (e.g., based on hashing the object, based on characteristics of the object, and so forth). Any of a variety of portions of the object information can be used (e.g., the least significant bits of the object information). A stochastic partitioning process is then used to identify which of the plurality of computers to communicate the object information to for identification of potentially identical objects on the plurality of computers.

Abstract: The clocks of remote computing devices are synchronized within a range of certainty through the determination of an upper bound and a lower bound around a reference time. A message from a computing device is propagated up a network tree of devices to a device having a reference time, which encodes the reference time and returns the message down the tree. Each receiving device can determine that the reference time could not have occurred before their transmission of the message, nor could it have occurred after their receipt of the return message. Cryptographic hashes can be used to guard against malicious computing devices. Alternate paths and scheduling of messages can be used to provide a narrower spread between the upper and lower bounds, and clock drift can be accounted for by increasing the spread over time.

Abstract: A handle administration system is described in which software agents receive handles to various resources that they can use to obtain the resources. The described embodiments provide multiple states that can be assumed by the handles. An unassigned state is provided in which handles are not assigned to a particular resource, nor can they be dereferenced to obtain pointers to any resources. An assigned state is provided in which handles are assigned to a particular resource and can be dereferenced to obtain a pointer to the resource. A suspended state is provided in which the handles are assigned to a particular resource but cannot be dereferenced to obtain a pointer to that resource. Advantageously, a suspended handle can be reinstated to assume the assigned state. In one embodiment, the handle system is implemented by incorporating a suitable field in a handle database that is used to indicate that a handle is suspended. In another embodiment, no additional fields are necessary.

Abstract: Potentially identical objects (e.g., files) are located across multiple computers based on stochastic partitioning of workload. For each of a plurality of objects stored on a plurality of computers in a network, a portion of object information corresponding to the object is selected. The object information can be generated in a variety of manners (e.g., based on hashing the object, based on characteristics of the object, and so forth). Any of a variety of portions of the object information can be used (e.g., the least significant bits of the object information). A stochastic partitioning process is then used to identify which of the plurality of computers to communicate the object information to for identification of potentially identical objects on the plurality of computers.