Abstract: The present disclosure provides techniques for deallocating previously allocated storage blocks. The techniques include obtaining a list of chunk IDs to analyze, choosing a chunk ID, and determining the storage blocks spanned by the chunk corresponding to the chosen chunk ID. The technique further includes determining whether any file references any storage blocks spanned by the chunk. The determining may be performed by comparing an internal reference count to a total reference count, where the internal reference count is the number of reference to the storage block by a chunk ID data structure. If no files reference any of the storage blocks spanned by the chunk, then all the storage blocks of the chunk can be deallocated.

Abstract: The present disclosure provides techniques for deduplicating files. The techniques include creating a cache or subset of a large data structure. The large data structure organizes information by random hash values. The random hash values result in a random organization of information within the data structure, with the information spanning a large number of storage blocks within a storage system. The cache, however, is within memory and is small relative to the data structure. The cache is created so as to contain information that is likely to be needed during deduplication of a file. Having needed information within memory rather than in storage results in faster read and write operations to that information, improving the performance of a computing system.

Abstract: The disclosure provides techniques for deduplicating files. The techniques include, upon creating or modifying a file, placing a logical timestamp of the current logical time, within a queue associated with the directory of the file. The techniques further include placing the logical timestamp within a queue of each parent directory of the directory of the file. To determine a set of files for deduplication, the techniques disclosed herein identify files that have been modified within a logical time range. The set of files modified within a logical time is identified by traversing directories of a storage system, the directories being organized within a tree structure. If a directory's queue does not contain a timestamp that is within the logical time range, then all child directories can be skipped over for further processing, such that no files within the child directories end up being within the set of files for deduplication.

Abstract: Disclosed techniques include deduplication. Techniques include determining whether a file is unique, and depending on whether the file is unique, deduplicating only part of the file or the entire file. The techniques include processing the first chunk of a file to determine whether the hash of the chunk hash is already within a chunk hash table, and if not, then a percentage of chunks of the file is similarly processed. If any of the hashes of chunks are already in the chunk hash table, then at least some of file has been previously deduplicated, and file is not unique the storage system. If none of the processed chunks have a hash that is already in the chunk hash table, then the file is considered to be unique within chunk store and only a partial percentage of the file's chunks are deduplicated. Not all of a unique file's chunks are deduplicated.

Abstract: The present disclosure provides techniques for deduplicating files. The techniques include creating a data structure that organizes metadata about chunks of files, the organization of the metadata preserving order and locality of the chunks within files. The organization of the metadata within storage blocks of storage devices matches the order of chunks within files. Upon a read or write operation to a metadata, the preservation of locality of metadata results in the likely fetching, from storage into a memory cache, metadata of subsequent and contiguous chunks. The preserved locality results in faster subsequent read and write operations of metadata, because the read and write operations are likely to be executed from memory rather than from storage.

Abstract: The present disclosure provides techniques for scaling out deduplication of files among a plurality of nodes. The techniques include designating a master component for the coordination of deduplication. The master component divides files to be deduplicated among several slave nodes, and provides to each slave node a set of unique identifiers that are to be assigned to chunks during the deduplication process. The techniques herein preserve integrity of the deduplication process that has been scaled out among several nodes. The scaled out deduplication process deduplicates files faster by allowing several deduplication modules to work in parallel to deduplicate files.

Abstract: Distributed storage systems support SQL while also providing scalability and fault tolerance by utilizing an SQL library (the SQL execution component) layered on top of a transactional key-value system (the storage component). The SQL library comprises a parser and an execution engine running at the client, while the storage component comprises a key-value system for storing the data. The SQL library parses SQL queries received from client applications and maps them to transactions on the key-value system. The key-value system stores data items identified by key and provides concurrency control mechanisms to handle concurrent transactions. The key-value system may be a distributed system with several storage servers, each attached to one or more storage devices.

Abstract: A remote procedure call chain is provided that replaces multiple consecutive remote procedure calls to multiple servers from a client by allowing a client to specify multiple functions to be performed consecutively at multiple servers in a single remote procedure call chain. The remote procedure call chain is executed by a sequence of multiple servers. Each server executes a service function and a chaining function of the remote procedure call chain. The chaining function uses the state of the remote procedure call chain in the sequence of servers to determine the next server to receive the remote procedure call chain, and the service function to be executed by that server. After the last service function is performed, the last server in the sequence of servers sends the results of the executed service functions to the client that originated the remote procedure call chain.

Abstract: Distributed storage systems support SQL while also providing scalability and fault tolerance by utilizing an SQL library (the SQL execution component) layered on top of a transactional key-value system (the storage component). The SQL library comprises a parser and an execution engine running at the client, while the storage component comprises a key-value system for storing the data. The SQL library parses SQL queries received from client applications and maps them to transactions on the key-value system. The key-value system stores data items identified by key and provides concurrency control mechanisms to handle concurrent transactions. The key-value system may be a distributed system with several storage servers, each attached to one or more storage devices.

Abstract: Distributed storage systems support SQL while also providing scalability and fault tolerance by utilizing an SQL library (the SQL execution component) layered on top of a transactional key-value system (the storage component). The SQL library comprises a parser and an execution engine running at the client, while the storage component comprises a key-value system for storing the data. The SQL library parses SQL queries received from client applications and maps them to transactions on the key-value system. The key-value system stores data items identified by key and provides concurrency control mechanisms to handle concurrent transactions. The key-value system may be a distributed system with several storage servers, each attached to one or more storage devices.

Abstract: According to one embodiment, a method comprises evaluating messages between nodes of a distributed computing environment. Based on timing relationships of the messages, probability of causal links between pairs of messages determined. Based on the determined probability, at least one causal path comprising a plurality of causal links is inferred. Further, an overall probability that the inferred causal path is accurate is determined. In certain embodiments, the overall probability is output for the corresponding causal path to provide an indication of the confidence of the accuracy of such inferred causal path.

Abstract: A cloud statistics server generates statistics for a cloud service based on an identified data item and an identified operation. The cloud service may include various computing nodes and storage nodes. The cloud statistics may include expected completion times for the identified operation and the identified data item with respect to each of the computing nodes. A computing node may be selected to execute the identified operation based on the expected completion times. The generated statistics may be generated by the cloud statistics server using a network topology associated with the data item that is based on the latencies or expected transfer times between the various storage nodes and computing nodes, and a replication strategy used by the cloud service. The topology may be implemented as a directed graph with edge weights corresponding to expected transfer times between each node.

Abstract: Embodiments include methods, apparatus, and systems for modifying data structures in distributed file systems. One method of software execution includes using minitransactions that include a set of compare items and a set of write items to atomically modify data structures in a distributed file system.

Abstract: An application programming interface is provided that allows applications to assign multiple service-level agreements to their data transactions. The service-level agreements include latency bounds and consistency guarantees. The applications may assign utility values to each of the service-level agreements. A monitor component monitors the various replica nodes in a cloud storage system for latency and consistency, and when a transaction is received from an application, the monitor determines which of the replica nodes can likely fulfill the transaction in satisfaction of any of the service-level agreements. Where multiple service-level agreements can be satisfied, the replica node that can fulfill the transaction according to the service-level agreement with the greatest utility is selected. The application may be charged for the transaction based on the utility of the service-level agreement that was satisfied.

Abstract: Embodiments include methods, apparatus, and systems for snapshots in distributed storage systems. One method of software execution includes using a version tree to determine what data blocks are shared between various storage nodes in the version tree in order to create a clone or a snapshot of a storage volume in a distributed storage system that uses quorum-based replication.

Abstract: To access data, a distributed balanced tree having nodes distributed across plural servers is accessed. Version information is associated with the nodes of the distributed balanced tree. During an operation that accesses the distributed balanced tree, the version information is checked to determine whether content of one or more nodes of the distributed balanced tree has changed during the operation.

Abstract: Distributed storage systems support SQL while also providing scalability and fault tolerance by utilizing an SQL library (the SQL execution component) layered on top of a transactional key-value system (the storage component). The SQL library comprises a parser and an execution engine running at the client, while the storage component comprises a key-value system for storing the data. The SQL library parses SQL queries received from client applications and maps them to transactions on the key-value system. The key-value system stores data items identified by key and provides concurrency control mechanisms to handle concurrent transactions. The key-value system may be a distributed system with several storage servers, each attached to one or more storage devices.

Abstract: An application programming interface is provided that allows applications to assign multiple service-level agreements to their data transactions. The service-level agreements include latency bounds and consistency guarantees. The applications may assign utility values to each of the service-level agreements. A monitor component monitors the various replica nodes in a cloud storage system for latency and consistency, and when a transaction is received from an application, the monitor determines which of the replica nodes can likely fulfill the transaction in satisfaction of any of the service-level agreements. Where multiple service-level agreements can be satisfied, the replica node that can fulfill the transaction according to the service-level agreement with the greatest utility is selected. The application may be charged for the transaction based on the utility of the service-level agreement that was satisfied.

Abstract: In accordance with an embodiment of the invention, a method of writing and reading redundant data is provided. Data is written by storing a copy of the data along with a timestamp and a signature at each of a set of storage devices. The data is read by retrieving the copy of the data, the timestamp and the signature from each of a plurality of the set of data storage devices. One of the copies of the data is selected to be provided to a requestor of the data. Each of the storage devices of the set is requested to certify the selected copy of the data. Provided that a proof of certification of the selected copy of the data is valid, the storage devices of the set are instructed to store the selected copy of the data along with a new timestamp.

Abstract: According to at least one embodiment, a method comprises identifying at least one causal path that includes a node of a distributed computing environment that is of interest. The method further comprises analyzing the identified at least one causal path to determine at least one time interval when the node is active in such causal path, and correlating consumption of a resource by the node to the node's activity in the at least one causal path.