Abstract: A method of processing transactions associated with a command in a storage system is provided. The method includes receiving, at a first authority of the storage system, a command relating to user data. The method includes sending a transaction of the command, from the first authority to a second authority of the storage system, wherein a token accompanies the transaction and writing data in accordance with the transaction as permitted by the token into a partition that is allocated to the second authority in a storage device of the storage system.

Abstract: A method of processing transactions associated with a command in a storage system is provided. The method includes receiving, at a first authority of the storage system, a command relating to user data. The method includes sending a transaction of the command, from the first authority to a second authority of the storage system, wherein a token accompanies the transaction and writing data in accordance with the transaction as permitted by the token into a partition that is allocated to the second authority in a storage device of the storage system.

Abstract: Systems and methods of deduplication aware scalable content placement are described. A method may include receiving data to be stored on one or more nodes of a storage array and calculating a plurality of hashes corresponding to the data. The method further includes determining a first subset of the plurality of hashes, determining a second subset of the plurality of hashes of the first subset, and generating a node candidate placement list. The method may further include sending the first subset to one or more nodes represented on the node candidate placement list and receiving, from the nodes represented on the node candidate placement list, characteristics corresponding to the nodes represented on the candidate placement list. The method may further include identifying one of the one or more nodes represented on the candidate placement list in view of the characteristic and sending the data to the identified node.

Abstract: A storage cluster is provided. The storage cluster includes a plurality of storage nodes within a single chassis. Each of the plurality of storage nodes has nonvolatile solid-state memory for storage of user data. The plurality of storage nodes are configured to distribute the user data and metadata throughout the plurality of storage nodes with erasure coding of the user data such that the plurality of storage nodes can access the user data, via the erasure coding, with a failure of two of the plurality of storage nodes. The plurality of storage nodes are configured to employ the erasure coding to reconfigure redundancy of the user data responsive to one of adding or removing a storage node.

Abstract: A method of distributed file deletion, performed by a storage system, is provided. The method includes receiving, at the storage system, a request to delete a directory and contents of the directory and adding the directory to a first set, listed in a memory in the storage system. The method includes operating on the first set, by examining each directory in the first set to identify subdirectories, adding each identified subdirectory to the first set as a directory, and adding each examined directory to a second set listed in the memory. The method includes deleting in a distributed manner across the storage system without concern for order, contents of directories, and the directories, listed in the second set.

Abstract: A method for adaptive concurrency for write persistence in a storage system, performed by the storage system, is provided. The method includes selecting a write process from among a plurality of write processes, responsive to receiving a write request for writing data into the storage system, and writing the data into the storage system in accordance with the selected write process. One of the plurality of write processes includes transferring the data into the storage system, locking an inode associated with file information of the data in memory, updating the file information in the inode while the inode is locked, committing the data while the inode is locked, and unlocking the inode.

Abstract: A method for distributed file deletion or truncation, performed by a storage system, is provided. The method includes determining, by an authority owning an inode of a file, which authorities own data portions to be deleted, responsive to a request for the file deletion or truncation. The method includes recording, by the authority owning the inode, the file deletion or truncation in a first memory, and deleting, in background by the authorities that own the data portions to be deleted, the data portions in one of a first memory or a second memory. A system and computer readable media are also provided.

Abstract: The present invention extends to methods, systems, and computer program products for indicating parallel operations with user-visible events. Event markers can be used to indicate an abstracted outer layer of execution as well as expose internal specifics of parallel processing systems, including systems that provide data parallelism. Event markers can be used to show a variety of execution characteristics including higher-level markers to indicate the beginning and end of an execution program (e.g., a query). Inside the execution program (query) individual fork/join operations can be indicated with sub-levels of markers to expose their operations. Additional decisions made by an execution engine, such as, for example, when elements initially yield, when queries overlap or nest, when the query is cancelled, when the query bails to sequential operation, when premature merging or re-partitioning are needed can also be exposed.

Abstract: A storage cluster is provided. The storage cluster includes a plurality of storage nodes within a single chassis. Each of the plurality of storage nodes has nonvolatile solid-state memory for storage of user data. The plurality of storage nodes are configured to distribute the user data and metadata throughout the plurality of storage nodes with erasure coding of the user data such that the plurality of storage nodes can access the user data, via the erasure coding, with a failure of two of the plurality of storage nodes.

Abstract: A method of processing transactions associated with a command in a storage system is provided. The method includes receiving, at a first authority of the storage system, a command relating to user data. The method includes sending a transaction of the command, from the first authority to a second authority of the storage system, wherein a token accompanies the transaction and writing data in accordance with the transaction as permitted by the token into a partition that is allocated to the second authority in a storage device of the storage system.

Abstract: A method for proactively rebuilding user data in a plurality of storage nodes of a storage cluster is provided. The method includes distributing user data and metadata throughout the plurality of storage nodes such that the plurality of storage nodes can read the user data, using erasure coding, despite loss of two of the storage nodes. The method includes determining that one of the storage nodes is unreachable and determining to rebuild the user data for the one of the storage nodes that is unreachable. The method includes reading the user data across a remainder of the plurality of storage nodes, using the erasure coding and writing the user data across the remainder of the plurality of storage nodes, using the erasure coding. A plurality of storage nodes within a single chassis that can proactively rebuild the user data stored within the storage nodes is also provided.

Abstract: A storage cluster is provided. The storage cluster includes a plurality of storage nodes within a single chassis. Each of the plurality of storage nodes has nonvolatile solid-state memory for storage of user data. The plurality of storage nodes are configured to distribute the user data and metadata throughout the plurality of storage nodes with erasure coding of the user data such that the plurality of storage nodes can access the user data, via the erasure coding, with a failure of two of the plurality of storage nodes. The plurality of storage nodes are configured to employ the erasure coding to reconfigure redundancy of the user data responsive to one of adding or removing a storage node.

Abstract: A processor-based method for locating data and metadata closely together in a storage system is provided. The method includes writing a first range of a file and a first metadata relating to attributes of the file into at least one segment controlled by a first authority of the file. The method includes delegating, by the first authority, a second authority for a second range of the file, and writing the second range of the file and second metadata relating to the attributes of the file into at least one segment controlled by the second authority.

Abstract: A plurality of storage nodes is provided. Each of the plurality of storage nodes includes nonvolatile solid-state memory for user data storage. The plurality of storage nodes is configured to distribute the user data and metadata associated with the user data throughout the plurality of storage nodes such that the plurality of storage nodes maintain the ability to read the user data, using erasure coding, despite a loss of two of the plurality of storage nodes. The plurality of storage nodes is configured to initiate an action based on the redundant copies of the metadata, responsive to achieving a level of redundancy for the redundant copies of the metadata. A method for accessing user data in a plurality of storage nodes having nonvolatile solid-state memory is also provided.

Abstract: A method for distribution of directories in a storage system is provided. The method includes distributing information, regarding location in the storage system of a plurality of files in a directory, to a plurality of owners in a plurality of storage nodes of the storage system, wherein ownership of differing subsets of the plurality of files of the directory is distributed among differing owners in differing storage nodes. The method includes receiving a request for location information in the storage system of a file of the plurality of files in the directory and determining, based on a file name of the file and an identity of the directory, which of the plurality of owners has ownership of the file and the location information for the file.

Abstract: A method for accessing a file in a storage system is provided. The method includes determining, for each file chunk of the file, an authority among differing storage nodes of the storage system and receiving from the authority having ownership of the file chunk, location information for the file chunk. The method includes accessing file chunks of the file as directed by each of the determined authorities.

Abstract: A plurality of storage nodes in a single chassis is provided. The plurality of storage nodes in the single chassis is configured to communicate together as a storage cluster. Each of the plurality of storage nodes includes nonvolatile solid-state memory for user data storage. The plurality of storage nodes is configured to distribute the user data and metadata associated with the user data throughout the plurality of storage nodes such that the plurality of storage nodes maintain the ability to read the user data, using erasure coding, despite a loss of two of the plurality of storage nodes. The plurality of storage nodes configured to initiate an action based on the redundant copies of the metadata, responsive to achieving a level of redundancy for the redundant copies of the metadata. A method for accessing user data in a plurality of storage nodes having nonvolatile solid-state memory is also provided.

Abstract: Dynamically allocated thread storage in a computing device is disclosed. The dynamically allocated thread storage is configured to work with a process including two or more threads. Each thread includes a statically allocated thread-local slot configured to store a table. Each table is configured to include a table slot corresponding with a dynamically allocated thread-local value. A dynamically allocated thread-local instance corresponds with the table slot.

Abstract: A dataflow of a distributed application is visualized in a locally simulated execution environment. A scheduler receives a job graph which includes a graph of computational vertices that are designed to be executed on multiple distributed computer systems. The scheduler queries a graph manager to determine which computational vertices of the job graph are ready for execution in a local execution environment. The scheduler queries a cluster manager to determine the organizational topology of the distributed computer systems to simulate the determined topology in the local execution environment. The scheduler queries a data manager to determine data storage locations for each of the computational vertices indicated as being ready for execution in the local execution environment. The scheduler also indicates an instance of each computational vertex to be spawned and executed in the local execution environment based on the organizational topology and indicated data storage locations.

Abstract: Dynamically allocated thread storage in a computing device is disclosed. The dynamically allocated thread storage is configured to work with a process including two or more threads. Each thread includes a statically allocated thread-local slot configured to store a table. Each table is configured to include a table slot corresponding with a dynamically allocated thread-local value. A dynamically allocated thread-local instance corresponds with the table slot.