Abstract: Some embodiments of the invention provide a method for reporting congestion in a network that includes several forwarding elements. In a data plane circuit of one of the forwarding elements, the method detects that a queue in the switching circuit of the data plane circuit is congested, while a particular data message is stored in the queue as it is being processed through the data plane circuit. In the data plane circuit, the method then generates a report regarding the detected queue congestion, and sends this report to a data collector external to the forwarding element. To send the report, the data plane circuit in some embodiments duplicates the particular data message, stores it in the duplicate data message information regarding the detected queue congestion, and sends the duplicate data message to the external data collector.

Abstract: Some embodiments of the invention provide a path-and-latency tracking (PLT) method. At a forwarding element, this method in some embodiments detects the path traversed by a data message through a set of forwarding elements, and the latency that the data message experiences at each of the forwarding elements in the path. In some embodiments, the method has a forwarding element in the path insert its forwarding element identifier and path latency in a header of the data message that it forwards. The method of some embodiments also uses fast PLT operators in the data plane of the forwarding elements to detect new data message flows, to gather PLT data from these data message flows, and to detect path or latency changes for previously detected data message flows. In some embodiments, the method then uses control plane processes (e.g., of the forwarding elements or other devices) to collect and analyze the PLT data gathered in the data plane from new or existing flows.

Abstract: Some embodiments of the invention provide a forwarding element that can be configured through in-band data-plane messages from a remote controller that is a physically separate machine from the forwarding element. The forwarding element of some embodiments has data plane circuits that include several configurable message-processing stages, several storage queues, and a data-plane configurator. A set of one or more message-processing stages of the data plane are configured (1) to process configuration messages received by the data plane from the remote controller and (2) to store the configuration messages in a set of one or more storage queues. The data-plane configurator receives the configuration messages stored in the set of storage queues and configures one or more of the configurable message-processing stages based on configuration data in the configuration messages.

Abstract: Some embodiments of the invention provide a path-and-latency tracking (PLT) method. At a forwarding element, this method in some embodiments detects the path traversed by a data message through a set of forwarding elements, and the latency that the data message experiences at each of the forwarding elements in the path. In some embodiments, the method has a forwarding element in the path insert its forwarding element identifier and path latency in a header of the data message that it forwards. The method of some embodiments also uses fast PLT operators in the data plane of the forwarding elements to detect new data message flows, to gather PLT data from these data message flows, and to detect path or latency changes for previously detected data message flows. In some embodiments, the method then uses control plane processes (e.g., of the forwarding elements or other devices) to collect and analyze the PLT data gathered in the data plane from new or existing flows.

Abstract: Some embodiments of the invention provide a forwarding element that can be configured through in-band data-plane messages from a remote controller that is a physically separate machine from the forwarding element. The forwarding element of some embodiments has data plane circuits that include several configurable message-processing stages, several storage queues, and a data-plane configurator. A set of one or more message-processing stages of the data plane are configured (1) to process configuration messages received by the data plane from the remote controller and (2) to store the configuration messages in a set of one or more storage queues. The data-plane configurator receives the configuration messages stored in the set of storage queues and configures one or more of the configurable message-processing stages based on configuration data in the configuration messages.

Abstract: A manager agent access a grain hierarchy that represents a state for a computation that is to performed by a distributed processing system, wherein the grain hierarchy includes manager agent grains including metadata for manager agent processes that manage the performance of the computation by the distributed processing system, and worker agent grains including, for tasks to be performed by the distributed processing system, metadata for worker agents that each correspond to a subset of the plurality of data processors for performing the task. A manager agent performs processes defined by a manager agent grain to manage the computation by worker agents and storing, within the grain, metadata describing the manager agent process performed by the manager agent, and worker agents perform tasks assigned to the worker agents based on an assignment of a respective worker agent grain to the worker agent.

Abstract: Some embodiments of the invention provide a forwarding element that has one or more data plane, message-processing pipelines with key-value processing circuits. The forwarding element's data plane key-value circuits allow the forwarding element to perform key-value services that would otherwise have to be performed by data compute nodes connected by the network fabric that includes the forwarding element. In some embodiments, the key-value (KV) services of the forwarding element and other similar forwarding elements supplement the key-value services of a distributed set of key-value servers by caching a subset of the most commonly used key-value pairs in the forwarding elements that connect the set of key-value servers with their client applications.

Abstract: Some embodiments of the invention provide a forwarding element (e.g., a switch, a router, etc.) that has one or more data plane, message-processing pipelines with key-value processing circuits. The forwarding element's data plane key-value circuits allow the forwarding element to perform key-value services that would otherwise have to be performed by data compute nodes connected by the network fabric that includes the forwarding element. In some embodiments, the key-value (KV) services of the forwarding element and other similar forwarding elements supplement the key-value services of a distributed set of key-value servers by caching a subset of the most commonly used key-value pairs in the forwarding elements that connect the set of key-value servers with their client applications.

Abstract: A manager agent access a grain hierarchy that represents a state for a computation that is to performed by a distributed processing system, wherein the grain hierarchy includes manager agent grains including metadata for manager agent processes that manage the performance of the computation by the distributed processing system, and worker agent grains including, for tasks to be performed by the distributed processing system, metadata for worker agents that each correspond to a subset of the plurality of data processors for performing the task. A manager agent performs processes defined by a manager agent grain to manage the computation by worker agents and storing, within the grain, metadata describing the manager agent process performed by the manager agent, and worker agents perform tasks assigned to the worker agents based on an assignment of a respective worker agent grain to the worker agent.

Abstract: A manager agent access a grain hierarchy that represents a state for a computation that is to performed by a distributed processing system, wherein the grain hierarchy includes manager agent grains including metadata for manager agent processes that manage the performance of the computation by the distributed processing system, and worker agent grains including, for tasks to be performed by the distributed processing system, metadata for worker agents that each correspond to a subset of the plurality of data processors for performing the task. A manager agent performs processes defined by a manager agent grain to manage the computation by worker agents and storing, within the grain, metadata describing the manager agent process performed by the manager agent, and worker agents perform tasks assigned to the worker agents based on an assignment of a respective worker agent grain to the worker agent.

Abstract: Methods, systems, and apparatus are described for managing metadata for a distributed processing system. In one aspect, a method includes receiving a computation specification that includes a set of grains that specifies an initial state for a computation that is to be performed by a distributed processing system, wherein each grain comprises metadata that specifies a portion of the initial state for the computation; storing a grain hierarchy that represents a state for the computation based on a grain type associated with each grain, the grain hierarchy comprising subscription grains for subscriptions for the grain hierarchy, each subscription corresponding to one or more grains included in the grain hierarchy, and each subscription specifying one or more actions to be performed by the hub device; and performing, during performance of the computation, at least one of the actions specified by at least one of the subscriptions.

Abstract: A method is provided for scrubbing information stored in a data storage system where the information is stored as a plurality of encoded fragments across multiple storage devices. The method includes maintaining on a first storage device a list of metadata entries corresponding to values that are stored in the data storage system at an At Maximum Redundancy (AMR) state, verifying that encoded fragments associated with each of the metadata entries are stored on a second storage, verifying that a corresponding metadata entry is stored on the first storage device for each encoded fragment that is stored on the second storage device, and scheduling for recovery any missing encoded fragments and/or any missing metadata entry.

Abstract: A method for shard assignment in a large-scale data processing job is provided. Datasets are divided into a plurality of shards and the shards are indexed and aggregated into one or more groups. A worker process is initially assigned an indexed shard from a group. The initial assignment can assigned based on a simple algorithm. The worker's subsequent shard assignment is based on the index of the initially assigned shard.

Abstract: Embodiments include methods, apparatus, and systems for retrieving data from multiple locations in storage systems. One embodiment includes a method that determines that data is stored in multiple locations remote to a computer, estimates a latency to retrieve the data from the multiple locations, and requests the data from the plural locations.

Abstract: Methods, systems, and apparatus are described for managing metadata for a distributed processing system. In one aspect, a method includes receiving a computation specification that includes a set of grains that specifies an initial state for a computation that is to be performed by a distributed processing system, wherein each grain comprises metadata that specifies a portion of the initial state for the computation; storing a grain hierarchy that represents a state for the computation based on a grain type associated with each grain, the grain hierarchy comprising subscription grains for subscriptions for the grain hierarchy, each subscription corresponding to one or more grains included in the grain hierarchy, and each subscription specifying one or more actions to be performed by the hub device; and performing, during performance of the computation, at least one of the actions specified by at least one of the subscriptions.

Abstract: A system comprises a processor and storage containing software executable by the processor. The storage also contains a trace log that contains information pertaining to read and write operations and, for each read and write operation, the information is indicative of a start time, a completion time, and a value targeted by the read or write operation, Based on the trace log, the software causes the processor to construct a directed graph comprising nodes as well as edges interconnecting at least some of the nodes, each node representing a read or write operation and determine whether the constructed directed graph has a cycle. At least one edge is at least one of a data edge representing a data precedence between operations and a time edge representing a time precedence between operations, and at least one edge is a hybrid edge representing both time and data precedence between operations.

Abstract: A method of recovering a value in a data storage system includes maintaining a list of entries corresponding to values that have been inserted into the data storage system, determining whether the values are at an At Maximum Redundancy (AMR) state, and if not, the converging the values to the AMR state.

Abstract: A method for verifying a consistency level in a key-value store, in which a value is stored in a cloud-based storage system comprising a read/write register identified by a key. At a centralized monitor node, a history of operations including writes and reads performed at the key is created, and a distance between a read of a value at the key and a latest write to the key is determined. It can then be ascertained whether the distance satisfies a relaxed atomicity property.

Abstract: Systems and methods of enhanced backup job scheduling are disclosed. An example method may include determining a number of jobs (n) in a backup set, determining a number of tape drives (m) in the backup device, and determining a number of concurrent disk agents (maxDA) configured for each tape drive. The method may also include defining a scheduling problem based on n, m, and maxDA. The method may also include solving the scheduling problem using an integer programming (IP) formulation to derive a bin-packing schedule that minimizes makespan (S) for the backup set.

Abstract: A method for verifying a consistency level in a key-value store, in which a value is stored in a cloud-based storage system comprising a read/write register identified by a key. At a centralized monitor node, a history of operations including writes and reads performed at the key is created, and a distance between a read of a value at the key and a latest write to the key is determined. It can then be ascertained whether the distance satisfies a relaxed atomicity property.