Abstract: A computer implemented processing service for efficient streaming of data input from one or more sources to one or more receivers is disclosed. The processing service includes a schema manager that receives commands from a tenant of the data stream processing service. The processing service includes a processing services gateway that validates the data and sends the data to a transformation processor. The transformation processor receives the data and parses the data into the one or more data formats in accordance with the instructions and sends the data in the one or more data formats to an egress service. The egress service outputs the data in the one or more data formats to the one or more receivers, each receiver receiving the data in a data pipeline in one of the one or more formats.

Abstract: A computer implemented processing service for efficient streaming of data input from one or more sources to one or more receivers is disclosed. The processing service includes a schema manager that receives commands from a tenant of the data stream processing service. The processing service includes a processing services gateway that validates the data and sends the data to a transformation processor. The transformation processor receives the data and parses the data into the one or more data formats in accordance with the instructions and sends the data in the one or more data formats to an egress service. The egress service outputs the data in the one or more data formats to the one or more receivers, each receiver receiving the data in a data pipeline in one of the one or more formats.

Abstract: A synchronous packet-processing pipeline whose data paths are populated with data-plane stateful processing units (DSPUs) is provided. A DSPU is a programmable processor whose operations are synchronous with the dataflow of the packet-processing pipeline. A DSPU performs every computation with fixed latency. Each DSPU is capable of maintaining a set of states and perform its computations based on its maintained set of states. The programming of a DSPU determines how and when the DSPU updates one of its maintained states. Such programming may configure the DSPU to update the state based on its received packet data, or to change the state regardless of the received packet data.

Abstract: General-purpose distributed data-parallel computing using a high-level language is disclosed. Data parallel portions of a sequential program that is written by a developer in a high-level language are automatically translated into a distributed execution plan. The distributed execution plan is then executed on large compute clusters. Thus, the developer is allowed to write the program using familiar programming constructs in the high level language. Moreover, developers without experience with distributed compute systems are able to take advantage of such systems.

Abstract: General-purpose distributed data-parallel computing using a high-level language is disclosed. Data parallel portions of a sequential program that is written by a developer in a high-level language are automatically translated into a distributed execution plan. The distributed execution plan is then executed on large compute clusters. Thus, the developer is allowed to write the program using familiar programming constructs in the high level language. Moreover, developers without experience with distributed compute systems are able to take advantage of such systems.

Abstract: General-purpose distributed data-parallel computing using a high-level language is disclosed. Data parallel portions of a sequential program that is written by a developer in a high-level language are automatically translated into a distributed execution plan. The distributed execution plan is then executed on large compute clusters. Thus, the developer is allowed to write the program using familiar programming constructs in the high level language. Moreover, developers without experience with distributed compute systems are able to take advantage of such systems.

Abstract: A synchronous packet-processing pipeline whose data paths are populated with data-plane stateful processing units (DSPUs) is provided. A DSPU is a programmable processor whose operations are synchronous with the dataflow of the packet-processing pipeline. A DSPU performs every computation with fixed latency. Each DSPU is capable of maintaining a set of states and perform its computations based on its maintained set of states. The programming of a DSPU determines how and when the DSPU updates one of its maintained states. Such programming may configure the DSPU to update the state based on its received packet data, or to change the state regardless of the received packet data.

Abstract: A document index is generated from a set of documents and is used to identify documents that match one or more queries. A tree is generated for each document with a node corresponding to each object of the document. The nodes of the generated trees are merged or combined to generate the document index, which is itself a tree. In addition, an inverted index is generated for each node of the index that identifies the tree(s) that the node originated from. When a query is received, the query is first executed against the document index tree: during the execution, proper set operations are applied to the inverted indices associated with the nodes matched by the query. The resulted set identifies the documents that may match the query. The query is then executed on the identified documents.

Abstract: General-purpose distributed data-parallel computing using a high-level language is disclosed. Data parallel portions of a sequential program that is written by a developer in a high-level language are automatically translated into a distributed execution plan. The distributed execution plan is then executed on large compute clusters. Thus, the developer is allowed to write the program using familiar programming constructs in the high level language. Moreover, developers without experience with distributed compute systems are able to take advantage of such systems.

Abstract: A distributed data-parallel execution (DDPE) system splits a computational problem into a plurality of sub-problems using a branch-and-bound algorithm, designates a synchronous stop time for a “plurality of processors” (for example, a cluster) for each round of execution, processes the search tree by recursively using a branch-and-bound algorithm in multiple rounds (without inter-processor communications), determines if further processing is required based on the processing round state data, and terminates processing on the processors when processing is completed.

Abstract: General-purpose distributed data-parallel computing using a high-level language is disclosed. Data parallel portions of a sequential program that is written by a developer in a high-level language are automatically translated into a distributed execution plan. The distributed execution plan is then executed on large compute clusters. Thus, the developer is allowed to write the program using familiar programming constructs in the high level language. Moreover, developers without experience with distributed compute systems are able to take advantage of such systems.

Abstract: A system for visualizing correlations between attributes in a data set or across multiple data sets is provided. A user may view a graphical representation (e.g., a histogram) of attribute values for a first attribute. The user may assign a variety of graphical indicators to various value ranges of the first attribute. The user may view a graphical representation of the second attributes. The user may “drag and drop” the graphical representation of the first attributes onto the graphical representation of the second attributes. The graphical representation of the second attributes may be updated to incorporate the graphical elements assigned by the user to the value ranges of the first attribute. The user may visually see potential correlations between the first and the second attributes based on the graphical elements associated with the first attributes displayed with the associated second attributes.

Abstract: A document index is generated from a set of documents and is used to identify documents that match one or more queries. A tree is generated for each document with a node corresponding to each object of the document. The nodes of the generated trees are merged or combined to generate the document index, which is itself a tree. In addition, an inverted index is generated for each node of the index that identifies the tree(s) that the node originated from. When a query is received, the query is first executed against the document index tree: during the execution, proper set operations are applied to the inverted indices associated with the nodes matched by the query. The resulted set identifies the documents that may match the query. The query is then executed on the identified documents.

Abstract: In a distributed system a plurality of devices (including computing units, storage and communication units) are monitored by an automated repair service that uses sensors and performs one or more repair actions on computing devices that are found to fail according to repair policies. The repair actions include automated repair actions and non-automated repair actions. The health of the computing devices is recorded in the form of states along with the repair actions that were performed on the computing devices and the times at which the repair actions were performed, and events generated by both sensors and the devices themselves. After some period of the time, the history of states of each device, the events, and the repair actions performed on the computing devices are analyzed to determine the effectiveness of the repair actions.

Abstract: A modular compiler architecture utilizes partial compiler modules that cooperatively produce object code for operation on a complex execution infrastructure. The partial compilers may invoke the services of other partial compilers, wherein each partial compiler operates as a self-contained “black-box” module. This structure, in turn, may allow the partial compilers of such implementations to be arranged in modular hierarchies for multi-level compilation and specialization of each partial compiler. These various implementations, in turn, produce compiled programs able to correctly run on large computer clusters comprising a mix of computational resources (machines, multiple cores, graphics cards, SQL server engines, etc.). Certain implementations may also be directed to compilers comprising modular partial compilers, and partial compilers may be formed from generalized forms of traditional compilers. Further disclosed is a set of high-level operations that manipulate partial compilers.

Abstract: A distributed data-parallel execution (DDPE) system splits a computational problem into a plurality of sub-problems using a branch-and-bound algorithm, designates a synchronous stop time for a “plurality of processors” (for example, a cluster) for each round of execution, processes the search tree by recursively using a branch-and-bound algorithm in multiple rounds (without inter-processor communications), determines if further processing is required based on the processing round state data, and terminates processing on the processors when processing is completed.

Abstract: In a distributed system a plurality of devices (including computing units, storage and communication units) are monitored by an automated repair service that uses sensors and performs one or more repair actions on computing devices that are found to fail according to repair policies. The repair actions include automated repair actions and non-automated repair actions. The health of the computing devices is recorded in the form of states along with the repair actions that were performed on the computing devices and the times at which the repair actions were performed, and events generated by both sensors and the devices themselves. After some period of the time, the history of states of each device, the events, and the repair actions performed on the computing devices are analyzed to determine the effectiveness of the repair actions.

Abstract: The Global Dynamic Critical Path is used to optimize the design of a system-on-a-chip (SoC), where hardware modules are in different clock domains. Control signal transitions of the hardware modules are analyzed to identify the Global Dynamic Critical Path. Rules are provided for handling specific situations such as when concurrent input control signals are received by a hardware module. A configuration of the hardware modules is modified in successive iterations to converge at an optimum design, based on a cost function. The cost function can account for processing time as well as other metrics, such as power consumed. For example, during the iterations, hardware modules which are in the Global Dynamic Critical Path can have their clock speed increased and/or additional resources can be added, while hardware modules which are not in the Global Dynamic Critical Path can have their clock speed decreased and/or unnecessary resources can be removed.

Abstract: A system for visualizing correlations between attributes in a data set or across multiple data sets is provided. A user may view a graphical representation (e.g., a histogram) of attribute values for a first attribute. The user may assign a variety of graphical indicators to various value ranges of the first attribute. The user may view a graphical representation of the second attributes. The user may “drag and drop” the graphical representation of the first attributes onto the graphical representation of the second attributes. The graphical representation of the second attributes may be updated to incorporate the graphical elements assigned by the user to the value ranges of the first attribute. The user may visually see potential correlations between the first and the second attributes based on the graphical elements associated with the first attributes displayed with the associated second attributes.

Abstract: General-purpose distributed data-parallel computing using a high-level language is disclosed. Data parallel portions of a sequential program that is written by a developer in a high-level language are automatically translated into a distributed execution plan. The distributed execution plan is then executed on large compute clusters. Thus, the developer is allowed to write the program using familiar programming constructs in the high level language. Moreover, developers without experience with distributed compute systems are able to take advantage of such systems.