Abstract: In general, a value of a numerical attribute of a record stored in a data structure is received. A numerical range is generated that includes the value of the numerical attribute. An entry is stored, in an index associated with the data structure, that specifies a location of the record within the data structure and that includes a first index key and a second index key. The first index key corresponds to a value of an attribute of the record different from the numerical attribute, and the second index key corresponds to the generated numerical range.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for specifying a user interface. One of the methods includes receiving an identification of at least one configurable application, each configurable application being configurable by one or more parameters. The method includes identifying a set of parameters associated with the at least one configurable application. The method includes displaying, in a second user interface, a graphical display element, the graphical display element having properties and associated with a location relative to the first user interface. The method includes displaying, in the second user interface, one or more graphical representations of the set of parameters. The method includes receiving a mapping between a property of the graphical display element and a parameter of the set of parameters. The method also includes storing a representation of the mapping and the location.

Abstract: A graph-based program specification includes components corresponding to tasks and directed links between ports of the components, including: a first type of link configuration defined by respective output and input ports of linked components, and a second type of link configuration defined by respective output and input ports of linked components. A compiler recognizes different types of link configurations and provides in a target program specification occurrences of a target primitive for executing a function for each occurrence of a data element flowing over a link of the second type. A computing node initiates execution of the target program specification, and determines at runtime, for components associated with the occurrences of the target primitive, an order in which instances of tasks corresponding to the components are to be invoked, and/or a computing node on which instances of tasks corresponding to the components are to be executed.

Abstract: In general, a specification of multiple contexts that are related according to a hierarchy is received. Relationships are determined among three or more metadata objects, and at least some of the metadata objects are grouped into one or more respective groups. Each of at least some of the groups is based on a selected one of the contexts and is represented by a node in a diagram. Relationships among the nodes are determined based on the relationships among the metadata objects in the groups represented by the nodes, and a visual representation is generated of the diagram including the nodes and the relationships among the nodes.

Abstract: Information representative of a graph-based program specification has a plurality of components, each of which corresponds to a task, and directed links between ports of said components. A program corresponding to said graph-based program specification is executed. A first component includes a first data port, a first control port, and a second control port. Said first data port is configured to receive data to be processed by a first task corresponding to said first component, or configured to provide data that was processed by said first task corresponding to said first component. Executing a program corresponding to said graph-based program specification includes: receiving said first control information at said first control port, in response to receiving said first control information, determining whether or not to invoke said first task, and after receiving said first control information, providing said second control information from said second control port.

Abstract: Presenting a diagram indicating relationships among data items stored in a data management system includes: receiving a request that identifies a first data item stored in the data management system from a user interface; retrieving stored configuration information that includes a plurality of selection specifications for selecting data items in the data management system that are related to a given data item of a predetermined type, where each selection specification is associated with a different respective predetermined type; querying the data management system to identify a set of one or more data items according to a selection specification from the configuration information that is associated with a type of the first data item; for each of multiple returned data items in the identified set, querying the data management system to determine whether additional data items are identified according to a selection specification from the configuration information that is associated with a type of the returned d

Abstract: Processing data includes: receiving units of work that each include one or more work elements, and processing a first unit of work using a first compiled dataflow graph (160) loaded into a data processing system (100) in response to receiving the first unit of work. The processing includes: analysis to determine a characteristic of the first unit of work; identifying one or more compiled dataflow graphs from graphs stored in a data storage system (107) that include at least some that were compiled for processing a unit of work having the determined characteristic; loading one of the identified compiled dataflow graphs into the data processing system (100) as the first compiled dataflow graph (160); and generating one or more output work elements from at least one work element in the first unit of work.

Abstract: A graph-based program specification includes components, at least one having at least one input port for receiving a collection of data elements, or at least one collection type output port for providing a collection of data elements. Executing a program specified by the graph-based program specification at a computing node, includes: receiving data elements of a first collection into a first storage in a first order via a link connected to a collection type output port of a first component and an input port of a second component, and invoking a plurality of instances of a task corresponding to the second component to process data elements of the first collection, including retrieving the data elements from the first storage in a second order, without blocking invocation of any of the instances until after any particular instance completes processing one or more data elements.

Abstract: One or more expressions are evaluated that represent one or more characteristics of a dataflow graph that includes vertices representing data processing components connected by links representing flows of work elements between the components. A request is received by a computing system to evaluate the one or more expressions that include one or more operations on one or more variables; and the one or more expressions are evaluated by the computing system. The evaluating includes: defining a data structure that includes one or more fields, collecting, during execution of the dataflow graph, tracking information associated with one or more components of the dataflow graph, storing values associated with the tracking information in the one or more fields, and replacing one or more variables of the one or more expressions with the values stored in the one or more fields to compute a result of evaluating the one or more expressions.

Abstract: This specification describes technologies relating to generating transforms based on rule sets. In general, one aspect described in this specification can be embodied in methods that include receiving a rule set including execution cases, where at least one execution case in the rule set includes one or more trigger conditions and a specification of an output that is to be generated when the one or more trigger conditions are all satisfied. The methods may further include generating a control structure including a sequence of rows corresponding to one or more execution cases in the rule set. Each row may include a sequence of one or more trigger conditions and information specifying the output for a corresponding execution case. For at least one of the trigger conditions, when the trigger condition is failed, the control structure may direct processing to skip at least one row in the sequence of rows.

Abstract: A data processing system configured to store a plurality of data entities in volatile memories of multiple different computing devices. The data processing system comprises a first computing device having a first volatile memory configured to store a first data entity; and a second computing device having a second volatile memory configured to store a copy of the first data entity. The first computing device is configured to perform: receiving an indication to update the first data entity; after receiving the indication, updating the first data entity in the first volatile memory, and providing to the second computing device an indication to update the copy of the first data entity; and providing an indication that the first data entity has been updated, after receiving information from the second computing device indicating that the copy of the first data entity has been updated in the second volatile memory.

Abstract: According to some aspects, a method of operating a data processing system is provided wherein at least one computer program is configured, the data processing system comprising least a first control, a second control and a third control, the first, second and third controls comprising at least user interface portions and operational portions, the method comprising rendering a first user interface based on the user interface portion of the first control, receiving first user input through the first user interface, the first user input providing configuration information for the at least one program, identifying the second control based at least in part on the operational portion of the first control, rendering a second user interface based on the user interface portion of the identified second control, and receiving second user input through the second user interface, the second user input providing configuration information for the at least one program.

Abstract: A method for promoting fault tolerance and recovery in a computing system including at least one processing node includes promoting availability and recovery of a first processing node, by, at the first processing node, generating first spawn using a spawner that has been assigned a first generation-indicator so that its spawn inherits the first generation indicator, beginning a checkpoint interval to generate nodal recovery information, suspending the spawner from generating spawn, assigning, to the spawner, a second generation-indicator that differs from the first one, resuming the spawner, so that it generates second spawn that inherits the second generation-indicator, controlling an extent to which the second spawn writes to memory, and after committing nodal recovery information acquired during the checkpoint to durable storage, releasing control over the extent to which the second spawn can write to memory.

Abstract: Controlling tasks includes: receiving ordering information that specifies at least a partial ordering among a plurality of tasks; and generating instructions for performing at least some of the tasks based at least in part on the ordering information. Instructions are stored for executing a first subroutine corresponding to a first task, including a first control section that controls execution of at least a second subroutine corresponding to a second task, the first control section including a function configured to change state information associated with the second task, and to determine whether or not to initiate execution of the second subroutine based on the changed state information. Instructions are stored for executing the second subroutine, including a task section for performing the second task and a second control section that controls execution of a third subroutine corresponding to a third task.

Abstract: One method includes receiving a database query, receiving information about a database table in data storage populated with data elements, producing a structural representation of the database table that includes a formatted data organization reflective of the database table and is absent the data elements of the database table, and providing the structural representation and the database query to a plan generator capable of producing a query plan representing operations for executing the database query on the database table. Another method includes receiving a query plan from a plan generator, the plan representing operations for executing a database query on a database table, and producing a dataflow graph from the query plan, wherein the dataflow graph includes at least one node that represents at least one operation represented by the query plan, and includes at least one link that represents at least one dataflow associated with the query plan.

Abstract: A data storage system stores at least one dataset including a plurality of records. A data processing system, coupled to the data storage system, processes the plurality of records to produce codes representing data patterns in the records, the processing including: for each of multiple records in the plurality of records, associating with the record a code encoding one or more elements, wherein each element represents a state or property of a corresponding field or combination of fields as one of a set of element values, and, for at least one element of at least a first code, the number of element values in the set is smaller than the total number of data values that occur in the corresponding field or combination of fields over all of the plurality of records in the dataset.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for specifying a user interface. One of the methods includes receiving an identification of at least one configurable application, each configurable application being configurable by one or more parameters. The method includes identifying a set of parameters associated with the at least one configurable application. The method includes displaying, in a second user interface, a graphical display element, the graphical display element having properties and associated with a location relative to the first user interface. The method includes displaying, in the second user interface, one or more graphical representations of the set of parameters. The method includes receiving a mapping between a property of the graphical display element and a parameter of the set of parameters. The method also includes storing a representation of the mapping and the location.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating conditional validation rules. One of the methods includes rendering a plurality of cells arranged in a two-dimensional grid having a first axis and a second axis, the two-dimensional grid including one or more subsets of the cells, each subset associated with a respective field of an element of the dataset, and multiple subsets of the cells extending in a direction along the second axis of the two-dimensional grid, one or more of the multiple subsets associated with a respective validation rule. The method includes applying one or more validation rules to an element of the dataset based on user input received from at least some of the cells. A condition cell associated with a field includes an input element for receiving input.

Abstract: In a first aspect, a method includes, at a node of a Hadoop cluster, the node storing a first portion of data in HDFS data storage, executing a first instance of a data processing engine capable of receiving data from a data source external to the Hadoop cluster, receiving a computer-executable program by the data processing engine, executing at least part of the program by the first instance of the data processing engine, receiving, by the data processing engine, a second portion of data from the external data source, storing the second portion of data other than in HDFS storage, and performing, by the data processing engine, a data processing operation identified by the program using at least the first portion of data and the second portion of data.

Abstract: A method for clustering data elements stored in a data storage system includes reading data elements from the data storage system. Clusters of data elements are formed with each data element being a member of at least one cluster. At least one data element is associated with two or more clusters. Membership of the data element belonging to respective ones of the two or more clusters is represented by a measure of ambiguity. Information is stored in the data storage system to represent the formed clusters.