Abstract: Profiling data includes processing an accessed collection of records, including: generating, for a first set of distinct values appearing in a first set of one or more fields, corresponding location information; generating, for the first set of fields, a corresponding list of entries identifying a distinct value from the first set of distinct values and the location information for the distinct value; generating, for a second set of one or more fields, a corresponding list of entries, with each entry identifying a distinct value from a second set of distinct values appearing in the second set of fields; and generating result information, based at least in part on: locating at least one record of the collection using the location information for at least one value appearing in the first set of fields, and determining at least one value appearing in the second set of fields of the located record.

Abstract: A computer-implemented method for managing data objects stored in a data repository and displaying elements associated with data objects on a graphical user interface is presented. The method includes using reusable logic to provide at least one displayed input element associated with a first data object stored in the data repository. In response to user input associated with the displayed input element, the method includes generating a second data object stored in the data repository, and linking the second data object with one or more related data objects in the data repository automatically based on the data object associated with the displayed element.

Abstract: Among other aspects disclosed are a method and system for processing a batch of input data in a fault tolerant manner. The method includes reading a batch of input data including a plurality of records from one or more data sources and passing the batch through a dataflow graph. The dataflow graph includes two or more nodes representing components connected by links representing flows of data between the components. At least one but fewer than all of the components includes a checkpoint process for an action performed for each of multiple units of work associated with one or more of the records. The checkpoint process includes opening a checkpoint buffer stored in non-volatile memory at the start of processing for the batch.

Abstract: A data storage system stores at least one dataset including a plurality of records. A data processing system, coupled to the data storage system, processes the plurality of records to produce codes representing data patterns in the records, the processing including: for each of multiple records in the plurality of records, associating with the record a code encoding one or more elements, wherein each element represents a state or property of a corresponding field or combination of fields as one of a set of element values, and, for at least one element of at least a first code, the number of element values in the set is smaller than the total number of data values that occur in the corresponding field or combination of fields over all of the plurality of records in the dataset.

Abstract: A method includes receiving information provided by a data processing application during execution of the data processing application. The information is indicative of at least one of a source of data for the data processing application and a destination of data from the data processing application. The method includes dynamically analyzing the information during execution of the data processing application to identify a queue in communication with the data processing application; and dynamically analyzing the information during execution of the data processing application to identify a relationship between the data processing application and the queue, including at least one of identifying that the queue is the source of data for the data processing application and identifying that the queue is the destination of data from the data processing application.

Abstract: Records received from one or more sources in a network are processed. For each of multiple intervals of time, a matching procedure is attempted on sets of one or more records, including comparing identifiers associated with different records to generate the sets and determining whether or not a completeness criterion is satisfied for one or more of the sets. The processing also includes, for at least some of the intervals of time, processing at least one complete set, consisting of one or more of the received records on which the matching procedure is first attempted during the interval of time and one or more records stored in a data store before the interval of time, and for at least some of the intervals of time, processing at least one incomplete set, consisting of one or more records stored in the data store before the interval of time.

Abstract: In a first aspect, a method includes, at a node of a Hadoop cluster, the node storing a first portion of data in HDFS data storage, executing a first instance of a data processing engine capable of receiving data from a data source external to the Hadoop cluster, receiving a computer-executable program by the data processing engine, executing at least part of the program by the first instance of the data processing engine, receiving, by the data processing engine, a second portion of data from the external data source, storing the second portion of data other than in HDFS storage, and performing, by the data processing engine, a data processing operation identified by the program using at least the first portion of data and the second portion of data.

Abstract: Determining metadata associated with a graph-based computation includes functionally transforming metadata associated with a first portion of a graph to generate transformed metadata associated with a second portion of the graph; determining a third portion of the graph related to the second portion of the graph; and propagating the transformed metadata from the second portion of the graph to the third portion of the graph.

Abstract: Profiling data includes processing an accessed collection of records, including: generating, for a first set of distinct values appearing in a first set of one or more fields, corresponding location information; generating, for the first set of fields, a corresponding list of entries identifying a distinct value from the first set of distinct values and the location information for the distinct value; generating, for a second set of one or more fields, a corresponding list of entries, with each entry identifying a distinct value from a second set of distinct values appearing in the second set of fields; and generating result information, based at least in part on: locating at least one record of the collection using the location information for at least one value appearing in the first set of fields, and determining at least one value appearing in the second set of fields of the located record.

Abstract: A method includes determining a first quantity of data records of a group of data records from a stream of data records received by an application having a plurality of modules. The method includes, for one or more of the modules of the application, determining a respective second quantity of data records output by the module during processing of the group of data records. The method includes determining whether the first and second quantities of data records satisfy a rule. The rule is indicative of a target relationship among a quantity of data records received by the application and a quantity of data records output by one or more modules of the application.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for specifying logical rules, one of the methods includes defining a logical rule, the logical rule applying operations based on a term. The method includes defining a mapping between fields and terms, the mapping including a mapping between a field and the term. The method includes storing the logical rule in association with the term. The method also includes applying the logical rule to data identified by the first field where respective fields are assigned to respective terms.

Abstract: Systems and techniques are disclosed that include in one aspect a computer implemented method storing a received stream of data elements in a buffer, applying a boundary condition to the data elements stored in the buffer after receiving each individual data element of the stream of data elements, and producing one or more data elements from the buffer based on the boundary condition as an output stream of data elements sorted according to a predetermined order.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for specifying a compound control. One of the methods includes identifying a first application. The method includes displaying a canvas. The method includes displaying, in the canvas, a first display object associated with the first application. The method includes identifying a second application, the second application being a computer executable program. The method includes displaying, in the user interface, a second display object associated with a second application. The method includes, in response to a user action that associates the second display object with the first display object, configuring the first application to invoke the second application. The method includes creating a third display object that includes the first set of selector objects and the second set of selector objects.

Abstract: Techniques are disclosed that include a computer-implemented method including transmitting a message in response to a predetermined event through a process stage including at least first and second processes being executed as one or more tasks, the message instructing the abortion of the executing of the one or more tasks, and initiating abortion of execution of the one or more tasks by the one or more of the processes on receiving the messages.

Abstract: In one aspect, in general, a method of generating a dataflow graph representing a database query includes receiving a query plan from a plan generator, the query plan representing operations for executing a database query on at least one input representing a source of data, producing a dataflow graph from the query plan, wherein the dataflow graph includes at least one node that represents at least one operation represented by the query plan, and includes at least one link that represents at least one dataflow associated with the query plan, and altering one or more components of the dataflow graph based on at least one characteristic of the at least one input representing the source of data.

Abstract: Interactions with data stored in a data storage system are managed through a view of an interface provided on a display. Information specifying multiple data elements and multiple groupings of related data elements is received. At least one of the data elements is a member of multiple of the groupings. A view is rendered that includes tokens representing respective data elements. At least some tokens include a selectable indicator including portions that indicate membership of the data element represented by that token in each of multiple groupings of related data elements. A first selection of a portion of one of the selectable indicators is received, corresponding to a first selected grouping of related data elements. In response to the first selection, tokens representing one or more other members of the first selected grouping of related data elements are rendered in the view.

Abstract: Systems and techniques are disclosed that include in one aspect a computer implemented method storing a received stream of data elements in a buffer, applying a boundary condition to the data elements stored in the buffer after receiving each individual data element of the stream of data elements, and producing one or more data elements from the buffer based on the boundary condition as an output stream of data elements sorted according to a predetermined order.

Abstract: Techniques are disclosed that include a computer-implemented method, including storing information related to an initial state of a process upon being initialized, wherein execution of the process includes executing at least one execution phase and upon completion of the executing of the execution phase storing information representative of an end state of the execution phase; aborting execution of the process in response to a predetermined event; and resuming execution of the process from one of the saved initial and end states without needing to shut down the process.

Abstract: Maintaining data in a data management system including at least one processor and a storage system includes accessing a first data source providing one or more source elements of data; accessing a storage system storing one or more subsets of data, each subset including at least one element of data, and storing one or more descriptors identifying a data source for a subset and identifying a data source for each of one or more elements of the subset; and processing rules that determine if, according to the descriptors, an element in the storage system is allowed to be changed.