Abstract: Combining specifications of dataflow graphs includes receiving: a first dataflow graph specification that specifies two or more components connected by links representing flows of data, and a second dataflow graph specification that specifies at least one component, and at least one sub-graph interface. The sub-graph interface includes at least one flow junction representing a connection between: (1) a flow of data outside the sub-graph interface, and (2) a flow of data inside the sub-graph interface. The method includes processing information including the first dataflow graph specification and the second dataflow graph specification, to generate a combined dataflow graph specification, including: identifying an association between the sub-graph interface and the first dataflow graph specification, for at least a first flow junction, determining a direction associated with transferring a value of a descriptor, and transferring a value of a descriptor according to the determined direction.

Abstract: Specifying a dataflow graph includes: rendering, in a first user interface, a representation of a first dataflow graph. At least one component represents a computation associated with at least one of data flowing into an input port or data flowing out of an output port. A link between components is rendered based on user input indicating a flow of data. In a second user interface, a representation of a sub-graph of the first dataflow graph is rendered. A first flow junction represents a connection between: (1) a flow of data outside the sub-graph interface, and (2) a flow of data inside the sub-graph interface. A link is rendered between the first flow junction and a second flow junction based on user input indicating a relationship between a first descriptor and a second descriptor.

Abstract: Combining specifications of dataflow graphs includes receiving: a first dataflow graph specification that specifies two or more components connected by links representing flows of data, and a second dataflow graph specification that specifies at least one component, and at least one sub-graph interface. The sub-graph interface includes at least one flow junction representing a connection between: (1) a flow of data outside the sub-graph interface, and (2) a flow of data inside the sub-graph interface. The method includes processing information including the first dataflow graph specification and the second dataflow graph specification, to generate a combined dataflow graph specification, including: identifying an association between the sub-graph interface and the first dataflow graph specification, for at least a first flow junction, determining a direction associated with transferring a value of a descriptor, and transferring a value of a descriptor according to the determined direction.

Abstract: Combining specifications of dataflow graphs includes receiving: a first dataflow graph specification that specifies two or more components connected by links representing flows of data, and a second dataflow graph specification that specifies at least one component, and at least one sub-graph interface. The sub-graph interface includes at least one flow junction representing a connection between: (1) a flow of data outside the sub-graph interface, and (2) a flow of data inside the sub-graph interface. The method includes processing information including the first dataflow graph specification and the second dataflow graph specification, to generate a combined dataflow graph specification, including: identifying an association between the sub-graph interface and the first dataflow graph specification, for at least a first flow junction, determining a direction associated with transferring a value of a descriptor, and transferring a value of a descriptor according to the determined direction.

Abstract: In certain aspects, a method includes: receiving a first graph that includes components and flows, the components representing operations performed on data records, the flows representing flows of data records between components; receiving a specification that is separate from the first graph, the specification defining one or more insertions, each of the insertions associated with a flow of the first graph; generating one or more components that each corresponds to one of the insertions; and generating a second graph that includes components and flows that correspond to at least some of the components and flows of the first graph and the one or more generated components.

Abstract: Specifying a dataflow graph includes: rendering, in a first user interface, a representation of a first dataflow graph. At least one component represents a computation associated with at least one of data flowing into an input port or data flowing out of an output port. A link between components is rendered based on user input indicating a flow of data. In a second user interface, a representation of a sub-graph of the first dataflow graph is rendered. A first flow junction represents a connection between: (1) a flow of data outside the sub-graph interface, and (2) a flow of data inside the sub-graph interface. A link is rendered between the first flow junction and a second flow junction based on user input indicating a relationship between a first descriptor and a second descriptor.

Abstract: Processing data includes profiling data from a data source, including reading the data from the data source, computing summary data characterizing the data while reading the data, and storing profile information that is based on the summary data. The data is then processed from the data source. This processing includes accessing the stored profile information and processing the data according to the accessed profile information.

Abstract: In certain aspects, a method includes: receiving a first graph that includes components and flows, the components representing operations performed on data records, the flows representing flows of data records between components; receiving a specification that is separate from the first graph, the specification defining one or more insertions, each of the insertions associated with a flow of the first graph; generating one or more components that each corresponds to one of the insertions; and generating a second graph that includes components and flows that correspond to at least some of the components and flows of the first graph and the one or more generated components.

Abstract: Processing data includes profiling data from a data source, including reading the data from the data source, computing summary data characterizing the data while reading the data, and storing profile information that is based on the summary data. The data is then processed from the data source. This processing includes accessing the stored profile information and processing the data according to the accessed profile information.

Abstract: Combining specifications of dataflow graphs includes receiving: a first dataflow graph specification that specifies two or more components connected by links representing flows of data, and a second dataflow graph specification that specifies at least one component, and at least one sub-graph interface. The sub-graph interface includes at least one flow junction representing a connection between: (1) a flow of data outside the sub-graph interface, and (2) a flow of data inside the sub-graph interface. The method includes processing information including the first dataflow graph specification and the second dataflow graph specification, to generate a combined dataflow graph specification, including: identifying an association between the sub-graph interface and the first dataflow graph specification, for at least a first flow junction, determining a direction associated with transferring a value of a descriptor, and transferring a value of a descriptor according to the determined direction.

Abstract: Processing data includes profiling data from a data source, including reading the data from the data source, computing summary data characterizing the data while reading the data, and storing profile information that is based on the summary data. The data is then processed from the data source. This processing includes accessing the stored profile information and processing the data according to the accessed profile information.

Abstract: Processing data includes profiling data from a data source, including reading the data from the data source, computing summary data characterizing the data while reading the data, and storing profile information that is based on the summary data. The data is then processed from the data source. This processing includes accessing the stored profile information and processing the data according to the accessed profile information.

Abstract: A method for managing data includes receiving individually accessible data units, each identified by a key value; storing a plurality of blocks of data, each of at least some of the blocks being generated by combining a plurality of the data units; and providing an index that includes an entry for each of the blocks. One or more of the entries enable location, based on a provided key value, of a block that includes data units corresponding to a range of key values that includes the provided key value.

Abstract: A method for managing data includes receiving individually accessible data units, each identified by a key value; storing a plurality of blocks of data, each of at least some of the blocks being generated by combining a plurality of the data units; and providing an index that includes an entry for each of the blocks. One or more of the entries enable location, based on a provided key value, of a block that includes data units corresponding to a range of key values that includes the provided key value.

Abstract: A method for managing data includes receiving individually accessible data units, each identified by a key value; storing a plurality of blocks of data, each of at least some of the blocks being generated by combining a plurality of the data units; and providing an index that includes an entry for each of the blocks. One or more of the entries enable location, based on a provided key value, of a block that includes data units corresponding to a range of key values that includes the provided key value.

Abstract: Processing data includes accepting information characterizing values of a first field in records of a first data source and information characterizing values of a second field in records of a second data source. Quantities characterizing a relationship between the first field and the second field are computed based on the accepted information. Information relating the first field and the second field is presented.

Abstract: Processing data includes identifying a plurality of subsets of fields of data records of a data source, determining co-occurrence statistics for each of the plurality of subsets, and identifying one or more of the plurality of subsets as having a functional relationship among the fields of the identified subset.

Abstract: A method for managing data includes receiving individually accessible data units, each identified by a key value; storing a plurality of blocks of data, each of at least some of the blocks being generated by combining a plurality of the data units; and providing an index that includes an entry for each of the blocks. One or more of the entries enable location, based on a provided key value, of a block that includes data units corresponding to a range of key values that includes the provided key value.

Abstract: Processing data includes accepting information characterizing values of a first field in records of a first data source and information characterizing values of a second field in records of a second data source. Quantities characterizing a relationship between the first field and the second field are computed based on the accepted information. Information relating the first field and the second field is presented.

Abstract: Processing data includes profiling data from a data source, including reading the data from the data source, computing summary data characterizing the data while reading the data, and storing profile information that is based on the summary data. The data is then processed from the data source. This processing includes accessing the stored profile information and processing the data according to the accessed profile information.