Abstract: A computer-implemented method for managing data objects stored in a data repository and displaying elements associated with data objects on a graphical user interface is presented. The method includes using reusable logic to provide at least one displayed input element associated with a first data object stored in the data repository. In response to user input associated with the displayed input element, the method includes generating a second data object stored in the data repository, and linking the second data object with one or more related data objects in the data repository automatically based on the data object associated with the displayed element.

Abstract: A system for managing storage of data units includes a data storage system configured to store multiple data blocks, at least some of the data blocks containing multiple data units, and configured to store, for at least some of the data blocks, corresponding historical information about prior removal of one or more data units from that data block, the removal affecting at least some addresses of data units in that data block. The system is configured to perform at least one operation that accesses at least a first data unit stored in a first data block according to address information interpreted based on any stored historical information corresponding to the first data block.

Abstract: Information representative of a graph-based program specification has components, and directed links between ports of said components, defining a dependency between said components. A directed link exists between a port of a first component and a port of a second component. The first component specifies first-component execution code that when compiled enables execution of a first task. The second component specifies second-component execution code that when compiled enables execution of a second task. Compiling the graph-based program specification includes grafting first control code to said first-component execution code, which changes a state of said second component to a pending state, an active state, or a suppressed state. Based on said state, said first control code causes at least one of: invoking said second component if said state changes from pending to active, or suppressing said second component if said state changes from pending to suppressed.

Abstract: A first component of a graph-based program specification includes an output control port. A second component includes an input control port and an input data port. A third component includes an output data port. The output control port is connected to the input control port, and the output data port is connected to the input data port. The first component includes control code that when executed causes the output control port to provide, to the input control port, at least one of suppression information or invocation information. The second component includes control code that when executed causes a computing system configured by the graph-based program specification to begin processing data received at the input data port in response to the invocation information if no suppression information is received at the input control port before the invocation information is received at the input control port.

Abstract: This specification describes technologies relating to generating transforms based on rule sets. In general, one aspect described in this specification can be embodied in methods that include receiving a rule set including execution cases, where at least one execution case in the rule set includes one or more trigger conditions and a specification of an output that is to be generated when the one or more trigger conditions are all satisfied. The methods may further include generating a control structure including a sequence of rows corresponding to one or more execution cases in the rule set. Each row may include a sequence of one or more trigger conditions and information specifying the output for a corresponding execution case. For at least one of the trigger conditions, when the trigger condition is failed, the control structure may direct processing to skip at least one row in the sequence of rows.

Abstract: Presenting a diagram indicating relationships among data items stored in a data management system includes: receiving a request that identifies a first data item stored in the data management system from a user interface; retrieving stored configuration information that includes a plurality of selection specifications for selecting data items in the data management system that are related to a given data item of a predetermined type, where each selection specification is associated with a different respective predetermined type; querying the data management system to identify a set of one or more data items according to a selection specification from the configuration information that is associated with a type of the first data item; for each of multiple returned data items in the identified set, querying the data management system to determine whether additional data items are identified according to a selection specification from the configuration information that is associated with a type of the returned d

Abstract: A computer-implemented method includes accessing a plurality of data records, each data record having a plurality of data fields. The method further includes analyzing values for one or more of the data fields for at least some of the plurality of data records and generating a profile of the plurality of data records based on the analyzing. The method further includes formulating at least one subsetting rule based on the profile; and selecting a subset of data records from the plurality of data records based on the at least one subsetting rule.

Abstract: Software specification translation includes: receiving a first software specification specified in a first programming language, a second software specification specified in a second programming language, a third software specification specified in a third programming language, the third software specification defining one or more data relationships between the first software specification and the second software specification. A combined representation of the first software specification and the second software specification is formed in a fourth programming language different from the first, second, and third programming languages. Connections are formed in the fourth programming language between a representation of the first software specification and the representation of the second software specification according to identified data relationships.

Abstract: Specifying a dataflow graph includes: rendering, in a first user interface, a representation of a first dataflow graph. At least one component represents a computation associated with at least one of data flowing into an input port or data flowing out of an output port. A link between components is rendered based on user input indicating a flow of data. In a second user interface, a representation of a sub-graph of the first dataflow graph is rendered. A first flow junction represents a connection between: (1) a flow of data outside the sub-graph interface, and (2) a flow of data inside the sub-graph interface. A link is rendered between the first flow junction and a second flow junction based on user input indicating a relationship between a first descriptor and a second descriptor.

Abstract: Managing task execution includes: receiving a specification of a plurality of tasks to be performed by respective functional modules; processing a flow of input data using a dataflow graph that includes nodes representing data processing components connected by links representing flows of data between data processing components; in response to at least one flow of data provided by at least one data processing component, generating a flow of messages; and in response to each of the messages in the flow of messages, performing an iteration of a set of one or more tasks using one or more corresponding functional modules.

Abstract: A method includes analyzing, by a processor, a first version of a computer program. The analyzing includes identifying a first data processing element included in the first version of the computer program. The first data processing element references a first data source external to the first version of the computer program. The method includes generating a data source element that represents a second data source different from the first data source. The method includes generating a second version of the computer program. The second version of the computer program includes the generated data source element and a second data processing element that is based on the first data processing element. In the second version of the computer program, the second data processing element references the generated data source element.

Abstract: In general, metadata is stored in a data storage system. Summary data identifying one or more characteristics of each of multiple metadata objects stored in the data storage system is computed, and the summary data characterizing a given metadata object in association with the given metadata object is stored. A visual representation is generated of a diagram including nodes representing respective metadata objects and relationships among the nodes. Generating the visual representation includes superimposing a representation of a characteristic identified by the summary data characterizing a given metadata object in proximity to the node representing the given metadata object.

Abstract: A system for managing storage of data units includes a data storage system configured to store multiple data blocks, at least some of the data blocks containing multiple data units, with at least a group of the data blocks being stored contiguously, thereby supporting a first read operation that retrieves data units from at least two adjacent data blocks in the group. The system is configured to perform one or more operations with respect to data units, the operations including a delete operation that replaces a first data block containing a data unit to be deleted with a second data block that does not contain the deleted data unit, with the second data block having the same size as the first data block.

Abstract: A computing system for representing information, the computing system including at least one processor configured to process information. The processing includes defining a data structure representing a hierarchy of at least one programming attribute for developing an application. The data structure is stored in a file to allow the data structure to be used by other data structures stored in other files. The processing also includes producing a visual diagram including a graphical representation of the data structure and a graphical representation of the file storing the data structure. The visual diagram also includes a graphical representation of a relationship between the data structure and another data structure and a graphical representation of a relationship between the file storing the data structure and another file storing the other data structure.

Abstract: Among other things, a method includes, at a computer system on which one or more computer programs are executing, receiving a specification defining types of state information, receiving an indication that an event associated with at least one of the computer programs has occurred, the event associated with execution of a function of the computer program, collecting state information describing the state of the execution of the computer program when the event occurred, generating an entry corresponding to the event, the entry including elements of the collected state information, the elements of state information formatted according to the specification, and storing the entry. The log can be parsed to generate a visualization of computer program execution.

Abstract: In some aspects, a method includes connecting over a network to a data storage system, the data storage system storing data objects. A dataflow graph that includes nodes representing data processing components connected by links that represent flows of data access an interface of the data storage system. The interface provides functions for accessing the data objects. At least one of the data processing components performs operations on a received input flow of data that enable the functions provided by the interface to modify one or more stored data objects, and performs operations in response to functions provided by the interface to generate an output flow of data.

Abstract: A graph-based program specification includes: a plurality of components, each corresponding to a processing task and including one or more ports, including scalar data ports for sending or receiving a single data element and collection data ports for sending or receiving a collection of multiple data elements; and one or more links, each connecting an output port of an upstream component to an input port of a downstream component. Prepared code is generated representing subsets of the plurality of components, including: identifying one or more subset boundaries, including identifying one or more links connecting a collection data port of a component to a scalar data port of a component; forming the subsets based on the identified subset boundaries; and generating prepared code for each formed subset that when used for execution by a runtime system causes processing tasks corresponding to the components in each formed subset to be performed.

Abstract: In general, a value of a numerical attribute of a record stored in a data structure is received. A numerical range is generated that includes the value of the numerical attribute. An entry is stored, in an index associated with the data structure, that specifies a location of the record within the data structure and that includes a first index key and a second index key. The first index key corresponds to a value of an attribute of the record different from the numerical attribute, and the second index key corresponds to the generated numerical range.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for specifying a user interface. One of the methods includes receiving an identification of at least one configurable application, each configurable application being configurable by one or more parameters. The method includes identifying a set of parameters associated with the at least one configurable application. The method includes displaying, in a second user interface, a graphical display element, the graphical display element having properties and associated with a location relative to the first user interface. The method includes displaying, in the second user interface, one or more graphical representations of the set of parameters. The method includes receiving a mapping between a property of the graphical display element and a parameter of the set of parameters. The method also includes storing a representation of the mapping and the location.

Abstract: A graph-based program specification includes components corresponding to tasks and directed links between ports of the components, including: a first type of link configuration defined by respective output and input ports of linked components, and a second type of link configuration defined by respective output and input ports of linked components. A compiler recognizes different types of link configurations and provides in a target program specification occurrences of a target primitive for executing a function for each occurrence of a data element flowing over a link of the second type. A computing node initiates execution of the target program specification, and determines at runtime, for components associated with the occurrences of the target primitive, an order in which instances of tasks corresponding to the components are to be invoked, and/or a computing node on which instances of tasks corresponding to the components are to be executed.