Abstract: A system for managing storage of data units includes a data storage system configured to store multiple data blocks, at least some of the data blocks containing multiple data units, with at least a group of the data blocks being stored contiguously, thereby supporting a first read operation that retrieves data units from at least two adjacent data blocks in the group. The system is configured to perform one or more operations with respect to data units, the operations including a delete operation that replaces a first data block containing a data unit to be deleted with a second data block that does not contain the deleted data unit, with the second data block having the same size as the first data block.

Abstract: Mapping one or more elements of an input dataset to one or more elements of an output dataset includes: receiving in an interface one or more mapped relationships between a given output and one or more inputs represented by input variables, at least one of the mapped relationships including a transformational expression executable on a data processing system, the transformational expression defining an output of a mapped relationship based on at least one input variable mapped to an element of an input dataset; receiving in the interface identification of elements of an output dataset mapped to outputs of respective mapped relationships; generating output data from the data processing system according to the transformational expression based on input data from the input dataset associated with the element of the input dataset mapped to the input variable; determining validation information in response to the generated output data based on validation criteria defining one or more characteristics of valid value

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for specifying a user interface. One of the methods includes receiving an identification of at least one configurable application, each configurable application being configurable by one or more parameters. The method includes identifying a set of parameters associated with the at least one configurable application. The method includes displaying, in a second user interface, a graphical display element, the graphical display element having properties and associated with a location relative to the first user interface. The method includes displaying, in the second user interface, one or more graphical representations of the set of parameters. The method includes receiving a mapping between a property of the graphical display element and a parameter of the set of parameters. The method also includes storing a representation of the mapping and the location.

Abstract: Controlling access to computational features includes: preparing a computational resource for execution by an execution system that has been provided a primary descriptor containing an identity value and that has associated a feature indicator with the primary descriptor; accessing a secondary descriptor containing the identity value and cryptographically assigned to the computational resource; and granting the computational resource access to a computational feature of the execution system based on the feature indicator.

Abstract: In one aspect, in general, a method is described for managing an archive. The archive is used for determining approximate matches associated with strings occurring in records. The method includes processing records to determine a set of string representations that correspond to strings occurring in the records. The method also includes generating, for each of at least some of the string representations in the set, a plurality of close representations that are each generated from at least some of the same characters in the string. The method also includes storing entries in the archive. Each stored entry represents a potential approximate match between at least two strings based on their respective close representations.

Abstract: Processing related datasets includes receiving over an input device or port records from multiple datasets, the records of a given dataset having one or more values for one or more respective fields; and processing records from each of the multiple datasets in a data processing system.

Abstract: Configurable testing of a computer program includes: storing a set of one or more testing specifications, and attribute information defining one or more attributes of a recognizable portion of the computer program; and processing, using at least one processor, the computer program according to at least a first testing specification associated with the computer program. The processing includes: traversing a representation of the computer program that includes elements that represent recognizable portions of the computer program, and while traversing the representation, recognizing recognizable portions of the computer program, and storing values of attributes, defined by the attribute information, of the recognized portions of the computer program.

Abstract: Systems and techniques include, in some implementations, a computer implemented method storing a portion of data elements present in a first buffer in a second buffer in response to detecting an overflow condition of the first buffer, wherein the data elements in the first buffer are sorted according to a predetermined order, and inserting a proxy data element in the first buffer to represent the portion of data elements stored to the second buffer.

Abstract: A method for supporting communication between a client and a server includes receiving a first message from a client. The method also includes creating an object in response to the first message. The method also includes sending a response to the first message to the client. The method also includes receiving changes to the object from a server. The method also includes storing the changes to the object. The method also includes receiving a second message from the client. The method also includes sending the stored changes to the client with a response to the second message.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for dynamic graph performance monitoring. One of the methods includes receiving input data by the data processing system, the input data provided by an application executing on the data processing system. The method includes determining a characteristic of the input data. The method includes identifying, by the application, a dynamic component from multiple available dynamic components based on the determined characteristic, the multiple available dynamic components being stored in a data storage system. The method includes processing the input data using the identified dynamic component. The method also includes determining one or more performance metrics associated with the processing.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for dynamic graph performance monitoring. One of the methods includes receiving multiple units of work that each include one or more work elements. The method includes determining a characteristic of the first unit of work. The method includes identifying, by a component of the first dataflow graph, a second dataflow graph from multiple available dataflow graphs based on the determined characteristic, the multiple available dataflow graphs being stored in a data storage system. The method includes processing the first unit of work using the second dataflow graph. The method includes determining one or more performance metrics associated with the processing.

Abstract: Profiling data includes processing an accessed collection of records, including: generating, for a first set of distinct values appearing in a first set of one or more fields, corresponding location information; generating, for the first set of fields, a corresponding list of entries identifying a distinct value from the first set of distinct values and the location information for the distinct value; generating, for a second set of one or more fields, a corresponding list of entries, with each entry identifying a distinct value from a second set of distinct values appearing in the second set of fields; and generating result information, based at least in part on: locating at least one record of the collection using the location information for at least one value appearing in the first set of fields, and determining at least one value appearing in the second set of fields of the located record.

Abstract: Processing transactions using graph-based computations includes determining that at least one of a plurality of graph elements of a computation graph of a set of one or more computation graphs includes a computation to be performed for a given transaction, associating the given transaction with an instance of the computation graph that includes reusable computation elements associated with respective graph elements, and executing the graph to perform the computation.

Abstract: A graph-based computation has a plurality of nodes representing graph components connected by one or more links representing data flows between the components. Preparing the computation includes: identifying at least a first set of components to process a plurality of transactions in the computation; and associating with a first transaction a first group of one or more data records and data operations corresponding to the data records, and associating with a second transaction a second group of one or more data records and data operations corresponding to the data records. Executing the computation to process at least the first and second transactions includes delaying execution by the first set of components of a data operation corresponding to an initial data record of the second group until after execution by the first set of components of a data operation corresponding to a final data record of the first group.

Abstract: A method for supporting communication between a client and a server includes receiving a first message from a client. The method also includes creating an object in response to the first message. The method also includes sending a response to the first message to the client. The method also includes receiving changes to the object from a server. The method also includes storing the changes to the object. The method also includes receiving a second message from the client. The method also includes sending the stored changes to the client with a response to the second message.

Abstract: Among other aspects disclosed are a method and system for processing a batch of input data in a fault tolerant manner. The method includes reading a batch of input data including a plurality of records from one or more data sources and passing the batch through a dataflow graph. The dataflow graph includes two or more nodes representing components connected by links representing flows of data between the components. At least one but fewer than all of the components includes a checkpoint process for an action performed for each of multiple units of work associated with one or more of the records. The checkpoint process includes opening a checkpoint buffer stored in non-volatile memory at the start of processing for the batch.

Abstract: A method for managing data includes receiving individually accessible data units, each identified by a key value; storing a plurality of blocks of data, each of at least some of the blocks being generated by combining a plurality of the data units; and providing an index that includes an entry for each of the blocks. One or more of the entries enable location, based on a provided key value, of a block that includes data units corresponding to a range of key values that includes the provided key value.

Abstract: Among other things, one aspect includes receiving one or more mapped relationships between a given output and one or more inputs represented by input variables, at least one of the mapped relationships including a transformational expression, the transformational expression defining an output of a mapped relationship based on at least one input variable mapped to an element of an input dataset; receiving identification of elements of an output dataset mapped to outputs of respective mapped relationships; generating output data according to the transformational expression based on input data from the input dataset associated with the element of the input dataset mapped to the input variable; determining validation information in response to the generated output data based on validation criteria defining one or more characteristics of valid values associated with one or more of the identified elements of the output dataset; and presenting visual feedback based on the determined validation information.

Abstract: A service request is processed according to a computation graph associated with the service by receiving inputs for the computation graph from a service client, providing the inputs to the computation graph as records of a data flow, receiving output from the computation graph, and providing the output to the service client. Data flows are processed concurrently in a graph-based computation by potentially concurrent execution of different types of requests, potentially concurrent execution of similar request types, and/or potentially concurrent execution of work elements within a request.

Abstract: Among other aspects disclosed are a method and system for processing a batch of input data in a fault tolerant manner. The method includes reading a batch of input data including a plurality of records from one or more data sources and passing the batch through a dataflow graph. The dataflow graph includes two or more nodes representing components connected by links representing flows of data between the components. At least one but fewer than all of the components includes a checkpoint process for an action performed for each of multiple units of work associated with one or more of the records. The checkpoint process includes opening a checkpoint buffer stored in non-volatile memory at the start of processing for the batch.