Abstract: A system, method and computer-readable medium format-preserving encryption of a numerical value, including storing a binary numerical value, the binary numerical value comprising a plurality of binary bits, dividing the plurality of binary bits into a plurality of bit groups and storing the plurality of bit groups in a plurality of bytes, encrypting each byte in the plurality of bytes using a radix value corresponding to a quantity of binary bits in a bit group corresponding to that byte to generate a plurality of ciphertext bytes, and combining a quantity of least-significant bits from each ciphertext byte in the plurality of ciphertext bytes to generate a binary ciphertext value, the quantity of least-significant bits combined from each ciphertext byte corresponding to the radix value used to generate that ciphertext byte.

Abstract: A method, apparatus, and computer-readable medium for dynamic hierarchical data flow mapping, including storing dynamic data flow mappings, each dynamic data flow mapping including a mapping of data from an input port corresponding to a source data container to an output port corresponding to a destination data container and a dynamic hierarchical field having a dynamic hierarchical data type, determining a rule language corresponding to each dynamic hierarchical field in the dynamic data flow mappings, the rule language defining acceptable parameters for sub-fields of that dynamic hierarchical data type, and generating static data flow mappings by resolving all dynamic hierarchical fields into static hierarchical fields during compilation of the dynamic data flow mappings with the source data container and the destination data container, the static data flow mappings being generated based at least in part on the rule language corresponding to each dynamic hierarchical field and underlying data in one or more

Abstract: Systems, methods, and techniques for optimizing a plurality of data integration tasks within a data integration collection by identifying, as a sub-set of the plurality of data integration tasks, a plurality of point-to-point data integration tasks defining a data integration transformation plan to include: generating one or more publication data integration tasks comprising publishing from each respective data source of the plurality of point-to-point data integration tasks to generate a single publication topic; and generating one or more subscription data integration tasks causing each respective target of the plurality of point-to-point data integration tasks to subscribe to the single publication topic; and generating a set of optimization instructions configured to cause the at least one computer to implement the data integration transformation plan; and executing the set of optimization instructions to generate the one or more publication data integration tasks and the one or more subscription tasks.

Abstract: Systems, methods, and related techniques and apparatus containing instructions which when executed by one or more computing devices for determining dataset rankings by determining a lineage ordering requirement for a collection of datasets; determining, from the lineage order requirement, one or more first lineage level datasets from the collection of datasets; generating one or more first lineage level asset ranks respectively for each one of the one or more first lineage level datasets, determining at least one second lineage level dataset having an outflow to the one or more first lineage level datasets; and generating a first dataset rank for the at least one second lineage level dataset as a first function of the outflow and at least one of the one or more first lineage level asset ranks.

Abstract: An apparatus, computer-readable medium, and computer-implemented method for generating a data proliferation graph, including receiving a selection of a target data store, identifying a plurality of data stores which have either received data that was previously on the target data store or which have sent data that was subsequently on the target data store, the plurality of data stores being divided into a plurality of proliferation levels corresponding to degrees of separation from the target data store and direction of data propagation relative to the target data store, generating a data proliferation graph, and transmitting at least one portion of the data proliferation graph.

Abstract: Apparatus, computer-readable medium, and computer-implemented method for dynamic binding of tasks in a data exchange, including generating an execution plan for the data exchange comprising a plurality of streams, each stream corresponding to a pairing of a producer task and a consumer task and comprising a producer task identifier and a consumer task identifier, binding each producer task to a node at execution time by associating a corresponding producer task identifier with a corresponding node identifier of the node, binding each consumer task to a node at execution time by associating a corresponding consumer task identifier with a node identifier of the corresponding node, and routing an output of each producer task from a node bound to the producer task to a node bound to a paired consumer task by writing the output to a stream corresponding to the pairing of the producer task and the consumer task.

Abstract: A system and a method are disclosed for processing hierarchical data structures, for example, XML (extensible markup language) documents. A tree view shows a visual representation of the input schema and the output schema and visual representations of associations between nodes from the input schema and output schema. A grid view shows mapping statements configured to map the input hierarchical data structure conforming to the input schema to an output hierarchical data structure conforming to the output schema. The grid view and the tree view are synchronized such that modifications performed using any one view result in corresponding modifications being presented in the other view. Modifications to mapping statements are validated to determine if path expressions used in the mapping statements are valid in the modified mapping statements.

Abstract: An apparatus, computer-readable medium, and computer-implemented method for visualizing relationships between pairs of columns, comprising identifying a relationship classification corresponding to two columns in a plurality of columns based on a data type of each column in the two columns, applying one or more statistical measures to data in the two columns to generate association data quantifying a plurality of relationships between data values in a first column of the two columns and data values in a second column of the two columns, wherein the one or more statistical measures are determined based at least in part on the relationship classification, and transforming the association data into a visualization, wherein the visualization comprises one or more indicators corresponding to one or more relationships in the plurality of relationships and wherein a layout of the visualization is determined based on the relationship classification.

Abstract: A system, method and computer-readable medium for generating a data masking syntactic definition for a data element of an unknown data type, including generating one or more alphabets corresponding to one or more element member positions of the data element based at least in part on element members occurring at each element member position in a plurality of data elements of the unknown type, each alphabet comprising a set of one or more sequential element members that have occurred in the plurality of data elements at an element member position and generating a positional map describing a syntactic structure of the data element by mapping at least one of the one or more alphabets to each element member position of the data element.

Abstract: A method, apparatus, and computer-readable medium for searching polymorphically encrypted data includes generating one or more pseudonymous tokens by encrypting a ciphertext using a first algorithm and an encryption key, the first algorithm comprising a polymorphic algorithm configured to generate a distinct pseudonymous token for each application of the polymorphic algorithm to the same plaintext, storing, the one or more pseudonymous tokens in one or more data stores, and identifying data in the one or more data stores that corresponds to the ciphertext by querying the data store using a search token generated by encrypting the plaintext using a second algorithm and the encryption key, the search token being distinct from the one or more pseudonymous tokens.

Abstract: A message broker computer includes a master broker, a plurality of slave message brokers and event stores. A client system sends messages for processing to the master broker. The master broker generates a message event in response to receiving such a message, and distributes the message event in parallel to the slave brokers and the event stores. Each of the event stores store the message event in persistent storage, and notifies the master broker that the message event has been persisted. The master broker considers the message stabilized n a quorum of the event stores. As the master broker does not take action until a messaging event is stabilized, in the event of failover, a new master broker is able to re-construct a broker state of the old master with no loss of data.

Abstract: A system, method and computer-readable medium for generation of missing data including transmitting indicators corresponding to locations of missing data in columns in tables of a database, each location of missing data corresponding to categorical values of categorical columns and each location of missing data being identified based on an expected count of data values at the corresponding location, receiving a selection of at least one indicator corresponding to at least one location of missing data, the at least one location of missing data corresponding to two or more categorical values of two or more categorical columns in the categorical columns, and generating sets of data records in at least one table in the tables of the database, each set of data records having two or more column values in two or more columns that correspond to the two or more categorical values of the two or more categorical columns.

Abstract: A system, method and computer-readable medium for data breach simulation and impact analysis in a computer network, including generating an exposure matrix corresponding to data stores connected to the computer network, the exposure matrix storing a correspondence between regulations and an exposure of each data store the regulations, generating approximation matrices from the exposure matrix by iteratively applying a data compression algorithm to the exposure matrix, each successive iteration of the data compression algorithm being configured to more closely approximate the exposure matrix than a previous iteration of the data compression algorithm, simulating a plurality of data breach and mitigation scenarios on the plurality of data stores based at least in part on the plurality of approximation matrices to determine an overall impact of each data breach and mitigation scenario, determining an optimal data breach and mitigation scenario in the plurality of data breach and mitigation scenarios.

Abstract: A system, method and computer-readable medium for data protection simulation and optimization in a computer network, including grouping data stored in data stores in the computer network into groupings according to an architectural or a conceptual attributes, storing, current values of risk metrics for each grouping, each of the metrics corresponding to sensitive domains, receiving a risk reduction goal corresponding to at least one risk metric in the risk metrics, the at least one risk metric corresponding to at least one sensitive domain in the sensitive domains, determining a simulated value of the at least one risk metric for each grouping in the groupings by simulating application of a protection mechanism to sensitive data in each corresponding data store, the sensitive data corresponding to the at least one sensitive domain, and ranking the groupings based on the at least one simulated value of the at least one risk metric for each grouping.

Abstract: A system, a method, and a non-transitory computer readable for generating a profile of one or more data objects comprising determining a format of the at least one data object and selecting a data transformation policy based on the format of the at least one data object and generating a model descriptive of the non-structured data contained in the at least one data object based on the data transformation policy and selecting at least a portion of the model indicative of a portion of the non-structured data and generating a profile of the portion of the non-structured data contained in all of the one or more data objects.

Abstract: A data profiling server extracts data from multiple data sources and stores the extracted data in a target data store. Validation functions are performed on the extracted to determine whether the data conforms to validation rules associated with the target data store and/or business logic definitions. For data that does not conform to the validation rules, the data profiling server identifies recommended transformations to be applied to the data to correct any errors that cause the data to not conform with the validation rules. A data profiling application presents to a user the data extracted from the multiple data sources in a grid structure. The grid structure also includes error indicators identifying the data that does not conform to the validation rules and specifies the recommended transformations identified by the data profiling server.

Abstract: A data flow design system is presented that includes support for both static elements and dynamic elements. Thus, the data flow design system provides a design environment graphical tool to users to design data flows that leverage both the repeatability of static elements and the adaptability of dynamic elements. Static elements process data that typically do not change over time, while dynamic elements process data that do change. For instance, new data fields are added to an input data source of a data flow. The dynamic elements automatically link the new data fields to mapping fragments of the data flow. Mapping fragments process data based on configuration parameters including expression and filter rules. Users use a user interface of the design environment to view and add mapping fragments, static links, and dynamic links to the data flow.

Abstract: A system and a method are disclosed for exporting a subset of a database from a source system to a target system. The database system identifies a first set of tables and determines the subset of data for these tables for transfer by executing one or more database queries. For example, the first set of tables is selected based on their sizes. The database system identifies a second set of tables for transferring by performing a file copy operation. For example, the remaining set of tables other than the first set of tables is transferred by performing a file copy operation. The database system moves the tables of the first set to a new table space for separating the files storing the first set of tables from the files storing the second set of tables.

Abstract: An apparatus, computer-readable medium, and computer-implemented method for performing a data exchange, including receiving tasks for execution, generating an execution plan for executing the tasks on a plurality of nodes, the execution plan comprising one or more data exchanges, each data exchange comprising at least one stream, and each stream identifying a producer task and a consumer task and being configured to transmit output of the producer task as input to the consumer task, executing one or more producer tasks on one or more first nodes in the plurality of nodes based at least in part on the execution plan, and transmitting an output of the one or more producer tasks from the one or more first nodes to one or more streams of the data exchange via a stream application programming interface (API).

Abstract: An apparatus, apparatus and computer-readable medium for generating an alternative implementation of a program on one or more engines, including receiving a job graph comprising a sequence of jobs corresponding to the program, translating each job into a set of candidate translations corresponding to that job based on one or more available engines, storing each set of candidate translations corresponding to each job in the job graph in a region of a job hypergraph, condensing the job hypergraph into an alternative job graph by iteratively replacing each pair of connected regions with a merged region comprising a merged candidate translation generated by combining a pair of candidate translations in that pair of connected regions, and generating the alternative implementation of the program based at least in part on the alternative job graph.