Abstract: Computer-implemented techniques for data extraction are described. The techniques include a method and system for retrieving an extraction job specification, wherein the extraction job specification comprises a source repository identifier that identifies a source repository comprising a plurality of data records; a data recipient identifier that identifies a data recipient; and a schedule that indicates a timing of when to retrieve the plurality of data records. The method and system further include retrieving the plurality of data records from the source repository based on the schedule, creating an extraction transaction from the plurality of data records, wherein the extraction transaction comprises a subset of the plurality of data records and metadata, and sending the extraction transaction to the data recipient.

Abstract: A computer-implemented method enforces data security constraints in a data pipeline. The data pipeline takes one or more source datasets as input and performs one or more data transformations on them. The method includes using data defining one or more data security constraints to configure the data pipeline to perform a data transformation on a restricted subset of entries of the source datasets. The restriction is defined by the data defining one or more data security constraints. The method further includes performing the data transformation according to the configuration to produce one or more transformed datasets. The method further includes using the data defining one or more data security constraints to perform a verification on one or more of the transformed datasets to ensure that entries in the one or more of the transformed datasets are restricted as defined by the one or more data security constraints.

Abstract: A method, performed by one or more processors, is disclosed, comprising providing, to a plurality of parties permitted to communicate data via a shared database, an ontology application associated with a common core ontology, the core ontology defining constraints required to be met for producing, from one or more received datasets, one or more data objects for storing in the shared database. The ontology application may be configured to receive one or more datasets from one or more parties and to use the core database ontology to determine if the received one or more datasets conform to the constraints of the core ontology, and store the received one or more datasets as data objects in the shared database, conditional on the constraints being met.

Abstract: Discussed herein are embodiments of methods and systems which allow engineers or administrators to create modular plugins which represent the logic for various fault detection tests that can be performed on data pipelines and shared among different software deployments. In some cases, the modular plugins each define a particular test to be executed against data received from the pipeline in addition to one or more configuration points. The configuration points represent configurable arguments, such as variables and/or functions, referenced by the instructions which implement the tests and that can be set according to the specific operation environment of the monitored pipeline.

Abstract: Computer-implemented techniques for data extraction are described. The techniques include a method and system for retrieving an extraction job specification, wherein the extraction job specification comprises a source repository identifier that identifies a source repository comprising a plurality of data records; a data recipient identifier that identifies a data recipient; and a schedule that indicates a timing of when to retrieve the plurality of data records. The method and system further include retrieving the plurality of data records from the source repository based on the schedule, creating an extraction transaction from the plurality of data records, wherein the extraction transaction comprises a subset of the plurality of data records and metadata, and sending the extraction transaction to the data recipient.

Abstract: Computer-implemented techniques for data extraction are described. The techniques include a method and system for retrieving an extraction job specification, wherein the extraction job specification has a source repository identifier that identifies a source repository including a plurality of data records; a data recipient identifier that identifies a data recipient; and a schedule that indicates a timing of when to retrieve the plurality of data records. The method and system further include retrieving the plurality of data records from the source repository based on the schedule, creating an extraction transaction from the plurality of data records, wherein the extraction transaction includes a subset of the plurality of data records and metadata, and sending the extraction transaction to the data recipient.

Abstract: Discussed herein are embodiments of methods and systems which allow engineers or administrators to create modular plugins which represent the logic for various fault detection tests that can be performed on data pipelines and shared among different software deployments. In some cases, the modular plugins each define a particular test to be executed against data received from the pipeline in addition to one or more configuration points. The configuration points represent configurable arguments, such as variables and/or functions, referenced by the instructions which implement the tests and that can be set according to the specific operation environment of the monitored pipeline.

Abstract: Discussed herein are embodiments of methods and systems which allow engineers or administrators to create modular plugins which represent the logic for various fault detection tests that can be performed on data pipelines and shared among different software deployments. In some cases, the modular plugins each define a particular test to be executed against data received from the pipeline in addition to one or more configuration points. The configuration points represent configurable arguments, such as variables and/or functions, referenced by the instructions which implement the tests and that can be set according to the specific operation environment of the monitored pipeline.

Abstract: Computer-implemented techniques for data extraction are described. The techniques include a method and system for retrieving an extraction job specification, wherein the extraction job specification comprises a source repository identifier that identifies a source repository comprising a plurality of data records; a data recipient identifier that identifies a data recipient; and a schedule that indicates a timing of when to retrieve the plurality of data records. The method and system further include retrieving the plurality of data records from the source repository based on the schedule, creating an extraction transaction from the plurality of data records, wherein the extraction transaction comprises a subset of the plurality of data records and metadata, and sending the extraction transaction to the data recipient.

Abstract: Computer-implemented techniques for data extraction are described. The techniques include a method and system for retrieving an extraction job specification, wherein the extraction job specification comprises a source repository identifier that identifies a source repository comprising a plurality of data records; a data recipient identifier that identifies a data recipient; and a schedule that indicates a timing of when to retrieve the plurality of data records. The method and system further include retrieving the plurality of data records from the source repository based on the schedule, creating an extraction transaction from the plurality of data records, wherein the extraction transaction comprises a subset of the plurality of data records and metadata, and sending the extraction transaction to the data recipient.

Abstract: Computer-implemented techniques for data extraction are described. The techniques include a method and system for retrieving an extraction job specification, wherein the extraction job specification comprises a source repository identifier that identifies a source repository comprising a plurality of data records; a data recipient identifier that identifies a data recipient; and a schedule that indicates a timing of when to retrieve the plurality of data records. The method and system further include retrieving the plurality of data records from the source repository based on the schedule, creating an extraction transaction from the plurality of data records, wherein the extraction transaction comprises a subset of the plurality of data records and metadata, and sending the extraction transaction to the data recipient.

Abstract: Discussed herein are embodiments of methods and systems which allow engineers or administrators to create modular plugins which represent the logic for various fault detection tests that can be performed on data pipelines and shared among different software deployments. In some cases, the modular plugins each define a particular test to be executed against data received from the pipeline in addition to one or more configuration points. The configuration points represent configurable arguments, such as variables and/or functions, referenced by the instructions which implement the tests and that can be set according to the specific operation environment of the monitored pipeline.

Abstract: Discussed herein are embodiments of methods and systems which allow engineers or administrators to create modular plugins which represent the logic for various fault detection tests that can be performed on data pipelines and shared among different software deployments. In some cases, the modular plugins each define a particular test to be executed against data received from the pipeline in addition to one or more configuration points. The configuration points represent configurable arguments, such as variables and/or functions, referenced by the instructions which implement the tests and that can be set according to the specific operation environment of the monitored pipeline.

Abstract: Discussed herein are embodiments of methods and systems which allow engineers or administrators to create modular plugins which represent the logic for various fault detection tests that can be performed on data pipelines and shared among different software deployments. In some cases, the modular plugins each define a particular test to be executed against data received from the pipeline in addition to one or more configuration points. The configuration points represent configurable arguments, such as variables and/or functions, referenced by the instructions which implement the tests and that can be set according to the specific operation environment of the monitored pipeline.