Abstract: Techniques for propagation of deletion operations among a plurality of related datasets are described herein.

Abstract: A method and system for serving assets is disclosed, comprising receiving an asset request to serve an asset, wherein the asset request originates at an application, and wherein the asset request comprises an advertisement of an asset to be served and a request for the network address of an asset server configured to serve the requested asset. The method further comprises causing a service discovery server to identify an asset server configured to serve the requested asset, and causing the requested asset to be served to the application.

Abstract: A method and system for serving assets is disclosed, comprising receiving an asset request to serve an asset, wherein the asset request originates at an application, and wherein the asset request comprises an advertisement of an asset to be served and a request for the network address of an asset server configured to serve the requested asset. The method further comprises causing a service discovery server to identify an asset server configured to serve the requested asset, and causing the requested asset to be served to the application.

Abstract: Discussed herein are embodiments of methods and systems which allow engineers or administrators to create modular plugins which represent the logic for various fault detection tests that can be performed on data pipelines and shared among different software deployments. In some cases, the modular plugins each define a particular test to be executed against data received from the pipeline in addition to one or more configuration points. The configuration points represent configurable arguments, such as variables and/or functions, referenced by the instructions which implement the tests and that can be set according to the specific operation environment of the monitored pipeline.

Abstract: A system comprises an interface and a processor. The interface is configured to provide a request to join a publish group from a client or a point to point communication link of a lattice mesh; and receive a group key or a host public key. The processor is configured to determine whether a message has been received; in response to the message having been received, determine whether the message is to be sent on; in response to the message being determined not to be sent on, decode the message using the group key or the host public key; determine whether to store the message in a backfill database; in response to determining to store the message in the backfill database, store the message in the backfill database.

Abstract: Data transformation in a distributed system of applications and data repositories is described. The subsystems for the overall framework are distributed, thereby allowing for customization to require only isolated changes to one or more subsystems. In one embodiment, a source code repository is used to receive and store source code. A build subsystem can retrieve source code from the source code repository and build it, using one or more criteria. By building the source code, the build subsystem can generate an artifact, which is executable code, such as a JAR or SQL file. Likewise, by building the source code, the build subsystem can generate one or more job specifications for executing the executable code. In one embodiment, the artifact and job specification may be used to launch an application server in a cluster. The application server can then receive data transformation instructions and execute the data transformation instructions.

Abstract: Computer-implemented techniques for data extraction are described. The techniques include a method and system for retrieving an extraction job specification, wherein the extraction job specification comprises a source repository identifier that identifies a source repository comprising a plurality of data records; a data recipient identifier that identifies a data recipient; and a schedule that indicates a timing of when to retrieve the plurality of data records. The method and system further include retrieving the plurality of data records from the source repository based on the schedule, creating an extraction transaction from the plurality of data records, wherein the extraction transaction comprises a subset of the plurality of data records and metadata, and sending the extraction transaction to the data recipient.

Abstract: Aspects of the present disclosure relate to version control. A version control server receives, from a client device, a request for making changes to a portion of data stored in a data store. The version control server verifies that the client device has permission to make the changes to the portion of the data stored in the data store. The version control server updates, using a version control servlet, the data store according to the request. The version control sever creates, forks, merges or deletes, using a repository management API (application programming interface), a repository and/or the files in the repository in the data store based on the request. The version control server triggers, using the repository management API, one or more webhooks in response to the request.

Abstract: A system for a lattice mesh comprises an interface and a processor. The interface is configured to receive a request to join a publish group from a client. The processor is configured to determine a group key; determine whether the client is allowed access to a topic of the publish group; in response to the client being allowed access to the topic of the publish group, provide the group key to the client; publish a group message including metadata; determine whether it is time to rotate the group key; and in response to it being time to rotate the group key, determine a new group key.

Abstract: Computer-implemented techniques for data extraction are described. The techniques include a method and system for retrieving an extraction job specification, wherein the extraction job specification has a source repository identifier that identifies a source repository including a plurality of data records; a data recipient identifier that identifies a data recipient; and a schedule that indicates a timing of when to retrieve the plurality of data records. The method and system further include retrieving the plurality of data records from the source repository based on the schedule, creating an extraction transaction from the plurality of data records, wherein the extraction transaction includes a subset of the plurality of data records and metadata, and sending the extraction transaction to the data recipient.

Abstract: A method and system for data pipeline monitoring receives an event data object and a current status data object from one or more subsystems of a pipeline. The system analyzes the event data object and the current status data object to determine a first and second validation value. The system, in response to determining that either the first or second validation value is not valid, sends a notification.

Abstract: A system for a lattice mesh comprises an interface and a processor. The interface is configured to receive a request to join a publish group from a client. The processor is configured to determine a group key; determine whether the client is allowed access to a topic of the publish group; in response to the client being allowed access to the topic of the publish group, provide the group key to the client; publish a group message including metadata; determine whether it is time to rotate the group key; and in response to it being time to rotate the group key, determine a new group key.

Abstract: Techniques for propagation of deletion operations among a plurality of related datasets are described herein.

Abstract: Discussed herein are embodiments of methods and systems which allow engineers or administrators to create modular plugins which represent the logic for various fault detection tests that can be performed on data pipelines and shared among different software deployments. In some cases, the modular plugins each define a particular test to be executed against data received from the pipeline in addition to one or more configuration points. The configuration points represent configurable arguments, such as variables and/or functions, referenced by the instructions which implement the tests and that can be set according to the specific operation environment of the monitored pipeline.

Abstract: A system for a lattice mesh comprises an interface and a processor. The interface is configured to receive a request to register from a host, wherein the request to register includes a key and a set of asset IDs that the host wishes to claim. The processor is configured to sign the key to generate a resource authority (RA) certificate signed key with an RA certificate; update an asset database with the RA certificate signed key; distribute the RA certificate signed host public key through the network; and provide the host with the RA certificate signed key. In some embodiments, the system further comprises a memory that is coupled to the processor and configured to provide the processor with instructions.

Abstract: A system for a lattice mesh comprises an interface and a processor. The interface is configured to receive a request to register from a host, wherein the request to register includes a key and a set of asset IDs that the host wishes to claim. The processor is configured to sign the key to generate a resource authority (RA) certificate signed key with an RA certificate; update an asset database with the RA certificate signed key; distribute the RA certificate signed host public key through the network; and provide the host with the RA certificate signed key. In some embodiments, the system further comprises a memory that is coupled to the processor and configured to provide the processor with instructions.

Abstract: A method and system for data pipeline monitoring receives an event data object and a current status data object from one or more subsystems of a pipeline. The system analyzes the event data object and the current status data object to determine a first and second validation value. The system, in response to determining that either the first or second validation value is not valid, sends a notification.

Abstract: Discussed herein are embodiments of methods and systems which allow engineers or administrators to create modular plugins which represent the logic for various fault detection tests that can be performed on data pipelines and shared among different software deployments. In some cases, the modular plugins each define a particular test to be executed against data received from the pipeline in addition to one or more configuration points. The configuration points represent configurable arguments, such as variables and/or functions, referenced by the instructions which implement the tests and that can be set according to the specific operation environment of the monitored pipeline.

Abstract: Data transformation in a distributed system of applications and data repositories is described. The subsystems for the overall framework are distributed, thereby allowing for customization to require only isolated changes to one or more subsystems. In one embodiment, a source code repository is used to receive and store source code. A build subsystem can retrieve source code from the source code repository and build it, using one or more criteria. By building the source code, the build subsystem can generate an artifact, which is executable code, such as a JAR or SQL file. Likewise, by building the source code, the build subsystem can generate one or more job specifications for executing the executable code. In one embodiment, the artifact and job specification may be used to launch an application server in a cluster. The application server can then receive data transformation instructions and execute the data transformation instructions.

Abstract: A method and system for data pipeline monitoring receives an event data object and a current status data object from one or more subsystems of a pipeline. The system analyzes the event data object and the current status data object to determine a first and second validation value. The system, in response to determining that either the first or second validation value is not valid, sends a notification.