Abstract: A data analytics system is disclosed that can include a data repository configured to store data for multiple clients, a metadata repository separate from the data store, an access control system, and a policy store. The data analytics system can automatically generate metadata for data in the data repository using a metadata engine, the metadata including technical metadata and usage metadata, and store the metadata in the metadata repository. The data analytics system can obtain a client policy governing access to the data. The data analytics system can receive a request to provide the data, the request including instructions to create a pipeline to provide the data. The data analytics system can authorize, by the access control system, the request using the policy and usage metadata; create the pipeline using the technical metadata; and provide the data using the pipeline.

Abstract: A data analytics system is disclosed that can include a data repository configured to store data for multiple clients, a metadata repository separate from the data store, an access control system, and a policy store. The data analytics system can automatically generate metadata for data in the data repository using a metadata engine, the metadata including technical metadata and usage metadata, and store the metadata in the metadata repository. The data analytics system can obtain a client policy governing access to the data. The data analytics system can receive a request to provide the data, the request including instructions to create a pipeline to provide the data. The data analytics system can authorize, by the access control system, the request using the policy and usage metadata; create the pipeline using the technical metadata; and provide the data using the pipeline.

Abstract: A data analytics system configured to perform operations is disclosed. The operations can include creating, in response to instructions received from a user, a first pipeline. This pipeline can be configured to extract data from an append-only first data store, extract identifying characteristics from the extracted data, provide the identifying characteristics to an identity service, and receive a tenancy identifier from the identity service. The pipeline can further be configured to create a data object in a second data store using the extracted data; create a tenancy object in a metadata store, the tenancy object associated with the data object, the metadata store implementing a hierarchical data object ownership graph; and associate the tenancy object with a parent object in the hierarchical data object ownership graph. The data analytics system can then tear down the first pipeline.

Abstract: A data analytics system configured to perform operations is disclosed. The operations can include creating, in response to instructions received from a user, a first pipeline. This pipeline can be configured to extract data from an append-only first data store, extract identifying characteristics from the extracted data, provide the identifying characteristics to an identity service, and receive a tenancy identifier from the identity service. The pipeline can further be configured to create a data object in a second data store using the extracted data; create a tenancy object in a metadata store, the tenancy object associated with the data object, the metadata store implementing a hierarchical data object ownership graph; and associate the tenancy object with a parent object in the hierarchical data object ownership graph. The data analytics system can then tear down the first pipeline.

Abstract: A data analytics system is disclosed that is configured to perform operations comprising creating at least one data storage, creating a metadata store separate from the at least one data storage, creating a flow storage, and configuring a flow service using first received instructions. The flow service is configured to obtain a first flow from the flow storage, obtain metadata from the metadata storage, and execute the flow. The flow execution can include obtaining input data from the at least one data storage, generating output data at least in part by validating, transforming, and serializing the input data using the metadata, and generating additional metadata describing the output data. The flow execution can further include providing the output data for storage in the at least one data storage and providing the additional metadata for storage in the metadata storage.

Abstract: A data analytics system including an append-only first data store accessible to multiple clients and a second data store is disclosed. The data analytics system can be configurable to, in response to receiving first instructions from a first target system of a first client, the first target system separate from the data analytics system, create a first pipeline between the append-only first data store and the second data store. The first pipeline can be configured according to the first instructions to generate a client-specific data object and store the client-specific data object in the second data store. The data analytics system can be configurable to tear down the first pipeline upon completion of storing the client-specific data object in the second data store.

Abstract: A data analytics system is disclosed that is configured to perform operations including receiving input data at a first storage location and configuring a flow service to execute a flow. The flow execution can include creating a pipeline using the flow and metadata associated with the flow, the pipeline configured to perform a data transformation specified in the flow. The flow execution can further include determining a tenancy associated with the input data using the flow. The flow execution can also include generating, using the pipeline, output data from the input data and storing, using the pipeline, the output data in a second storage location associated with the tenancy.

Abstract: A data analytics system is disclosed that can include a data repository configured to store data for multiple clients, a metadata repository separate from the data store, an access control system, and a policy store. The data analytics system can automatically generate metadata for data in the data repository using a metadata engine, the metadata including technical metadata and usage metadata, and store the metadata in the metadata repository. The data analytics system can obtain a client policy governing access to the data. The data analytics system can receive a request to provide the data, the request including instructions to create a pipeline to provide the data. The data analytics system can authorize, by the access control system, the request using the policy and usage metadata; create the pipeline using the technical metadata; and provide the data using the pipeline.

Abstract: A data analytics system configured to perform operations is disclosed. The operations can include creating, in response to instructions received from a user, a first pipeline. This pipeline can be configured to extract data from an append-only first data store, extract identifying characteristics from the extracted data, provide the identifying characteristics to an identity service, and receive a tenancy identifier from the identity service. The pipeline can further be configured to create a data object in a second data store using the extracted data; create a tenancy object in a metadata store, the tenancy object associated with the data object, the metadata store implementing a hierarchical data object ownership graph; and associate the tenancy object with a parent object in the hierarchical data object ownership graph. The data analytics system can then tear down the first pipeline.