Abstract: A system and method for authenticating users of a data processing platform stores a mapping of a unique user platform identifier to multiple user identity provider identifiers associated with multiple realms for a same user. In some examples, the method includes receiving a request from a client device to establish an access session to perform one or more actions on data of the data processing platform and receiving, from at least one of the first external identity provider of the first realm or the second external identity provider of the second realm, a user identity provider identifier associated with the request. In certain examples, the method includes granting permission to perform the one or more actions on the data of the data processing platform based at least in part on the received user identity provider identifier.

Abstract: A database system comprised of a decoupled compute layer and storage layer is implemented to store, build, and maintain a canonical dataset, a temporary buffer, and projection datasets. The canonical dataset is a set of batch updated data. The data is appended in chunks to the canonical dataset such that the canonical dataset becomes a historical dataset over time. The buffer is a write ahead log that contains the most recent chunks of data and provides atomicity and durability for the database system. The projection datasets are indexes of the canonical dataset and/or the buffer that may have single or multiple column sort-orders and/or particular data formats. The writes to the canonical dataset, projection datasets, and buffer may be asynchronous and therefore the database system is advantageously less resource constrained.

Abstract: In some embodiments, a method comprises obtaining a pipeline of operations, the pipeline of operations including a plurality of functions providing any of one or more modification operations or visualization operations for a plurality of datasets. A first dynamic visualization of the pipeline of operations at a first level of granularity is generated. A second dynamic visualization of the pipeline of operations at a second level of granularity is generated in response to user input.

Abstract: A system and method for authenticating users of a data processing platform stores a mapping of a unique user platform identifier to multiple user identity provider identifiers associated with multiple realms for a same user. In some examples, the method includes receiving a request from a client device to establish an access session to perform one or more actions on data of the data processing platform and receiving, from at least one of the first external identity provider of the first realm or the second external identity provider of the second realm, a user identity provider identifier associated with the request. In certain examples, the method includes granting permission to perform the one or more actions on the data of the data processing platform based at least in part on the received user identity provider identifier.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media for data security protection are provided. One of the methods includes: receiving a job associated with a project, wherein the project is associated with one or more data sources; identifying a plurality of inputs and a plurality of outputs associated with the job; determining a plurality of required permissions associated with the job, wherein each of the required permissions comprises an operation on a required data source, the operation corresponding to at least one of the inputs or the outputs; verifying that the one or more data sources associated with the project comprise the required data source associated with each of the required permissions; and generating a token associated with the job, the token encoding the required permissions associated with the job, wherein the token is required for execution of the job.

Abstract: Systems and methods for isolating applications associated with multiple tenants within a computing platform receive a request from a client associated with a tenant for running an application on a computing platform. Hosts connected to the platform are associated with a network address and configured to run applications associated with multiple tenants. A host is identified based at least in part on the request. One or more broadcast domain(s) including the identified hosts are generated. The broadcast domains are isolated in the network at a data link layer. A unique tenant identification number corresponding to the tenant is assigned to the broadcast domains. In response to launching the application on the host: the unique tenant identification number is assigned to the launched application and is added to the network address of the host; and the network address of the host is sent to the client associated with the tenant.

Abstract: Systems and methods for isolating applications associated with multiple tenants within a computing platform receive a request from a client associated with a tenant for running an application on a computing platform. Hosts connected to the platform are associated with a network address and configured to run applications associated with multiple tenants. A host is identified based at least in part on the request. One or more broadcast domain(s) including the identified hosts are generated. The broadcast domains are isolated in the network at a data link layer. A unique tenant identification number corresponding to the tenant is assigned to the broadcast domains. In response to launching the application on the host: the unique tenant identification number is assigned to the launched application and is added to the network address of the host; and the network address of the host is sent to the client associated with the tenant.

Abstract: A computer-implemented system or process is programmed or configured to use a configuration file to specify one or more tasks to apply to raw ingested data. A task may be a sequence of instructions programmed or configured to format raw ingested data into a dataset in a CSV format. Examples of tasks may include: a parser to parse Cobol data into a CSV, a parser to parse XML into a CSV, a parser to parse text using fixed-width fields to a CSV, a parser to parse files in a zip archive into a CSV, a regular expression search/replace function, or formatting logic to remove lines or blank lines from raw ingested data. In one embodiment, the configuration file may specify a schema definition for a task to use for generating a dataset. In one embodiment, the configuration file may also include one or more access control list (ACL) definitions for the generated dataset. In one embodiment, the building of datasets using the configuration file is automated, for example, on a nightly basis.

Abstract: A computer-implemented system or process is programmed or configured to use a configuration file to specify one or more tasks to apply to raw ingested data. A task may be a sequence of instructions programmed or configured to format raw ingested data into a dataset in a CSV format. Examples of tasks may include: a parser to parse Cobol data into a CSV, a parser to parse XML into a CSV, a parser to parse text using fixed-width fields to a CSV, a parser to parse files in a zip archive into a CSV, a regular expression search/replace function, or formatting logic to remove lines or blank lines from raw ingested data. In one embodiment, the configuration file may specify a schema definition for a task to use for generating a dataset. In one embodiment, the configuration file may also include one or more access control list (ACL) definitions for the generated dataset. In one embodiment, the building of datasets using the configuration file is automated, for example, on a nightly basis.

Abstract: Systems and methods for isolating applications associated with multiple tenants within a computing platform receive a request from a client associated with a tenant for running an application on a computing platform. Hosts connected to the platform are associated with a network address and configured to run applications associated with multiple tenants. A host is identified based at least in part on the request. One or more broadcast domain(s) including the identified hosts are generated. The broadcast domains are isolated in the network at a data link layer. A unique tenant identification number corresponding to the tenant is assigned to the broadcast domains. In response to launching the application on the host: the unique tenant identification number is assigned to the launched application and is added to the network address of the host; and the network address of the host is sent to the client associated with the tenant.

Abstract: Systems, methods, and non-transitory computer readable media are provided for managing assignment of modules. A job from a client may be received. The job may be inserted into a job queue. The job within the job queue may be compared with a set of cooldown modules to determine whether a compatible cooldown module is available. The job may be assigned to the compatible cooldown module responsive to the compatible cooldown module being available.

Abstract: Data item deltas are generated for each of M updates of a plurality of updates, wherein M is greater than or equal to one, and a first first-level combined delta is generated representing N updates of the plurality of updates, wherein N is greater than M, and the N updates comprise the M updates and O=N?M other updates. A first second-level combined delta is generated representing J updates of the plurality of updates, wherein J is greater than N, and the J updates comprise the N updates and K other updates of the plurality of updates, wherein K=J?N. The deltas, the first first-level combined delta and the first second-level combined delta are stored for enabling subsequent reading of at least part of the data by accessing the data item, the first first-level combined delta and the first second-level combined delta.

Abstract: Systems, methods, and non-transitory computer readable media are provided for managing assignment of modules. A job from a client may be received. The job may be inserted into a job queue. The job within the job queue may be compared with a set of cooldown modules to determine whether a compatible cooldown module is available. The job may be assigned to the compatible cooldown module responsive to the compatible cooldown module being available.

Abstract: A computer-implemented system or process is programmed or configured to use a configuration file to specify one or more tasks to apply to raw ingested data. A task may be a sequence of instructions programmed or configured to format raw ingested data into a dataset in a CSV format. Examples of tasks may include: a parser to parse Cobol data into a CSV, a parser to parse XML into a CSV, a parser to parse text using fixed-width fields to a CSV, a parser to parse files in a zip archive into a CSV, a regular expression search/replace function, or formatting logic to remove lines or blank lines from raw ingested data. In one embodiment, the configuration file may specify a schema definition for a task to use for generating a dataset. In one embodiment, the configuration file may also include one or more access control list (ACL) definitions for the generated dataset. In one embodiment, the building of datasets using the configuration file is automated, for example, on a nightly basis.