Abstract: Systems and methods for isolating applications associated with multiple tenants within a computing platform receive a request from a client associated with a tenant for running an application on a computing platform. Hosts connected to the platform are associated with a network address and configured to run applications associated with multiple tenants. A host is identified based at least in part on the request. One or more broadcast domain(s) including the identified hosts are generated. The broadcast domains are isolated in the network at a data link layer. A unique tenant identification number corresponding to the tenant is assigned to the broadcast domains. In response to launching the application on the host: the unique tenant identification number is assigned to the launched application and is added to the network address of the host; and the network address of the host is sent to the client associated with the tenant.

Abstract: A system and method for authenticating users of a data processing platform stores a mapping of a unique user platform identifier to multiple user identity provider identifiers associated with multiple realms for a same user. In some examples, the method includes receiving a request from a client device to establish an access session to perform one or more actions on data of the data processing platform and receiving, from at least one of the first external identity provider of the first realm or the second external identity provider of the second realm, a user identity provider identifier associated with the request. In certain examples, the method includes granting permission to perform the one or more actions on the data of the data processing platform based at least in part on the received user identity provider identifier.

Abstract: An apparatus, computer-implemented method and computer program are disclosed for performing a cryptographic operation in a high-trust (HT) environment. The HT environment including a compute service and key storage service. The compute service receives from a user device, a user request for performing a cryptographic operation on at least a portion of a large-scale dataset. The user request including a user token associated with a user of the user device. The compute service sends to the key storage service, a cryptographic key access request corresponding to the received user request. The cryptographic key access request including data representative of the user token and/or a compute service token.

Abstract: Computing systems methods, and non-transitory storage media are provided for retrieving information regarding an operation to be performed by a platform, performing a preliminary validation of the operation, generating details regarding the preliminary validation, transmitting at least a subset of the details of the preliminary validation to the platform, and populating the generated details on an interface. If the preliminary validation fails, the platform refrains from performing the operation. Furthermore, the logic describing the operation can be executed on different platforms and is not bound or limited to one platform.

Abstract: A database system comprised of a decoupled compute layer and storage layer is implemented to store, build, and maintain a canonical dataset, a temporary buffer, and projection datasets. The canonical dataset is a set of batch updated data. The data is appended in chunks to the canonical dataset such that the canonical dataset becomes a historical dataset over time. The buffer is a write ahead log that contains the most recent chunks of data and provides atomicity and durability for the database system. The projection datasets are indexes of the canonical dataset and/or the buffer that may have single or multiple column sort-orders and/or particular data formats. The writes to the canonical dataset, projection datasets, and buffer may be asynchronous and therefore the database system is advantageously less resource constrained.

Abstract: Systems and methods for isolating applications associated with multiple tenants within a computing platform receive a request from a client associated with a tenant for running an application on a computing platform. Hosts connected to the platform are associated with a network address and configured to run applications associated with multiple tenants. A host is identified based at least in part on the request. One or more broadcast domain(s) including the identified hosts are generated. The broadcast domains are isolated in the network at a data link layer. A unique tenant identification number corresponding to the tenant is assigned to the broadcast domains. In response to launching the application on the host: the unique tenant identification number is assigned to the launched application and is added to the network address of the host; and the network address of the host is sent to the client associated with the tenant.

Abstract: In some embodiments, a method comprises obtaining a pipeline of operations, the pipeline of operations including a plurality of functions providing any of one or more modification operations or visualization operations for a plurality of datasets. A first dynamic visualization of the pipeline of operations at a first level of granularity is generated. A second dynamic visualization of the pipeline of operations at a second level of granularity is generated in response to user input.

Abstract: A computer-implemented system or process is programmed or configured to use a configuration file to specify one or more tasks to apply to raw ingested data. A task may be a sequence of instructions programmed or configured to format raw ingested data into a dataset in a CSV format. Examples of tasks may include: a parser to parse Cobol data into a CSV, a parser to parse XML into a CSV, a parser to parse text using fixed-width fields to a CSV, a parser to parse files in a zip archive into a CSV, a regular expression search/replace function, or formatting logic to remove lines or blank lines from raw ingested data. In one embodiment, the configuration file may specify a schema definition for a task to use for generating a dataset. In one embodiment, the configuration file may also include one or more access control list (ACL) definitions for the generated dataset. In one embodiment, the building of datasets using the configuration file is automated, for example, on a nightly basis.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media for data security protection are provided. One of the methods includes: receiving a job associated with a project, wherein the project is associated with one or more data sources; identifying a plurality of inputs and a plurality of outputs associated with the job; determining a plurality of required permissions associated with the job, wherein each of the required permissions comprises an operation on a required data source, the operation corresponding to at least one of the inputs or the outputs; verifying that the one or more data sources associated with the project comprise the required data source associated with each of the required permissions; and generating a token associated with the job, the token encoding the required permissions associated with the job, wherein the token is required for execution of the job.

Abstract: Systems and methods for isolating applications associated with multiple tenants within a computing platform receive a request from a client associated with a tenant for running an application on a computing platform. Hosts connected to the platform are associated with a network address and configured to run applications associated with multiple tenants. A host is identified based at least in part on the request. One or more broadcast domain(s) including the identified hosts are generated. The broadcast domains are isolated in the network at a data link layer. A unique tenant identification number corresponding to the tenant is assigned to the broadcast domains. In response to launching the application on the host: the unique tenant identification number is assigned to the launched application and is added to the network address of the host; and the network address of the host is sent to the client associated with the tenant.

Abstract: A database system comprised of a decoupled compute layer and storage layer is implemented to store, build, and maintain a canonical dataset, a temporary buffer, and projection datasets. The canonical dataset is a set of batch updated data. The data is appended in chunks to the canonical dataset such that the canonical dataset becomes a historical dataset over time. The buffer is a write ahead log that contains the most recent chunks of data and provides atomicity and durability for the database system. The projection datasets are indexes of the canonical dataset and/or the buffer that may have single or multiple column sort-orders and/or particular data formats. The writes to the canonical dataset, projection datasets, and buffer may be asynchronous and therefore the database system is advantageously less resource constrained.

Abstract: A computer-implemented system or process is programmed or configured to use a configuration file to specify one or more tasks to apply to raw ingested data. A task may be a sequence of instructions programmed or configured to format raw ingested data into a dataset in a CSV format. Examples of tasks may include: a parser to parse Cobol data into a CSV, a parser to parse XML into a CSV, a parser to parse text using fixed-width fields to a CSV, a parser to parse files in a zip archive into a CSV, a regular expression search/replace function, or formatting logic to remove lines or blank lines from raw ingested data. In one embodiment, the configuration file may specify a schema definition for a task to use for generating a dataset. In one embodiment, the configuration file may also include one or more access control list (ACL) definitions for the generated dataset. In one embodiment, the building of datasets using the configuration file is automated, for example, on a nightly basis.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media for data security protection are provided. One of the methods includes: receiving a job associated with a project, wherein the project is associated with one or more data sources; identifying a plurality of inputs and a plurality of outputs associated with the job; determining a plurality of required permissions associated with the job, wherein each of the required permissions comprises an operation on a required data source, the operation corresponding to at least one of the inputs or the outputs; verifying that the one or more data sources associated with the project comprise the required data source associated with each of the required permissions; and generating a token associated with the job, the token encoding the required permissions associated with the job, wherein the token is required for execution of the job.

Abstract: A computer-implemented system or process is programmed or configured to use a configuration file to specify one or more tasks to apply to raw ingested data. A task may be a sequence of instructions programmed or configured to format raw ingested data into a dataset in a CSV format. Examples of tasks may include: a parser to parse Cobol data into a CSV, a parser to parse XML into a CSV, a parser to parse text using fixed-width fields to a CSV, a parser to parse files in a zip archive into a CSV, a regular expression search/replace function, or formatting logic to remove lines or blank lines from raw ingested data. In one embodiment, the configuration file may specify a schema definition for a task to use for generating a dataset. In one embodiment, the configuration file may also include one or more access control list (ACL) definitions for the generated dataset. In one embodiment, the building of datasets using the configuration file is automated, for example, on a nightly basis.

Abstract: Data item deltas are generated for each of M updates of a plurality of updates, wherein M is greater than or equal to one, and a first first-level combined delta is generated representing N updates of the plurality of updates, wherein N is greater than M, and the N updates comprise the M updates and O=N?M other updates. A first second-level combined delta is generated representing J updates of the plurality of updates, wherein J is greater than N, and the J updates comprise the N updates and K other updates of the plurality of updates, wherein K=J?N. The deltas, the first first-level combined delta and the first second-level combined delta are stored for enabling subsequent reading of at least part of the data by accessing the data item, the first first-level combined delta and the first second-level combined delta.

Abstract: Systems and methods for isolating applications associated with multiple tenants within a computing platform receive a request from a client associated with a tenant for running an application on a computing platform. Hosts connected to the platform are associated with a network address and configured to run applications associated with multiple tenants. A host is identified based at least in part on the request. One or more broadcast domain(s) including the identified hosts are generated. The broadcast domains are isolated in the network at a data link layer. A unique tenant identification number corresponding to the tenant is assigned to the broadcast domains. In response to launching the application on the host: the unique tenant identification number is assigned to the launched application and is added to the network address of the host; and the network address of the host is sent to the client associated with the tenant.

Abstract: A system and method for authenticating users of a data processing platform stores a mapping of a unique user platform identifier to multiple user identity provider identifiers associated with multiple realms for a same user. In some examples, the method includes receiving a request from a client device to establish an access session to perform one or more actions on data of the data processing platform and receiving, from at least one of the first external identity provider of the first realm or the second external identity provider of the second realm, a user identity provider identifier associated with the request. In certain examples, the method includes granting permission to perform the one or more actions on the data of the data processing platform based at least in part on the received user identity provider identifier.

Abstract: In some embodiments, a method comprises obtaining a pipeline of operations, the pipeline of operations including a plurality of functions providing any of one or more modification operations or visualization operations for a plurality of datasets. A first dynamic visualization of the pipeline of operations at a first level of granularity is generated. A second dynamic visualization of the pipeline of operations at a second level of granularity is generated in response to user input.

Abstract: Systems and methods for isolating applications associated with multiple tenants within a computing platform receive a request from a client associated with a tenant for running an application on a computing platform. Hosts connected to the platform are associated with a network address and configured to run applications associated with multiple tenants. A host is identified based at least in part on the request. One or more broadcast domain(s) including the identified hosts are generated. The broadcast domains are isolated in the network at a data link layer. A unique tenant identification number corresponding to the tenant is assigned to the broadcast domains. In response to launching the application on the host: the unique tenant identification number is assigned to the launched application and is added to the network address of the host; and the network address of the host is sent to the client associated with the tenant.

Abstract: A system and method for authenticating users of a data processing platform stores a mapping of a unique user platform identifier to multiple user identity provider identifiers associated with multiple realms for a same user. In some examples, the method includes receiving a request from a client device to establish an access session to perform one or more actions on data of the data processing platform and receiving, from at least one of the first external identity provider of the first realm or the second external identity provider of the second realm, a user identity provider identifier associated with the request. In certain examples, the method includes granting permission to perform the one or more actions on the data of the data processing platform based at least in part on the received user identity provider identifier.