Abstract: Aspects of the disclosure are directed to a runtime catalog for a cloud storage engine that unifies data lakes and data warehouses. The runtime catalog can expose a single universe of cloud storage tables through an endpoint for query engines for data lakes and another endpoint for query engines for data warehouses. The runtime catalog can allow the query engines for data lakes and the query engines for data warehouses to query any cloud storage table by representing data warehouse native tables in a format compatible with data lakes and representing data lake native tables in a format compatible with data warehouses.

Abstract: Aspects of the disclosure are directed to a metadata cache for extending data warehouse features to data lakes. The metadata cache can accelerate query execution by directly accessing unmanaged data from the data lake rather than accessing the data through the data warehouse. The metadata cache can allow for filtering the unmanaged data to improve the speed of retrieving data for executing a query.

Abstract: A method for unstructured data analytics in data warehouses includes receiving an unstructured data query from a user, the unstructured data query requesting the data processing hardware determine one or more unstructured data files stored at a data repository that match query parameters. The method includes determining, using an object table, a set of unstructured data files stored at the data repository that matches the query parameters. The object table includes a plurality of rows, each row of the plurality of rows associated with a respective unstructured data file stored at the data repository, and a plurality of columns, each column of the plurality of columns comprising metadata associated with the respective unstructured data file of each row of the plurality of rows. The method includes returning, to the user, a structured data table including the determined set of unstructured data files.

Abstract: The present disclosure provides a storage engine that unifies data warehouses and lakes, by providing uniform fine-grained access control, performance acceleration across multi-cloud storage, and open formats. It provides an application programming interface (API) for query engines spanning across data warehouse and open source runtimes to access distributed data with consistent security and governance controls. Access is evaluated at the API layer, separate from the query engine, and is uniformly enforced across query engines.

Abstract: Direct storage loading may be used to add data to a database. New data may be added to a database, using nodes different than a database engine to access a database. The addition of the new data may be assigned to different nodes. The nodes may obtain the data and store the data to storage locations according allocated space in the database by the database engine. The new data can then be made available for access at the database engine.

Abstract: Storage level query processing may be implemented for processing database queries. Nodes that can access a database may perform parallel processing for at least a portion of a database query. An indication may be received that specifies parallel processing for the database query. The nodes can then be caused to perform the portion of the query as part of providing a result in response to the database query instead of a node, such as a query engine node, that received the database query.

Abstract: Direct storage loading may be used to add data to a database. New data may be added to a database, using nodes different than a database engine to access a database. The addition of the new data may be assigned to different nodes. The nodes may obtain the data and store the data to storage locations according allocated space in the database by the database engine. The new data can then be made available for access at the database engine.

Abstract: Direct storage loading may be used to add data to a database. New data may be added to a database, using nodes different than a database engine to access a database. The addition of the new data may be assigned to different nodes. The nodes may obtain the data and store the data to storage locations according allocated space in the database by the database engine. The new data can then be made available for access at the database engine.

Abstract: An identification system that may be used in heterogeneous computing environments provides a fail-free path to providing identifiers from a single canonical namespace. Objects or gateways requiring an identifier for access are accessed using an identifier for the canonical namespace. If an entity requests access using an identifier from another namespace, an external database is consulted to determine if a mapping exists for the identifier to another identifier the canonical namespace. If no mapping exists, or the external database is unavailable, then an identifier is automatically generated in the canonical namespace and is used for the access. An internal database is updated with the automatically generated identifier, providing a mechanism to add mappings without administrative intervention. To access resources requiring an identifier from another particular namespace, a canonical namespace identifier may be mapped to another identifier in the particular namespace, or a generic identifier may be used.

Abstract: Fine-grained virtualization provisioning may be performed for in-place database scaling. Computing resource utilization for a database on a host system is obtained for a period of time. The computing resource utilization may be evaluated with respect to a target capacity for the database. If a scaling event is detected based on the evaluation, a modified target capacity may be determined and used to make an adjustment of the computing resources permitted to be used by the database.

Abstract: An identification system that may be used in heterogeneous computing environments provides a fail-free path to providing identifiers from a single canonical namespace. Objects or gateways requiring an identifier for access are accessed using an identifier for the canonical namespace. If an entity requests access using an identifier from another namespace, an external database is consulted to determine if a mapping exists for the identifier to another identifier the canonical namespace. If no mapping exists, or the external database is unavailable, then an identifier is automatically generated in the canonical namespace and is used for the access. An internal database is updated with the automatically generated identifier, providing a mechanism to add mappings without administrative intervention. To access resources requiring an identifier from another particular namespace, a canonical namespace identifier may be mapped to another identifier in the particular namespace, or a generic identifier may be used.

Abstract: An identification system that may be used in heterogeneous computing environments provides a fail-free path to providing identifiers from a single canonical namespace. Objects or gateways requiring an identifier for access are accessed using an identifier for the canonical namespace. If an entity requests access using an identifier from another namespace, an external database is consulted to determine if a mapping exists for the identifier to another identifier the canonical namespace. If no mapping exists, or the external database is unavailable, then an identifier is automatically generated in the canonical namespace and is used for the access. An internal database is updated with the automatically generated identifier, providing a mechanism to add mappings without administrative intervention. To access resources requiring an identifier from another particular namespace, a canonical namespace identifier may be mapped to another identifier in the particular namespace, or a generic identifier may be used.

Abstract: A computer-implemented method of encryption of several units of a computerized system, wherein each of the units comprises data, includes generating distinct initialization vectors, or IVs, for the units, and storing the generated IVs; and for each unit of the several units: accessing a stored IV corresponding to the unit; and encrypting the unit according to the accessed IV and an encryption key.

Abstract: A computer-implemented method of encryption of several units of a computerized system, wherein each of the units comprises data, includes generating distinct initialization vectors, or IVs, for the units, and storing the generated IVs; and for each unit of the several units: accessing a stored IV corresponding to the unit; and encrypting the unit according to the accessed IV and an encryption key.

Abstract: Embodiments relate to dynamically allocating access to a shared resource based on quota shares. An aspect includes computing, by a client node, an expected size of quota shares needed by the client node based on a historical rate of resource consumption by the client node. The client node transmits a request for a count of quota shares based on the expected size. The client node receives an indication that the request for the count of quota shares is one of: granted, denied, and granted-in-part. The client node self-rules the shared resource based on the indication.

Abstract: Embodiments relate to dynamically allocating access to a shared resource based on quota shares. An aspect includes computing, by a client node, an expected size of quota shares needed by the client node based on a historical rate of resource consumption by the client node. The client node transmits a request for a count of quota shares based on the expected size. The client node receives an indication that the request for the count of quota shares is one of: granted, denied, and granted-in-part. The client node self-rules the shared resource based on the indication.

Abstract: An identification system that may be used in heterogeneous computing environments provides a fail-free path to providing identifiers from a single canonical namespace. Objects or gateways requiring an identifier for access are accessed using an identifier for the canonical namespace. If an entity requests access using an identifier from another namespace, an external database is consulted to determine if a mapping exists for the identifier to another identifier the canonical namespace. If no mapping exists, or the external database is unavailable, then an identifier is automatically generated in the canonical namespace and is used for the access. An internal database is updated with the automatically generated identifier, providing a mechanism to add mappings without administrative intervention. To access resources requiring an identifier from another particular namespace, a canonical namespace identifier may be mapped to another identifier in the particular namespace, or a generic identifier may be used.

Abstract: An identification system that may be used in heterogeneous computing environments provides a fail-free path to providing identifiers from a single canonical namespace. Objects or gateways requiring an identifier for access are accessed using an identifier for the canonical namespace. If an entity requests access using an identifier from another namespace, an external database is consulted to determine if a mapping exists for the identifier to another identifier the canonical namespace. If no mapping exists, or the external database is unavailable, then an identifier is automatically generated in the canonical namespace and is used for the access. An internal database is updated with the automatically generated identifier, providing a mechanism to add mappings without administrative intervention. To access resources requiring an identifier from another particular namespace, a canonical namespace identifier may be mapped to another identifier in the particular namespace, or a generic identifier may be used.

Abstract: Embodiments relate to dynamically allocating access to a shared resource based on quota shares. An aspect includes computing, by a client node, an expected size of quota shares needed by the client node based on a historical rate of resource consumption by the client node. The client node transmits a request for a count of quota shares based on the expected size. The client node receives an indication that the request for the count of quota shares is one of: granted, denied, and granted-in-part. The client node self-rules the shared resource based on the indication.

Abstract: A computer-implemented method of encryption of several units of a computerized system, wherein each of the units comprises data, includes generating distinct initialization vectors, or IVs, for the units, and storing the generated IVs; and for each unit of the several units: accessing a stored IV corresponding to the unit; and encrypting the unit according to the accessed IV and an encryption key.