Abstract: A shared database platform can interface with a cluster computing platform over a network through a database connector and one or more cluster connectors. The data transferred over the network can include telemetry metadata that can be distributed to execution nodes of the cluster computing platform for generation and transmission of cluster data to the shared database platform.

Abstract: The subject technology receives first code in a first programming language, the first code including a reference to a first dataset. The subject technology converting the first code to: second code in a second programming language different than the first programming language, and at least one data query in a second format different than the first programming language. Further the subject technology sends the converted first code and the at least one data query to a data system for execution.

Abstract: A system for monitoring job execution includes an interface and a processor. The interface is configured to receive an indication to start a cluster processing job. The processor is configured to determine whether processing a data instance associated with the cluster processing job satisfies a watchdog criterion; and in the event that processing the data instance satisfies the watchdog criterion, cause the processing of the data instance to be killed.

Abstract: Disclosed are systems, methods, and non-transitory computer-readable media for a providing stored procedures in a network-based database system. A database system executes a stored procedure within a sandbox process. The sandbox process separates the stored procedure from the other services and processes of the database system and may also limit the resources (e.g., storage, memory, etc.) and functionality available to the stored procedure. Stored procedures are commonly designed to execute database queries that are processed by other components of the database system. To provide for secure communication between the stored procedure and the other components of the database system, the sandbox process is loaded with java database connectivity (JDBC) libraries (e.g., JDBC instance) that is modified to limit the communication abilities of the stored procedure.

Abstract: The subject technology receives, in a computing process, a user defined function, the user defined function including code related to at least one operation to be performed. The subject technology determines by a security manager whether performing the at least one operation is permitted, the security manager determines restrictions, based at least in part on a security policy. The subject technology performs the at least one operation. The subject technology sends a result of the at least one operation to the computing process, where sending the result of the at least one operation utilizes a data transport mechanism that supports a network transfer of columnar data.

Abstract: The subject technology receives, in a first computing process, a user defined function, the user defined function including code related to at least one operation to be performed. The subject technology sends a request based at least in part on the at least one operation to a second computing process to perform. The subject technology determines, by a security manager executing within the second computing process, whether performing the at least one operation is permitted, the security manager determines restrictions, based at least in part on a security policy, on operations executing within a sandbox environment provided by the second computing process. The subject technology performs, in the second computing process, the at least one operation, the security manager executing within the second computing process.

Abstract: A system for code development and execution includes a client interface and a client processor. The client interface is configured to receive user code for execution and receive an indication of a server that will perform the execution. The client processor is configured to parse the user code to identify one or more data items referred to during the execution. The client processor is also configured to provide the server with an inquiry for metadata regarding the one or more data items, receive the metadata regarding the one or more data items, determine a logical plan based at least in part on the metadata regarding the one or more data items; and provide the logical plan to the server for execution.

Abstract: A shared database platform can interface with a cluster computing platform over a network through a database connector and one or more cluster connectors. The data transferred over the network can include telemetry metadata that can be distributed to execution nodes of the cluster computing platform for generation and transmission of cluster data to the shared database platform.

Abstract: A system for directory level atomic commits includes an interface and a processor. The interface is configured to receive an indication to provide a set of files. The processor is configured to determine whether a file in a directory has been either 1) atomically committed or 2) written by a non-atomic process and not designated as deleted and provide the file as one file of the set of files in the event that the file in the directory has been either 1) atomically committed or 2) written by a non-atomic process and not designated as deleted.

Abstract: A shared database platform can interface with a cluster computing platform over a network through a database connector and one or more cluster connectors. The data transferred over the network can include telemetry metadata that can be distributed to execution nodes of the cluster computing platform for generation and transmission of cluster data to the shared database platform.

Abstract: The subject technology receives, in a first computing process, a user defined function, the user defined function including code related to at least one operation to be performed. The subject technology sends a request based on the at least one operation to a second computing process to perform, the second computing process being different than the first computing process and comprising a sandbox for executing the at least one operation. The subject technology receives, by the second computing process, the request. The subject technology determines, using at least a security policy, whether performing the at least one operation is permitted. The subject technology performs, in the second computing process, the least one operation. The subject technology sends, by the second computing process, a result of the at least one operation to the first computing process.

Abstract: The allocation system comprises an interface and a processor. The interface is configured to receive an indication to deactivate idle cluster machines of a set of cluster machines. The processor is configured to determine a list of cluster machines storing one or more intermediate data files of a set of intermediate data files; determine a set of idle cluster machines of the set of cluster machines that are neither running one or more tasks of a set of tasks executing or pending on the set of cluster machines nor storing the one or more intermediate data files of the set of intermediate data files, where the set of intermediate data files is associated with the set of tasks executing or pending on the cluster machines; and deactivate each cluster machine of the set of idle cluster machines.

Abstract: The allocation system comprises an interface and a processor. The interface is configured to receive an indication to deactivate idle cluster machines of a set of cluster machines. The processor is configured to determine a set of tasks executing or pending on the set of cluster machines; determine a set of idle cluster machines of the set of cluster machines that are neither running one or more tasks of the set of tasks nor storing one or more intermediate data files of a set of intermediate data files, where the set of intermediate data files is associated with a set of tasks executing or pending on the cluster machines; and deactivate each cluster machine of the set of idle cluster machines.

Abstract: A system for code development and execution includes a client interface and a client processor. The client interface is configured to receive user code for execution and receive an indication of a server that will perform the execution. The client processor is configured to parse the user code to identify one or more data items referred to during the execution. The client processor is also configured to provide the server with an inquiry for metadata regarding the one or more data items, receive the metadata regarding the one or more data items, determine a logical plan based at least in part on the metadata regarding the one or more data items; and provide the logical plan to the server for execution.

Abstract: A system for monitoring job execution includes an interface and a processor. The interface is configured to receive an indication to start a cluster processing job. The processor is configured to determine whether processing a data instance associated with the cluster processing job satisfies a watchdog criterion; and in the event that processing the data instance satisfies the watchdog criterion, cause the processing of the data instance to be killed.

Abstract: A system for code development and execution includes a client interface and a client processor. The client interface is configured to receive user code for execution and receive an indication of a server that will perform the execution. The client processor is configured to parse the user code to identify one or more data items referred to during the execution. The client processor is also configured to provide the server with an inquiry for metadata regarding the one or more data items, receive the metadata regarding the one or more data items, determine a logical plan based at least in part on the metadata regarding the one or more data items; and provide the logical plan to the server for execution.

Abstract: A system for monitoring job execution includes an interface and a processor. The interface is configured to receive an indication to start a cluster processing job. The processor is configured to determine whether processing a data instance associated with the cluster processing job satisfies a watchdog criterion; and in the event that processing the data instance satisfies the watchdog criterion, cause the processing of the data instance to be killed.

Abstract: A system for code development and execution includes a client interface and a client processor. The client interface is configured to receive user code for execution and receive an indication of a server that will perform the execution. The client processor is configured to parse the user code to identify one or more data items referred to during the execution. The client processor is also configured to provide the server with an inquiry for metadata regarding the one or more data items, receive the metadata regarding the one or more data items, determine a logical plan based at least in part on the metadata regarding the one or more data items; and provide the logical plan to the server for execution.

Abstract: A system for cluster resource allocation includes an interface and a processor. The interface is configured to receive a process and input data. The processor is configured to determine an estimate for resources required for the process to process the input data; determine existing available resources in a cluster for running the process; determine whether the existing available resources are sufficient for running the process; in the event it is determined that the existing available resources are not sufficient for running the process, indicate to add new resources; determine an allocated share of resources in the cluster for running the process; and cause execution of the process using the share of resources.

Abstract: The disclosed embodiments provide a system for processing queries of a graph database. During operation, the system executes one or more processes for providing the graph database storing a graph, wherein the graph includes a set of nodes, a set of edges between pairs of nodes in the set of nodes, and a set of predicates. Next, the system receives a query containing a missing value parameter, wherein the missing value parameter designates an attribute in the graph database with a nonexistent value. The system then produces, from the query, a transformed query comprising one or more parameters that do not include the missing value parameter. Finally, the system uses the transformed query and the graph database to generate a result of the query and provides the result in a response to the query.