Abstract: A shared database platform can interface with a cluster computing platform over a network through a database connector and one or more cluster connectors. The data transferred over the network can include telemetry metadata that can be distributed to execution nodes of the cluster computing platform for generation and transmission of cluster data to the shared database platform.

Abstract: The subject technology receives, in a first computing process, a user defined function, the user defined function including code related to at least one operation to be performed. The subject technology sends a request based on the at least one operation to a second computing process to perform, the second computing process being different than the first computing process and comprising a sandbox for executing the at least one operation. The subject technology receives, by the second computing process, the request. The subject technology determines, using at least a security policy, whether performing the at least one operation is permitted. The subject technology performs, in the second computing process, the least one operation. The subject technology sends, by the second computing process, a result of the at least one operation to the first computing process.

Abstract: The allocation system comprises an interface and a processor. The interface is configured to receive an indication to deactivate idle cluster machines of a set of cluster machines. The processor is configured to determine a list of cluster machines storing one or more intermediate data files of a set of intermediate data files; determine a set of idle cluster machines of the set of cluster machines that are neither running one or more tasks of a set of tasks executing or pending on the set of cluster machines nor storing the one or more intermediate data files of the set of intermediate data files, where the set of intermediate data files is associated with the set of tasks executing or pending on the cluster machines; and deactivate each cluster machine of the set of idle cluster machines.

Abstract: The allocation system comprises an interface and a processor. The interface is configured to receive an indication to deactivate idle cluster machines of a set of cluster machines. The processor is configured to determine a set of tasks executing or pending on the set of cluster machines; determine a set of idle cluster machines of the set of cluster machines that are neither running one or more tasks of the set of tasks nor storing one or more intermediate data files of a set of intermediate data files, where the set of intermediate data files is associated with a set of tasks executing or pending on the cluster machines; and deactivate each cluster machine of the set of idle cluster machines.

Abstract: A system for code development and execution includes a client interface and a client processor. The client interface is configured to receive user code for execution and receive an indication of a server that will perform the execution. The client processor is configured to parse the user code to identify one or more data items referred to during the execution. The client processor is also configured to provide the server with an inquiry for metadata regarding the one or more data items, receive the metadata regarding the one or more data items, determine a logical plan based at least in part on the metadata regarding the one or more data items; and provide the logical plan to the server for execution.

Abstract: A system for monitoring job execution includes an interface and a processor. The interface is configured to receive an indication to start a cluster processing job. The processor is configured to determine whether processing a data instance associated with the cluster processing job satisfies a watchdog criterion; and in the event that processing the data instance satisfies the watchdog criterion, cause the processing of the data instance to be killed.

Abstract: A system for code development and execution includes a client interface and a client processor. The client interface is configured to receive user code for execution and receive an indication of a server that will perform the execution. The client processor is configured to parse the user code to identify one or more data items referred to during the execution. The client processor is also configured to provide the server with an inquiry for metadata regarding the one or more data items, receive the metadata regarding the one or more data items, determine a logical plan based at least in part on the metadata regarding the one or more data items; and provide the logical plan to the server for execution.

Abstract: A system for monitoring job execution includes an interface and a processor. The interface is configured to receive an indication to start a cluster processing job. The processor is configured to determine whether processing a data instance associated with the cluster processing job satisfies a watchdog criterion; and in the event that processing the data instance satisfies the watchdog criterion, cause the processing of the data instance to be killed.

Abstract: A system for code development and execution includes a client interface and a client processor. The client interface is configured to receive user code for execution and receive an indication of a server that will perform the execution. The client processor is configured to parse the user code to identify one or more data items referred to during the execution. The client processor is also configured to provide the server with an inquiry for metadata regarding the one or more data items, receive the metadata regarding the one or more data items, determine a logical plan based at least in part on the metadata regarding the one or more data items; and provide the logical plan to the server for execution.

Abstract: A system for cluster resource allocation includes an interface and a processor. The interface is configured to receive a process and input data. The processor is configured to determine an estimate for resources required for the process to process the input data; determine existing available resources in a cluster for running the process; determine whether the existing available resources are sufficient for running the process; in the event it is determined that the existing available resources are not sufficient for running the process, indicate to add new resources; determine an allocated share of resources in the cluster for running the process; and cause execution of the process using the share of resources.

Abstract: The disclosed embodiments provide a system for processing queries of a graph database. During operation, the system executes one or more processes for providing the graph database storing a graph, wherein the graph includes a set of nodes, a set of edges between pairs of nodes in the set of nodes, and a set of predicates. Next, the system receives a query containing a missing value parameter, wherein the missing value parameter designates an attribute in the graph database with a nonexistent value. The system then produces, from the query, a transformed query comprising one or more parameters that do not include the missing value parameter. Finally, the system uses the transformed query and the graph database to generate a result of the query and provides the result in a response to the query.

Abstract: The disclosed embodiments provide a system for processing and storing data. During operation, the system obtains a set of records sorted by a first integer value, with each record containing the first integer value and one or more additional integer values. The system sets, for each integer value in each record, a delta encoding bit that indicates whether the integer value is delta encoded with respect to at least one other integer value in the records. Next, the system uses a delta encoding state represented by the delta encoding bit to calculate an encoded value from the integer value. The system then encodes a length of the encoded value in a length tag for the integer value and replaces the integer value in the record with an encoded version containing the delta encoding bit, the length tag, and the encoded value.

Abstract: A system for cluster resource allocation includes an interface and a processor. The interface is configured to receive a process and input data. The processor is configured to determine an estimate for resources required for the process to process the input data; determine existing available resources in a cluster for running the process; determine whether the existing available resources are sufficient for running the process; in the event it is determined that the existing available resources are not sufficient for running the process, indicate to add new resources; determine an allocated share of resources in the cluster for running the process; and cause execution of the process using the share of resources.

Abstract: A system for directory level atomic commits includes an interface and a processor. The interface is configured to receive an indication to provide a set of files. The processor is configured to determine whether a file in a directory has been either 1) atomically committed or 2) written by a non-atomic process and not designated as deleted and provide the file as one file of the set of files in the event that the file in the directory has been either 1) atomically committed or 2) written by a non-atomic process and not designated as deleted.

Abstract: The disclosed embodiments provide a system for processing queries of a graph database. During operation, the system executes a set of processes for processing queries of a graph database storing a graph, wherein the graph comprises a set of nodes, a set of edges between pairs of nodes in the set of nodes, and a set of predicates. When a query of the graph database is received, the system performs a lookup of a hash map to obtain one or more offsets into an edge store for the graph database, which includes a one-linkage structure and a two-linkage structure. Next, the system accesses the offset(s) in the edge store to obtain a subset of the edges matching the query. The system then uses the subset of the edges to generate a result of the query and provides the result in a response to the query.

Abstract: The disclosed embodiments provide a system for processing queries of a graph database. During operation, the system executes one or more processes for processing queries of a graph database storing a graph, wherein the graph comprises a set of nodes, a set of edges between pairs of nodes in the set of nodes, and a set of predicates. When a query of the graph database is received, the system processes the query by matching a query time of the query to a virtual time in a log-based representation of the graph database. Next, the system uses an edge store for the graph database to access a subset of the edges matching the query. The system then generates a result of the query by materializing updates to the subset of the edges before the virtual time and provides the result in a response to the query.

Abstract: The disclosed embodiments provide a system for processing queries of a graph database. During operation, the system executes one or more processes for providing the graph database storing a graph, wherein the graph includes a set of nodes, a set of edges between pairs of nodes in the set of nodes, and a set of predicates. Next, the system receives a query containing a missing value parameter, wherein the missing value parameter designates an attribute in the graph database with a nonexistent value. The system then produces, from the query, a transformed query comprising one or more parameters that do not include the missing value parameter. Finally, the system uses the transformed query and the graph database to generate a result of the query and provides the result in a response to the query.

Abstract: The present invention extends to methods, systems, and computer program products for formulating global statistics for parallel databases. In general, embodiments of the invention merge (combine) information in multiple compute node level histograms to create a global histogram for a table that is distributed across a number of compute nodes. Merging can include aligning histogram step boundaries across the compute node histograms. Merging can include aggregating histogram step-level information, such as, for example, equality rows and average range rows (or alternately equality rows, range rows, and distinct range rows), across the compute node histograms into a single global step. Merging can account for distinct values that do not appear at one or more compute nodes as well as distinct values that are counted at multiple compute nodes. A resulting global histogram can be coalesced to reduce the step count.

Abstract: The present invention extends to methods, systems, and computer program products for optimizing parallel queries using interesting distributions. For each logical operator in an SQL server MEMO, in a top down manner from a root operator to the leaf operators, interesting distributions for the operators can be identified based on the properties of the operators. Identified interesting distributions can be propagated down to lower operators by annotating the lower operators with the interesting distributions. Thus, a SQL server MEMO can be annotated with interesting distributions propagated top down from root to leaf logical operators to generate an annotated SQL server MEMO. Parallel query plans can then be generated from the annotated SQL server MEMO in a bottom up manner from leaf operators to a root operator. Annotated interesting properties can be used to prune operators, thereby facilitating a more tractable search space for a parallel query plan.

Abstract: In order to facilitate the requesting of desired information from a graph database that stores a graph, a system may translate an initial query into a query that can be executed against the graph database. In particular, using primitives, the system may translate a query associated with a type of database (such as a relational database) into the query. The graph may include nodes, edges between the nodes, and predicates to represent and store data with index-free adjacency. Moreover, the primitives may include: a rule based on edges in the graph that expresses a relational schema in the type of database, and information associated with a compound key that specifies a relationship between nodes, edges and predicates in the graph corresponding to a table in the type of database. Then, the system may execute the query against the graph database, and may receive a result that includes a subset of the graph.