Abstract: A query referencing a function associated with a remote software component is received by a network-based data warehouse system. Temporary security credentials corresponding to a role at a cloud computing service platform are obtained. The role has permission to send calls to a web endpoint corresponding to the remote software component. A request comprising input data and electronically signed using the temporary security credentials is sent to a web Application Programming Interface (API) management system of the cloud computing service platform. The request, when received by the web API management system, causes the web API management system to invoke external functionality provided by the remote software component at the web endpoint with respect to the input data. A response comprising a result of invoking the external functionality is received from the web API management system, and the result data is processed according to the query.

Abstract: A user interface that includes a data flow pane and a profile pane. The data flow pane displays a flow diagram that identifies a data source. For each of multiple queries against the data source, the process issues the query against the data source asynchronously with an initial block size. Upon retrieval of the initial set of rows, the process repeats the query asynchronously with an updated block size until all of the rows have been retrieved. Periodically the process determines a high water mark for rows from the data source that have been retrieved for all of the queries. When the water mark changes, the process updates the profile pane to display data value histograms for multiple data fields in the data source. Each bar in each data value histogram counts the rows below the water mark that have a single specific data value or range of data values.

Abstract: A query referencing a function associated with a remote software component is received by a network-based data warehouse system. Temporary security credentials corresponding to a role at a cloud computing service platform are obtained. The role has permission to send calls to a web endpoint corresponding to the remote software component. A request comprising input data and electronically signed using the temporary security credentials is sent to a web Application Programming Interface (API) management system of the cloud computing service platform. The request, when received by the web API management system, causes the web API management system to invoke external functionality provided by the remote software component at the web endpoint with respect to the input data. A response comprising a result of invoking the external functionality is received from the web API management system, and the result data is processed according to the query.

Abstract: A computer system prepares data for subsequent analysis. The system displays a user interface that includes a data flow pane, a profile pane, and a data pane. The data flow pane displays a node/link diagram, including a first node representing a join between first and second data sources. When the first node is selected, the system updates the profile pane to display data value histograms for a plurality of data fields and updates the data pane to display rows from the data sources. The system also displays a join control region, including: join type selector; data field pairs that define the join between the data sources; and a join overview graphic including three portions, showing aggregate proportion of rows having an inner join, left outer join, or right outer join relationship. Selecting a portion from the join overview graphic updates the data value histograms and rows in the data pane.

Abstract: A method prepares data for analysis. A user interface includes a flow pane, which includes a flow diagram with nodes, each node having an intermediate set and specifying an operation. A first user input selects a first node, which specifies a first operation. In response, the method displays node internals for the first node, including a description of the first operation. A second user input combines a second node with the first node. The second node specifies a second operation. In response to the second user input, the method (i) removes the second node from the flow diagram, (ii) determines ordering for the first and second operations, (iii) updates the node internals for the first node to specify the first and second operations, ordered according to the ordering, and (iv) displays the updated node internals, including descriptions of the first and second operations, sequenced according to the determined ordering.

Abstract: A method is performed at a computer. The computer retrieves a dataset from a database. The dataset includes a plurality of fields, including a first field. The computer displays a data-preparation graphical user interface, which includes a first region displaying a grid with columns corresponding to the fields and a second region that displays a sequence of operations applied to the dataset. The computer receives a user action to specify a first filter for the first field. The computer removes rows from the grid whose corresponding data values in the first column do not satisfy the first filter. The computer identifies some additional rows from the dataset, not previously displayed in the grid, whose data values for the first field satisfy the first filter, and displays the additional rows in the grid. The computer inserts an operation identifier into the third region, which identifies the first filter.

Abstract: A query referencing a function associated with a remote software component is received by a network-based data warehouse system. Temporary security credentials corresponding to a role at a cloud computing service platform are obtained. The role has permission to send calls to a web endpoint corresponding to the remote software component. A request comprising input data and electronically signed using the temporary security credentials is sent to a web Application Programming Interface (API) management system of the cloud computing service platform. The request, when received by the web API management system, causes the web API management system to invoke external functionality provided by the remote software component at the web endpoint with respect to the input data. A response comprising a result of invoking the external functionality is received from the web API management system, and the result data is processed according to the query.

Abstract: A computer system transforms data. A first user builds a flow diagram. Each node in the flow diagram specifies an operation: to retrieve data, to transform data, or to create an output dataset. The flow diagram includes a subgraph having a plurality of nodes. When the first user initiates execution, the system performs the operations in the subgraph sequentially as specified. When the first user requests the system to save the flow diagram, the system stores the subgraph. The system displays a user interface and the subgraph in the user interface. A second user requests the system to build another flow diagram based on the subgraph. In response, the system retrieves the subgraph from memory. When the second user requests the system to execute the flow diagram built by the second user, the system performs the operations of the nodes in the subgraph sequentially as specified in the subgraph.

Abstract: A query referencing a function associated with a remote software component is received by a network-based data warehouse system. Temporary security credentials corresponding to a role at a cloud computing service platform are obtained. The role has permission to send calls to a web endpoint corresponding to the remote software component. A request comprising input data and electronically signed using the temporary security credentials is sent to a web Application Programming Interface (API) management system of the cloud computing service platform. The request, when received by the web API management system, causes the web API management system to invoke external functionality provided by the remote software component at the web endpoint with respect to the input data. A response comprising a result of invoking the external functionality is received from the web API management system, and the result data is processed according to the query.

Abstract: A user interface that includes a data flow pane and a profile pane. The data flow pane displays a flow diagram that identifies a data source. For each of multiple queries against the data source, the process issues the query against the data source asynchronously with an initial block size. Upon retrieval of the initial set of rows, the process repeats the query asynchronously with an updated block size until all of the rows have been retrieved. Periodically the process determines a high water mark for rows from the data source that have been retrieved for all of the queries. When the water mark changes, the process updates the profile pane to display data value histograms for multiple data fields in the data source. Each bar in each data value histogram counts the rows below the water mark that have a single specific data value or range of data values.

Abstract: An exemplary method includes displaying a user interface that includes multiple panes, and receiving a first user input to select an existing node from a flow diagram or a flow element template. The system displays a moveable icon representing a new node for placement in the flow diagram. The new node specifies an operation corresponding to the selected existing node or the selected flow element template. Drop targets are displayed in the flow diagram according to dependencies between the data flow operation of the new node and operations of existing nodes. A second user input places the moveable icon over a first drop target. In response to ceasing to detect the second user input, the system inserts the new node into the flow diagram at the first drop target. The new node performs the specified data flow operation.

Abstract: A computer system transforms data. A first user builds a flow diagram. Each node in the flow diagram specifies an operation: to retrieve data, to transform data, or to create an output dataset. The flow diagram includes a subgraph having a plurality of nodes. When the first user initiates execution, the system performs the operations in the subgraph sequentially as specified. When the first user requests the system to save the flow diagram, the system stores the subgraph. The system displays a user interface and the subgraph in the user interface. A second user requests the system to build another flow diagram based on the subgraph. In response, the system retrieves the subgraph from memory. When the second user requests the system to execute the flow diagram built by the second user, the system performs the operations of the nodes in the subgraph sequentially as specified in the subgraph.

Abstract: A computer system transforms data. The system displays a user interface including a data flow pane. A user builds a flow diagram in the data flow pane. Each node in the flow diagram specifies an operation: to retrieve data, to transform data, or to create an output dataset. The flow diagram includes a subtree having a data source node and transformation operation nodes. When the user initiates execution and the nodes in the subtree are configured to execute imperatively, the system performs the operations in the subtree sequentially as specified, retrieving data from the data source, transforming the data, and forming an intermediate dataset. When the user initiates execution and the nodes in the subtree are configured to execute declaratively, the system constructs a database query that is logically equivalent to the operations specified in the subtree and transmits the query to the data source to retrieve the intermediate dataset.

Abstract: A computer system prepares data for subsequent analysis. The system displays a user interface that includes a data flow pane, a profile pane, and a data pane. The data flow pane displays a node/link diagram, including a first node representing a join between first and second data sources. When the first node is selected, the system updates the profile pane to display data value histograms for a plurality of data fields and updates the data pane to display rows from the data sources. The system also displays a join control region, including: join type selector; data field pairs that define the join between the data sources; and a join overview graphic including three portions, showing aggregate proportion of rows having an inner join, left outer join, or right outer join relationship. Selecting a portion from the join overview graphic updates the data value histograms and rows in the data pane.

Abstract: A method is performed at a computer. The computer retrieves a dataset from a database. The dataset includes a plurality of fields, including a first field. The computer displays a data-preparation graphical user interface, which includes a first region displaying a grid with columns corresponding to the fields and a second region that displays a sequence of operations applied to the dataset. The computer receives a user action to specify a first filter for the first field. The computer removes rows from the grid whose corresponding data values in the first column do not satisfy the first filter. The computer identifies some additional rows from the dataset, not previously displayed in the grid, whose data values for the first field satisfy the first filter, and displays the additional rows in the grid. The computer inserts an operation identifier into the third region, which identifies the first filter.

Abstract: A computer system transforms data. The system displays a user interface including a data flow pane. A user builds a flow diagram in the data flow pane. Each node in the flow diagram specifies an operation: to retrieve data, to transform data, or to create an output dataset. The flow diagram includes a subtree having a data source node and transformation operation nodes. When the user initiates execution and the nodes in the subtree are configured to execute imperatively, the system performs the operations in the subtree sequentially as specified, retrieving data from the data source, transforming the data, and forming an intermediate dataset. When the user initiates execution and the nodes in the subtree are configured to execute declaratively, the system constructs a database query that is logically equivalent to the operations specified in the subtree and transmits the query to the data source to retrieve the intermediate dataset.

Abstract: An exemplary method includes displaying a UI that includes a data flow pane and a data grid pane, and in response to user selection of a node, displaying, in the data grid pane, a data grid corresponding to the user-selected node. The method further comprises receiving user input on a first cell in the data grid that changes a first data value v1 for the first cell (intersection between a first column and a first row of the data grid) to a first modified data value w1. The method further comprises determining a formula f(v1)=w1 corresponding to the input, inserting a new node into the flow diagram immediately after the user-selected node, and applying the formula f to the first column for each of the plurality of rows in the data grid, including changing a second data value v2 to a second modified data value w2, where v2?v1 and w2?w1.

Abstract: An exemplary method includes displaying a user interface that includes multiple panes, and receiving a first user input to select an existing node from a flow diagram or a flow element template. The system displays a moveable icon representing a new node for placement in the flow diagram. The new node specifies an operation corresponding to the selected existing node or the selected flow element template. Drop targets are displayed in the flow diagram according to dependencies between the data flow operation of the new node and operations of existing nodes. A second user input places the moveable icon over a first drop target. In response to ceasing to detect the second user input, the system inserts the new node into the flow diagram at the first drop target. The new node performs the specified data flow operation.

Abstract: A computer system displays a user interface that includes a flow pane, a tool pane, a profile pane, and a data pane. The flow pane displays a node/link flow diagram that identifies data sources, operations, and output datasets. The tool pane includes a data source selector that enables users to add data sources to the flow diagram, and includes an operation palette that enables users to insert nodes into the flow diagram for performing specific transformation operations. The profile pane displays schemas corresponding to selected nodes in the flow diagram, including information about data fields and statistical information about data values for the data fields and enables users to modify the flow diagram by interacting with individual data elements. The data pane displays rows of data corresponding to selected nodes in the flow diagram, and enables users to modify the flow diagram by interacting with individual data values.