Abstract: A system may use a large language model (“LLM”) to generate a data pipeline. The system can receive a natural language query and a selection of a plurality of data sets for generating a data pipeline and generate a prompt comprising at least: the natural language query, indications of the plurality of data sets, an indication of a format of a first computer language, and an indication of available data transformations. The system can transmit the prompt to an LLM and receive, from the LLM, a response to the prompt in the format of the first computer language. The system can parse the response in the first computer language to identify at least an indication of one or more recommended data transformations. The system can generate, based on the indication of the one or more recommended data transformations, the data pipeline using a second computer language.

Abstract: System and method for generating and displaying data pipelines according to certain embodiments. For example, a method includes: receiving a natural language (NL) query; receiving a model result generated based on the NL query, the model result including a query in a standard query language, the model result being generated using one or more computing models; and generating the data pipeline based at least in part on the query in the standard query language, the data pipeline comprising one or more data pipeline elements, at least one data pipeline element of the one or more pipeline elements being corresponding to a query component of the query in the standard query language.

Abstract: A method of determining and displaying metadata that represents provenance of columns in a data store comprises receiving query data expressed in a human-readable language and describing one or more transformations of data tables of a distributed database system or columns of the data tables; parsing the query data to create a data structure for unresolved transforms in which the data tables are not matched with the columns; based in part upon table schema metadata describing the columns of the data tables, creating an additional data structure for resolved transforms, the additional data structure matching the data tables with the columns and specifying inputs and outputs of each transformation of the one or more transformations; based on information in the additional data structure, creating and causing display of a visual graph of the columns of the data tables and relationships between the columns, relating to the one or more transformations.

Abstract: A system for enabling testing a target data pipeline within a graphical user interface (GUI) is disclosed. The system is programmed to present a GUI. The GUI shows a data pipeline including data transforms that are related based on data dependencies as a graph. The GUI then accept user inputs to identify a target pipeline from the data pipeline and set up a test for the target pipeline. The user inputs include interacting with the graph and providing test inputs and expected outputs for the target data pipeline, including specifying in each expected output for a test output one or more assertions related to the test output. The GUI further accepts user inputs to execute the test and review test results. The GUI facilitates specifying rich assertions, determining the source of a test failure, and updating the target data pipeline or the test based on the test results.

Abstract: Computing systems methods, and non-transitory storage media are provided for retrieving information regarding an operation to be performed by a platform, performing a preliminary validation of the operation, generating details regarding the preliminary validation, transmitting at least a subset of the details of the preliminary validation to the platform, and populating the generated details on an interface. If the preliminary validation fails, the platform refrains from performing the operation. Furthermore, the logic describing the operation can be executed on different platforms and is not bound or limited to one platform.

Abstract: Computer-implemented systems and methods are disclosed, including systems and methods utilizing language models for generating interactive workflow graphical user interfaces. A computer-implemented method may include: accessing one or more data stores that store a configuration of a software application; determining, based on the configuration of the software application, that first one or more actions and first one or more data object types are associated with the software application; based at least on the first one or more actions and the first one or more data object types associated with the software application, generating an interactive graphical representation comprising a first node representing the software application, a first user interface element of the first node, and a second user interface element of the first node.

Abstract: A method comprises storing, in a build catalog, for each update of a dataset, an entry including a branch identifier, an identifier and a version of the dataset, and build dependency information; receiving a first request to build a second branch having a first branch as a parent branch, the first branch being associated with a first version of a first driver program for building a first dataset from a set of child datasets, the second branch being associated with a second version of the first driver program; determining that the second branch does not have any version of a specific child dataset based on the build catalog; retrieving a latest version of the specific child dataset from the first branch; causing a build of the first dataset based on the latest version of the specific child dataset and the second version of the first driver program.

Abstract: Computing systems methods, and non-transitory storage media are provided for retrieving information regarding an operation to be performed by a platform, performing a preliminary validation of the operation, generating details regarding the preliminary validation, transmitting at least a subset of the details of the preliminary validation to the platform, and populating the generated details on an interface. If the preliminary validation fails, the platform refrains from performing the operation. Furthermore, the logic describing the operation can be executed on different platforms and is not bound or limited to one platform.

Abstract: A method of determining and displaying metadata that represents provenance of columns in a data store comprises receiving query data expressed in a human-readable language and describing one or more transformations of data tables of a distributed database system or columns of the data tables; parsing the query data to create a data structure for unresolved transforms in which the data tables are not matched with the columns; based in part upon table schema metadata describing the columns of the data tables, creating an additional data structure for resolved transforms, the additional data structure matching the data tables with the columns and specifying inputs and outputs of each transformation of the one or more transformations; based on information in the additional data structure, creating and causing display of a visual graph of the columns of the data tables and relationships between the columns, relating to the one or more transformations.

Abstract: Systems and methods are provided for improved graphical user interfaces. The system enables multiple separate applications, each of which may typically be in their own separate window or tab, to be interacted within a single window, such as a tab of a web browser application. The main window includes smaller sub-windows that can correspond to a distinct application with its own graphical user interface. A large sub-window within the main window is opened for the primary application where the user is currently interacting with a graphical user interface of the primary application. The user then is able to switch between applications (all within the same main window) and applications that are no longer being used can be minimized in smaller sub-windows off to the side of the primary sub-window. The system enables a user to drag and drop an item from one sub-window to another sub-window. Some of the interactions between the windows and data transformations are stored and can be visually presented in a graph.

Abstract: System and method for generating and displaying data pipelines according to certain embodiments. For example, a method includes: receiving a natural language (NL) query; receiving a model result generated based on the NL query, the model result including a query in a standard query language, the model result being generated using one or more computing models; and generating the data pipeline based at least in part on the query in the standard query language, the data pipeline comprising one or more data pipeline elements, at least one data pipeline element of the one or more pipeline elements being corresponding to a query component of the query in the standard query language.

Abstract: System and method for generating and displaying data pipelines according to certain embodiments. For example, a method includes: receiving a natural language (NL) query; receiving a model result generated based on the NL query, the model result including a query in a standard query language, the model result being generated using one or more computing models; and generating the data pipeline based at least in part on the query in the standard query language, the data pipeline comprising one or more data pipeline elements, at least one data pipeline element of the one or more pipeline elements being corresponding to a query component of the query in the standard query language.

Abstract: A method comprises storing first transaction data for a first transaction producing a first version of a first dataset, second transaction data for a second transaction transforming a first version of the first dataset to a second version of the first dataset, and third transaction data for a third transaction transforming the second version of the first dataset to a third version of the first dataset; storing dependency information indicating a first dependency of the third transaction on the second transaction and a second dependency of the second transaction on the first transaction; receiving a first instruction to revoke a first permission of a first user to access the second version of the first dataset; automatically revoking a certain permission of the first user to access the third version of the first dataset based on the dependency information.

Abstract: System and method for generating and displaying data pipelines according to certain embodiments. For example, a method includes: receiving a natural language (NL) query; receiving a model result generated based on the NL query, the model result including a query in a standard query language, the model result being generated using one or more computing models; and generating the data pipeline based at least in part on the query in the standard query language, the data pipeline comprising one or more data pipeline elements, at least one data pipeline element of the one or more pipeline elements being corresponding to a query component of the query in the standard query language.

Abstract: A resource dependency system may track data dependencies and data transformations for individual columns of the data sets over the span of the data pipeline (referred to as a provenance or lineage of a column). Column provenance/lineage can be logged using metadata or graph-like data structures, which the resource dependency system can generate, store, manage, and access. Column provenance/lineage can be used to generate user interfaces displaying visual node graphs with columns as nodes and the data dependencies and data transformations associated with the columns as edges between the nodes.

Abstract: A resource dependency system and its associated user interfaces, used for tracking data dependencies and data transformations between resources, may display visual node graphs with resources as nodes and the data dependencies and data transformations associated with the columns as edges between the nodes. The nodes representing the resources may be displayed differently based on relevant differences in the resources they represent, which can be set through various selectable criteria and schemes.