Abstract: Solutions for optimizing job runtimes via prediction-based token allocation includes receiving training data comprising historical run data, the historical run data comprising job characteristics, runtime results, and a token count for each of a plurality of prior jobs, and the job characteristics comprising an intermediate representation and job graph data; based at least on the training data, training a token estimator, the token estimator comprising a machine learning (ML) model; receiving job characteristics for a user-submitted job; based at least on the received job characteristics, generating, with the token estimator, token prediction data for the user-submitted job; selecting a token count for the user-submitted job, based at least on the token prediction data; identifying the selected token count to an execution environment; and executing, with the execution environment, the user-submitted job in accordance with the selected token count.

Abstract: Shuffling of into partitions by first grouping input vertices of a limited number. Each group of input vertices may then be simply shuffled into a corresponding group of intermediate vertices, such as by broadcasting. A second grouping occurs in which the intermediate vertices are grouped by partition. The intermediate vertices then shuffle into corresponding output vertices for the respective partitions of that group. If the intermediate vertices are still too large, then this shuffling may involve recursively performing the shuffling just described, until ultimately the number of intermediate vertices shuffling into the output vertices is likewise limited. Thus, the final shuffling into the output vertices might also be simply performed by broadcasting.

Abstract: Shuffling of into partitions by first grouping input vertices of a limited number. Each group of input vertices may then be simply shuffled into a corresponding group of intermediate vertices, such as by broadcasting. A second grouping occurs in which the intermediate vertices are grouped by partition. The intermediate vertices then shuffle into corresponding output vertices for the respective partitions of that group. If the intermediate vertices are still too large, then this shuffling may involve recursively performing the shuffling just described, until ultimately the number of intermediate vertices shuffling into the output vertices is likewise limited. Thus, the final shuffling into the output vertices might also be simply performed by broadcasting.

Abstract: Detection and use of relationships in tabular data is enhanced. A foreign-key-based relationship between two tables in memory is automatically discovered by a module. A view over table data is computationally augmented by adding an expandable hierarchical column. The expandable hierarchical column includes second table data that is related to rows of a first table by the discovered foreign-key-based relationship. A naming scheme for the added column is described. Sometimes the augmentation places an AddJoinColumn operator in a query expression tree; sometimes the tree is rewritten by moving a query operator or replacing a specified pattern of operators by a flat Join operation between two tables. Column expansion may be followed by reshaping data into a flat table, by getting a homogeneous query, by performing a join, or other operations.

Abstract: Providing for read/write access to captured free variables expressed as environment data objects is disclosed herein. By way of example, free variables can be rewritten as a field of a private class, and accessed via an instance of the private class captured within the environment data object. In such a manner, no need of a placeholder or proxy function is required to access the free variable. Consequently, the subject innovation provides a simplified method for capturing free variables within expression trees and providing read/write access to such variables.

Abstract: A method, system, and computer product for processing information embedded in a text file with a grammar programming language is provided. A text file is parsed according to a set of rules and candidate textual shapes corresponding to potential interpretations of the text file are provided by compiling a script. An output is provided, which may include either a processed value corresponding to a particular textual shape, or a textual representation of the text file that includes generic data structures that facilitate providing any of the candidate textual shapes, where the generic data structures are a function of the set of rules.

Abstract: A method and system for mapping between constructs in a domain modeling language and a relational storage language is provided. A source code authored in a source language is received and a set of constructs in the source code are identified. The set of constructs in the source code are mapped to a set of constructs in a target language. The source code is then compiled into a target code authored in the target language such that one of the source code or target code include a declarative constraint-based and/or order-independent execution model.

Abstract: Providing for read/write access to captured free variables expressed as environment data objects is disclosed herein. By way of example, free variables can be rewritten as a field of a private class, and accessed via an instance of the private class captured within the environment data object. In such a manner, no need of a placeholder or proxy function is required to access the free variable. Consequently, the subject innovation provides a simplified method for capturing free variables within expression trees and providing read/write access to such variables.