Abstract: Techniques are disclosed herein for generating a synthetic fine-tuning dataset to train a machine learning model for converting natural language to logical form. In one aspect, a method is disclosed that includes: accessing domain data comprising a set of domain identifiers and generating a set of synthetic training examples by generating a synthetic database schema based on the domain data, synthetic database content based on the synthetic database schema, a synthetic natural language utterance, and a synthetic logical form. A machine learning model may be fine-tuned with the set of synthetic training examples. A subset rehearsal training examples from a rehearsal dataset may be identified using execution-based validation and used to fine-tune the machine learning model.

Abstract: Techniques are disclosed herein to evaluating generative models for generating logical form queries. The techniques may include obtaining a query, wherein the query includes a natural language utterance; generating, by a generative model, a logical form query that corresponds to the natural language utterance; generating a normalized logical form query by modifying the logical form query based on a schema that defines how data is stored; executing the normalized logical form query on a data store to obtain results; identifying whether the results include at least reference results associated with execution of a reference logical form query on the data store; and indicating, in response to identifying that the results include at least the reference results, that the logical form query is acceptable.

Abstract: Techniques are provided for generating context-aware, multi-turn natural language to SQL training data for database query models. The method involves accessing reference data that includes natural language utterances and corresponding queries in a programming query language. An AI model may generate a set of follow-up natural language utterances based on contextual information related to an initial utterance, and may rewrite these utterances to produce self-contained, multi-turn queries. An AI model can retrieve examples from the reference data that are semantically similar to the multi-turn utterances, serving as in-context examples. An AI model may then generate candidate queries based on the rewritten utterances and the retrieved examples. A dual stage validation process can be executed to evaluate the candidate queries, generating performance reports indicating which queries pass validation.

Abstract: Techniques are disclosed herein for generating augmented training examples to fine-tune a machine learning model for converting natural language to logical form. In one aspect, a method is disclosed that includes: accessing a set of training examples, wherein each training example includes: (i) an original prompt having a database schema and (ii) a programming language completion corresponding to the original prompt, generating a set of augmented training examples based on the set of training examples, generating a fine-tuning dataset comprising the training examples and the augmented training examples, and fine-tuning a model with the fine-tuning dataset.

Abstract: Techniques are provided for a clustering algorithm to harvest in-context examples from Text-to-SQL logs. The method includes receiving vector representations of natural language utterances and corresponding queries in a programming query language. The clustering algorithm may be reparametrized using a training dataset by sampling the vector representations, estimating local radius parameter values from a distribution of pairwise distances, and iteratively executing the clustering algorithm while incrementally relaxing density constraints to identify paraphrases and unique queries. Hyperparameter tuning may be performed by adjusting clustering parameters based on evaluation metrics to obtain a fine-tuned algorithm. The method can further augment examples with ground truth examples and execute the fine-tuned clustering algorithm on the augmented set to generate clustered data points representing groups of paraphrases and unique queries.

Abstract: Techniques are disclosed herein for generating a distillation dataset to train a machine learning model for converting natural language to logical form. In one aspect, a method is disclosed that includes: accessing customization information comprising a set of reference natural language queries, generating a set of synthetic natural language queries based on the customization information and database schema information and generating a set of distillation training examples. Generating the set of distillation training examples can include generating a synthetic programming language query for each synthetic natural language query. Each distillation training example comprises (i) a synthetic natural language query, (ii) a database schema from the database schema information associated with the synthetic natural language query, and (iii) a synthetic programming language query corresponding with the synthetic natural language query.

Abstract: Disclosed are approaches for automatically generating and refining instructions for converting natural language questions to a logical form such as structured query language (SQL). An initial evaluation using a baseline prompt identifies errors in generated SQL queries. These errors are clustered to identify common failure patterns. For a selected error cluster, candidate instructions are generated to address the pattern. The candidate instructions are then validated to confirm they improve accuracy for the error cluster without degrading overall performance across a training set. Upon successful validation, the instructions are incorporated. This process of identifying error clusters, generating instructions, and validating them is repeated iteratively to progressively enhance the instruction set. The resulting instructions may also be automatically compressed to reduce length while maintaining performance, thereby improving operational efficiency.

Abstract: The present disclosure relates to machine learning techniques for In-Context-Learning (ICL) with pattern-based retrieval for the task of converting Natural Language (NL) to Structured Query Language (SQL). Aspects are directed towards acquiring a natural language utterance and a database schema, searching, using at least a portion of the natural language utterance as a key, a memory bank for one or more in-context examples that are relevant to the key, generating a prompt comprising the natural language query, the database schema, and the one or more in-context examples, transmitting the prompt to a first pretrained generative artificial intelligence model, receiving, from the first pretrained generative artificial intelligence model, a logical form corresponding to the natural language utterance based at least in part on the prompt, executing the logical form on a database to obtain a query result, and providing the query result to a user.

Abstract: Techniques are disclosed herein for automatically curating preference alignment training (PAT) data and subsequently using the PAT data to train a machine-learning model used to perform natural language-to-query language tasks. A first PAT dataset is generated using a first trained LLM in an execution-based data generation technique. A second PAT dataset is generated using a second trained LLM in a LLM-based data generation technique. The second PAT dataset is populated with synthetic data points generated by the second trained LLM. Each data point of first PAT dataset and the second PAT dataset comprises a natural language query and an executable and non-executable query language statement. The first PAT dataset and the second PAT dataset can be combined and filtered to generate a PAT dataset that is optimized for training a machine-learning model to overcome one or more weaknesses exhibited when generating query language statements responsive to natural language questions.

Abstract: Techniques are disclosed herein for automatically curating preference alignment training (PAT) data and subsequently using the PAT data to train a machine-learning model used to perform natural language-to-query language tasks. A first PAT dataset is generated using a first trained LLM in an execution-based data generation technique. A second PAT dataset is generated using a second trained LLM in a LLM-based data generation technique. The second PAT dataset is populated with synthetic data points generated by the second trained LLM. Each data point of first PAT dataset and the second PAT dataset comprises a natural language query and an executable and non-executable query language statement. The first PAT dataset and the second PAT dataset can be combined and filtered to generate a PAT dataset that is optimized for training a machine-learning model to overcome one or more weaknesses exhibited when generating query language statements responsive to natural language questions.

Abstract: Techniques are disclosed herein to augment a prompt for a generative model. The techniques may include accessing a set of training examples, generating a logical form query that corresponds to the natural language utterance for each example of the set of training examples, evaluating performance of the generative model using the logical form query generated for each example of the set of training examples, identifying, based on the evaluating, a subset of rejected examples from the set of examples, grouping the subset of rejected examples into at least one or more groups of rejected examples based on similarity between embedding vectors for the subset of rejected examples, generating, by a generative model, instructions for each group of the one or more groups of rejected examples, and modifying the baseline prompt to include the instructions for each group of the one or more groups of rejected examples.

Abstract: Techniques are disclosed herein towards a process for enhancing generative model robustness. The process includes accessing a training data set comprising examples, each with a natural language utterance and a database schema. Each example is augmented by generating augmentation prompts with perturbation instructions, which direct the generative model to modify the prompt using one or more categories of perturbations. These perturbations may alter the natural language utterance, the database schema, or both, resulting in variant prompts. The augmented examples are added to an augmented training data set. Using these augmented training examples, a pre-trained generative model is fine-tuned, yielding a fine-tuned generative model capable of accurately and consistently producing structured queries in response to diverse natural language inputs and schema configurations.

Abstract: Techniques are disclosed herein towards logical form data-time expression generation. For example, methods are provided for receiving a natural language (NL) utterance, generating a prompt including the NL utterance and instructions to transform the NL utterance into a logical form query including a coded-form expression, generating, by a generative model based on the prompt, a logical form query including a coded-form expression, transforming the coded-form expression into a period definition expression by executing the coded-form expression with one or more pre-defined period-definition content items, updating the logical form query to include the period definition expression by replacing the coded-form expression with the period definition expression, and providing at least one of i) the updated logical form query or ii) a query result obtained based on the updated logical form query, to a client system.

Abstract: Techniques for named entity bias detection and mitigation for sentence sentiment analysis. In one particular aspect, a method is provided that includes obtaining a training set of labeled examples for training a machine learning model to classify sentiment, preparing a list of named entities using one or more data sources, for each example in the training set of labeled examples with a named entity, replacing the named entity with a corresponding entity type tag to generate a labeled template data set, executing a sampling process for each entity type t within the labeled template data set to generate a augmented invariance data set comprising one or more invariance groups having labeled examples for each entity type t, and training the machine learning model using labeled examples from the augmented invariance data set.

Abstract: Techniques disclosed herein relate generally to language detection. In one particular aspect, a method is provided that includes obtaining a sequence of n-grams of a textual unit; using an embedding layer to obtain an ordered plurality of embedding vectors for the sequence of n-grams; using a deep network to obtain an encoded vector that is based on the ordered plurality of embedding vectors; and using a classifier to obtain a language prediction for the textual unit that is based on the encoded vector. The deep network includes an attention mechanism, and using the embedding layer to obtain the ordered plurality of embedding vectors comprises, for each n-gram in the sequence of n-grams: obtaining hash values for the n-gram; based on the hash values, selecting component vectors from among the plurality of component vectors; and obtaining an embedding vector for the n-gram that is based on the component vectors.

Abstract: Techniques disclosed herein relate generally to text classification and include techniques for fusing word embeddings with word scores for text classification. In one particular aspect, a method for text classification is provided that includes obtaining an embedding vector for a textual unit, based on a plurality of word embedding vectors and a plurality of word scores. The plurality of word embedding vectors includes a corresponding word embedding vector for each of a plurality of words of the textual unit, and the plurality of word scores includes a corresponding word score for each of the plurality of words of the textual unit. The method also includes passing the embedding vector for the textual unit through at least one feed-forward layer to obtain a final layer output, and performing a classification on the final layer output.

Abstract: The present disclosure relates to manufacturing training and testing data by leveraging data augmentation techniques to generate examples of long context database schemas. Aspects are directed towards accessing a training dataset comprising training examples where each training example may include i) a prompt including a natural language utterance and a database schema having one or more tables, and ii) a gold logical form corresponding to the natural language utterance, combining the tables from the database schemas in the training examples may generate a combined database schema set, generating a set of long context training examples based on the training dataset and the combined database schema set, and incorporating the long context database schema into the selected training example to generate a long context training example to train a generative artificial intelligence model with at least the set of long context training examples to generate a trained generative artificial intelligence model.

Abstract: Techniques for augmentation and batch balancing of training data to enhance negation and fairness of a machine learning model. In one particular aspect, a method is provided that includes obtaining a training set of labeled examples for training a machine learning model to classify sentiment, searching the training set of labeled examples or an unlabeled corpus of text on target domains for sentiment examples having negation cues, sentiment laden words, words with sentiment prefixes or suffixes, or a combination thereof, rewriting the sentiment examples to create negated versions thereof and generate a labeled negation pair data set, and training the machine learning model using labeled examples from the labeled negation pair data set.

Abstract: The present disclosure relates to machine learning techniques for In-Context-Learning (ICL) with pattern-based retrieval for the task of converting Natural Language (NL) to Structured Query Language (SQL). Aspects are directed towards acquiring a natural language utterance and a database schema, searching, using at least a portion of the natural language utterance as a key, a memory bank for one or more in-context examples that are relevant to the key, generating a prompt comprising the natural language query, the database schema, and the one or more in-context examples, transmitting the prompt to a first pretrained generative artificial intelligence model, receiving, from the first pretrained generative artificial intelligence model, a logical form corresponding to the natural language utterance based at least in part on the prompt, executing the logical form on a database to obtain a query result, and providing the query result to a user.

Abstract: A method includes accessing document including sentences, document being associated with configuration flag indicating whether ABSA, SLSA, or both are to be performed; inputting the document into language model that generates chunks of token embeddings for the document; and, based on the configuration flag, performing at least one from among the ABSA and the SLSA by inputting the chunks of token embeddings into a multi-task model. When performing the SLSA, a part of token embeddings in each of the chunks is masked, and the masked token embeddings do not belong to a particular sentence on which the SLSA is performed.