Abstract: Systems and methods are provided for generating nested queries from natural language queries. In particular, system and methods are provided to implement natural language interfaces to databases (NLIDB) frameworks which are configured to apply intelligent reasoning over domain semantics to detect and generate nested queries across different domains without the need for domain specific training or utilizing domain-specific semantic templates for mapping a natural language query to a structured query.

Abstract: Methods, systems, and computer program products for determining model-related bias associated with training data are provided herein. A computer-implemented method includes obtaining, via execution of a first model, class designations attributed to data points used to train the first model; identifying any of the data points associated with an inaccurate class designation and/or a low-confidence class designation; training a second model using the data points from the dataset, but excluding the identified data points; determining bias related to at least a portion of those data points used to train the second model by: modifying one or more of the data points used to train the second model; executing the first model using the modified data points; and identifying a change to one or more class designations attributed to the modified data points as compared to before the modifying; and outputting identifying information pertaining to the determined bias.

Abstract: Methods, systems, and computer program products for improving trustworthiness of artificial intelligence models in presence of anomalous data are provided herein. A method includes obtaining a machine learning model and a set of training data; determining one or more anomalous data points in said set of training data; for a given one of said anomalous data points, identifying attributes that decrease confidence with respect to at least one output of said machine learning model; determining that a root cause of said decreased confidence corresponds to one of: a class imbalance issue related to said at least one attribute, a confused class issue related to said at least one attribute, a low density issue related to said at least one attribute, and an adversarial issue related to said at least one attribute; and performing step(s) to improve said confidence based at least in part on said determined root cause.

Abstract: Methods, systems, and computer program products for determining data representative of bias within a model are provided herein. A computer-implemented method includes obtaining a first dataset on which a model was trained, wherein the first dataset contains protected attributes, and a second dataset on which the model was trained, wherein the protected attributes have been removed from the second dataset; identifying, for each of the one or more protected attributes in the first dataset, one or more attributes in the second dataset correlated therewith; determining bias among at least a portion of the identified correlated attributes; and outputting, to at least one user, identifying information pertaining to the one or more instances of bias.

Abstract: Methods, systems and computer program products for automated test case generation are provided herein. A computer-implemented method includes selecting sample input data as a test case for a system under test, executing the test case on the system under test to obtain a result, and applying the result to a local explainer function to obtain at least a portion of a corresponding decision tree. The method further includes determining at least one path constraint from the decision tree, solving the path constraint to obtain a solution, and generating at least one other test case for the system under test based at least in part on the solution of the path constraint. The steps of the method are illustratively repeated in each of one or more additional iterations until at least one designated stopping criterion is met. The resulting test cases form a test suite for testing of a deep neural network (DNN) or other system.

Abstract: Methods, systems and computer program products for automated testing of dialog systems are provided herein. A computer-implemented method includes receiving information pertaining to a given conversation workspace of an automated dialog system and identifying test case inputs to the automated dialog system, the test case inputs comprising user input for the given conversation workspace that has portions thereof modified and which the automated dialog system maps to a different intent and/or a different entity relative to the user input. The method further includes generating human-interpretable explanations of mappings of portions of the test case inputs to the different intent and/or entity, generating suggestions for modifying intents, entities and dialog flows of the given conversation workspace such that the test case inputs map to the same intent and/or the same entity as their corresponding user input, and outputting the suggestions and the human-interpretable explanations to a user.

Abstract: Methods, systems and computer program products for automated testing of dialog systems are provided herein. A computer-implemented method includes receiving selection of a conversation workspace of the automated dialog system and identifying test case inputs to the automated dialog system, the test case inputs comprising example user input for the given conversation workspace that has portions thereof modified and which the automated dialog system maps to a different intent and/or a different entity relative to the example user input. The method further includes generating human-interpretable explanations of mappings of portions of the test case inputs to the different intent and/or entity, generating suggestions for modifying intents, entities and dialog flows of the given conversation workspace such that the test case inputs map to the same intent and/or the same entity as their corresponding example user input, and outputting the suggestions and the human-interpretable explanations to a user.

Abstract: Methods, systems, and computer program products for automated generation of test cases for analyzing natural-language-interface-to-database systems are provided herein.

Abstract: Methods, systems, and computer program products for domain aware explainable anomaly and drift detection for multi-variate raw data using a constraint repository are provided herein. A computer-implemented method includes obtaining a set of data and information indicative of a domain of said set of data; obtaining constraints from a domain-indexed constraint repository based on said set of data and said information, wherein the domain-indexed constraint repository comprises a knowledge graph having a plurality of nodes, wherein each node comprises an attribute associated with at least one of a plurality of domains and constraints corresponding to the attribute; detecting anomalies in said set of data based on whether portions of said set of data violate said retrieved constraints; generating an explanation corresponding to each of the anomalies that describe the attributes corresponding to the violated constraints; and outputting an indication of the anomalies and the corresponding explanation.

Abstract: Methods, systems and computer program products for automated test case generation are provided herein. A computer-implemented method includes selecting sample input data as a test case for a system under test, executing the test case on the system under test to obtain a result, and applying the result to a local explainer function to obtain at least a portion of a corresponding decision tree. The method further includes determining at least one path constraint from the decision tree, solving the path constraint to obtain a solution, and generating at least one other test case for the system under test based at least in part on the solution of the path constraint. The steps of the method are illustratively repeated in each of one or more additional iterations until at least one designated stopping criterion is met. The resulting test cases form a test suite for testing of a deep neural network (DNN) or other system.

Abstract: One embodiment provides a method, including: receiving, within a conversational agent creation framework used to create conversational agents that perform negotiations on behalf of users, (i) a selection of one of a plurality of conversational agent roles (ii) negotiation constraints for the selected role, and (iii) an object for the negotiation; creating a conversational agent (i) having the selected role and (ii) programmed with the negotiation constraints; identifying at least one other conversational agent (i) having an opposing role with respect to the object and (ii) being programmed with opposing role negotiation constraints; and conducting a conversational session between the conversational agent and the at least one other conversational agent, wherein the conversational session comprises the negotiation for the object and wherein the conversational agent and the at least one other conversational agent perform the negotiation in view of the negotiation constraints and the opposing role negotiation con

Abstract: One embodiment provides a method, including: identifying at least one protected attribute of a task of annotating information, wherein the task is assigned to at least one crowdsourced worker for performance of the task; generating at least one question for detecting bias of the at least one crowdsourced worker with respect to the at least one protected attribute, the at least one question comprising a previously provided annotation; providing the at least one question to the at least one crowdsourced worker at a period during performance of the task; computing a bias of the at least one crowdsourced worker with respect to the at least one protected attribute by comparing (i) an annotation provided by the crowdsourced worker to the at least one question and (ii) the previously provided annotation; and updating a profile of the at least one crowdsourced worker with the computed bias.

Abstract: Methods, systems, and computer program products for active explanation guided learning are provided herein. A computer-implemented method includes identifying a subset of training examples, from a set of training examples, based on at least one of (i) an uncertainty metric computed for each one of the training examples and (ii) an influence metric computed for each one of the training examples; outputting said subset of training examples to a user; obtaining, from the user, a user explanation for each training example in said subset of training examples, wherein each of the user explanations identifies at least one part of the corresponding training example; and training a machine learning model based at least in part on the user explanations, wherein said training comprises prioritizing the identified parts of the training examples in the subset.

Abstract: Methods, systems, and computer program products for bootstrapping of state-based dialog systems are provided herein. A computer-implemented method includes determining parameters for state automata by partitioning an ontology graph into sub-graphs and a knowledge graph into sub-graphs, wherein the ontology graph and the knowledge graph are based on a user question and domain knowledge pertaining to the user question; generating a structured query for each of the sub-graphs; determining intentions of a dialog pertaining to the at least one user question by translating each of the generated structured queries to a respective natural language query; creating one or more dialog states for each of the determined dialog intentions; creating one or more connecting dialog states between pairs of the created dialog states; and generating an automata dialog framework associated with the user question based on the created dialog states and the created connecting dialog states.

Abstract: One embodiment provides a method, including: receiving a natural language query from a user; identifying a plurality of interpretations for interpreting the natural language query, wherein the plurality of interpretations are based upon at least one ambiguity in the received natural language query; generating, for each of the plurality of interpretations, a plurality of example queries; generating, for each of the interpretations, both (i) an answer to the received natural language query and (ii) an answer to each of the generated plurality of example queries; and providing, to the user, (i) the generated answer for each interpretation of the natural language query and (ii) a plurality of question/answer pairs for each of the identified plurality of interpretations that assists in disambiguating the ambiguity, wherein each question/answer pair comprises at least one of the generated plurality of example queries and the corresponding generated answer to the example query.

Abstract: Embodiments are disclosed for correcting a natural language interface database (NLIDB) system. The techniques include receiving feedback indicating that an answer provided in response to a question for an NLIDB system is inaccurate. The techniques further include finding an ontology element for a datastore of the NLIDB system that matches to the feedback. The techniques also include selecting candidate annotations for the NLIDB system based on the ontology element and a data type of the ontology element. Additionally, the techniques include generating a question-answer (QA) pair for each of the candidate annotations. Further, the techniques include adding one of the candidate annotations to annotations for a natural language query (NLQ) engine of the NLIDB system based on a client verification of the QA pair.

Abstract: Techniques for the automatic semantic analysis and comparison of chatbot capabilities are disclosed. A first chatbot specification associated with a first chatbot is obtained that includes a first plurality of characteristics arranged in a plurality of categories. A second chatbot specification associated with a second chatbot is obtained that includes a second plurality of characteristics arranged in the plurality of categories. One or more differences between the first plurality of characteristics and the second plurality of characteristics for each of the plurality of categories are identified based at least in part on the first plurality of characteristics and the second plurality of characteristics. A natural language expression corresponding to the identified one or more differences is generated and presented to a user via a graphical user interface.

Abstract: Aspects of the present disclosure relate to automated ontology refinement based on query inputs and provided feedback. A query input is received for an ontology. Features of the query input are analyzed, wherein analyzation includes determining syntactical and semantic characteristics of the features of the query input. Based on the determined syntactical and semantic characteristics, ontological elements are classified for each feature of the query input. The ontological element for each feature of the query input is then compared to a set of ontological elements of the ontology. Based on the comparison, a response to the query input is received, along with a request for feedback regarding the response. Feedback is then received regarding the response. Based on the feedback, the ontology is analyzed to determine at least one deficiency of the ontology. The ontology is then refined to correct the at least one deficiency.

Abstract: Methods, systems, and computer program products for generating a data migration plan for in-place encryption of data are provided herein. A computer-implemented method includes receiving, from a user, a request to generate a migration plan for performing in-place encryption of data within a database, wherein the migration plan indicates periods of time in which portions of the data are to be encrypted; determining a set of constraints for performing the in-place encryption; generating the migration plan based at least in part on the set of constraints; and performing the in-place encryption of the data in accordance with the migration plan such that only a single copy of each of the portions is maintained during the in-place encryption, wherein the single copy comprises one of a plaintext copy of data corresponding to the portion, and an encrypted copy of data corresponding to the portion.

Abstract: One embodiment provides a method, including: receiving a dataset and a model corresponding to a bias checker, wherein the bias checker detects bias within both the dataset and the model, based upon a bias checking algorithm and a bias checking policy, wherein the dataset comprises a plurality of attributes; testing the bias checking algorithm of the bias checker by (i) generating test cases that modify the dataset by introducing bias therein and (ii) running the bias checker against the modified dataset; testing the bias checking policy of the bias checker by generating a plurality of test cases and running the bias checker against the plurality of test cases; and providing a notification to a user regarding whether the bias checker failed to indicate bias for one or more of the plurality of attributes.