Abstract: Systems and methods for automatic evaluation of the quality of NLG outputs. In some aspects of the technology, a learned evaluation model may be pretrained first using NLG model pretraining tasks, and then with further pretraining tasks using automatically generated synthetic sentence pairs. In some cases, following pretraining, the evaluation model may be further fine-tuned using a set of human-graded sentence pairs, so that it learns to approximate the grades allocated by the human evaluators.

Abstract: Systems and methods for automatic evaluation of the quality of NLG outputs. In some aspects of the technology, a learned evaluation model may be pretrained first using NLG model pretraining tasks, and then with further pretraining tasks using automatically generated synthetic sentence pairs. In some cases, following pretraining, the evaluation model may be further fine-tuned using a set of human-graded sentence pairs, so that it learns to approximate the grades allocated by the human evaluators.

Abstract: Systems and methods for automatic evaluation of the quality of NLG outputs. In some aspects of the technology, a learned evaluation model may be pretrained first using NLG model pretraining tasks, and then with further pretraining tasks using automatically generated synthetic sentence pairs. In some cases, following pretraining, the evaluation model may be further fine-tuned using a set of human-graded sentence pairs, so that it learns to approximate the grades allocated by the human evaluators. In some cases, following fine-tuning, the learned evaluation model may be distilled into a student model.

Abstract: Systems and methods for automatic evaluation of the quality of NLG outputs. In some aspects of the technology, a learned evaluation model may be pretrained first using NLG model pretraining tasks, and then with further pretraining tasks using automatically generated synthetic sentence pairs. In some cases, following pretraining, the evaluation model may be further fine-tuned using a set of human-graded sentence pairs, so that it learns to approximate the grades allocated by the human evaluators. In some cases, following fine-tuning, the learned evaluation model may be distilled into a student model.

Abstract: Systems and methods for automatic evaluation of the quality of NLG outputs. In some aspects of the technology, a learned evaluation model may be pretrained first using NLG model pretraining tasks, and then with further pretraining tasks using automatically generated synthetic sentence pairs. In some cases, following pretraining, the evaluation model may be further fine-tuned using a set of human-graded sentence pairs, so that it learns to approximate the grades allocated by the human evaluators.

Abstract: Systems and methods for automatic evaluation of the quality of NLG outputs. In some aspects of the technology, a learned evaluation model may be pretrained first using NLG model pretraining tasks, and then with further pretraining tasks using automatically generated synthetic sentence pairs. In some cases, following pretraining, the evaluation model may be further fine-tuned using a set of human-graded sentence pairs, so that it learns to approximate the grades allocated by the human evaluators.

Abstract: Scoring candidate evidence passages for criteria validation. Evidence data associated with a criteria, such that the evidence data entries include a decision indicator indicating that the criteria is either met or not met by the evidence data, is collected. Candidate evidences, making up a corpus of data associated with the criteria, against which the criteria is to be validated, are generated. Each candidate evidence is evaluated against the evidence data. A score indicating the validity of the criteria with respect to the candidate evidence is generated, based on the decision indicators associated with the evidence data entries.

Abstract: Scoring candidate evidence passages for criteria validation. Evidence data associated with a criteria, such that the evidence data entries include a decision indicator indicating that the criteria is either met or not met by the evidence data, is collected. Candidate evidences, making up a corpus of data associated with the criteria, against which the criteria is to be validated, are generated. Each candidate evidence is evaluated against the evidence data. A score indicating the validity of the criteria with respect to the candidate evidence is generated, based on the decision indicators associated with the evidence data entries.

Abstract: Systems and methods for automatic evaluation of the quality of NLG outputs. In some aspects of the technology, a learned evaluation model may be pretrained first using NLG model pretraining tasks, and then with further pretraining tasks using automatically generated synthetic sentence pairs. In some cases, following pretraining, the evaluation model may be further fine-tuned using a set of human-graded sentence pairs, so that it learns to approximate the grades allocated by the human evaluators. In some cases, following fine-tuning, the learned evaluation model may be distilled into a student model.

Abstract: Systems and methods for automatic evaluation of the quality of NLG outputs. In some aspects of the technology, a learned evaluation model may be pretrained first using NLG model pretraining tasks, and then with further pretraining tasks using automatically generated synthetic sentence pairs. In some cases, following pretraining, the evaluation model may be further fine-tuned using a set of human-graded sentence pairs, so that it learns to approximate the grades allocated by the human evaluators.

Abstract: One or more table content documents (TCDs) can be constructed for a set of tabular data or portion thereof. A set of query features corresponding to a question can be matched to one or more TCDs. A respective candidate answer can be generated for each of the one or more TCDs having a set of features matching the set of query features above a threshold. Zero or more candidate answers can be output to a user consumable data object.

Abstract: Generating distractors for text-based MCT items. An MCT item stem is received. The stem is transmitted to a QA system and a plurality of candidate answers related to the stem is received from the QA system. Incorrect answers in the plurality of candidate answers are identified. Textual features are extracted from the stem. A set of semantic criteria associated with the extracted textual features is generated. Based on the generated semantic criteria, a subset of the incorrect candidate answers is selected.

Abstract: Software that extracts contextually relevant terms from a text sample (or corpus) by performing the following steps: (i) identifying a first term from a corpus, based, at least in part, on a set of initial contextual characteristic(s), where each initial contextual characteristic of the set of initial contextual characteristic(s) relates to the contextual use of at least one category related term of a set of category related term(s) in the corpus; (ii) adding the first term to the set of category related term(s), thereby creating a revised set of category related term(s) and a set of first term contextual characteristic(s), where each first term contextual characteristic of the set of first term contextual characteristic(s) relates to the contextual use of the first term in the corpus; and (iii) identifying a second term from the corpus, based, at least in part, on the set of first term contextual characteristic(s).

Abstract: Scoring candidate evidence passages for criteria validation. Evidence data associated with a criteria, such that the evidence data entries include a decision indicator indicating that the criteria is either met or not met by the evidence data, is collected. Candidate evidences, making up a corpus of data associated with the criteria, against which the criteria is to be validated, are generated. Each candidate evidence is evaluated against the evidence data. A score indicating the validity of the criteria with respect to the candidate evidence is generated, based on the decision indicators associated with the evidence data entries.

Abstract: Scoring candidate evidence passages for criteria validation. Evidence data associated with a criteria, such that the evidence data entries include a decision indicator indicating that the criteria is either met or not met by the evidence data, is collected. Candidate evidences, making up a corpus of data associated with the criteria, against which the criteria is to be validated, are generated. Each candidate evidence is evaluated against the evidence data. A score indicating the validity of the criteria with respect to the candidate evidence is generated, based on the decision indicators associated with the evidence data entries.

Abstract: Generating distractors for text-based MCT items. An MCT item stem is received. The stem is transmitted to a QA system and a plurality of candidate answers related to the stem is received from the QA system. Incorrect answers in the plurality of candidate answers are identified. Textual features are extracted from the stem. A set of semantic criteria associated with the extracted textual features is generated. Based on the generated semantic criteria, a subset of the incorrect candidate answers is selected.

Abstract: Generating distractors for text-based MCT items. An MCT item stem is received. The stem is transmitted to a QA system and a plurality of candidate answers related to the stem is received from the QA system. Incorrect answers in the plurality of candidate answers are identified. Textual features are extracted from the stem. A set of semantic criteria associated with the extracted textual features is generated. Based on the generated semantic criteria, a subset of the incorrect candidate answers is selected.

Abstract: One or more table content documents (TCDs) can be constructed for a set of tabular data or portion thereof. A set of query features corresponding to a question can be matched to one or more TCDs. A respective candidate answer can be generated for each of the one or more TCDs having a set of features matching the set of query features above a threshold. Zero or more candidate answers can be output to a user consumable data object.

Abstract: One or more table content documents (TCDs) can be constructed for a set of tabular data or portion thereof. A set of query features corresponding to a question can be matched to one or more TCDs. A respective candidate answer can be generated for each of the one or more TCDs having a set of features matching the set of query features above a threshold. Zero or more candidate answers can be output to a user consumable data object.

Abstract: Scoring candidate evidence passages for criteria validation. Evidence data associated with a criteria, such that the evidence data entries include a decision indicator indicating that the criteria is either met or not met by the evidence data, is collected. Candidate evidences, making up a corpus of data associated with the criteria, against which the criteria is to be validated, are generated. Each candidate evidence is evaluated against the evidence data. A score indicating the validity of the criteria with respect to the candidate evidence is generated, based on the decision indicators associated with the evidence data entries.