Abstract: Automatic content generation by one or more computing devices can include receiving data comprising an original sentence with a grammar artifact of interest. Thereafter, a plurality of distractor candidates are generated based on the original sentence with the grammar artifact of interest. At least one machine learning-based language model then scores each of the distractor candidates. These scores characterize a likelihood of the corresponding distractor candidate being selected as part of an assessment by a subject. The distractor candidates can be filtered to result in a filtered list of distractor candidates from which the x top scoring distractor candidates can be selected. A grammar practice item is then generated based on the original sentence and the x top scoring distractor candidates. The grammar practice can then be provided. Related apparatus, systems, and articles are also described.

Abstract: Quality of a narrative is characterized by receiving data that includes a narrative text. This narrative text is then tokenized and events are extracted from the tokenized words. The extraction can use, in parallel, two or more different extraction techniques. The extracted events are then extracted so that a waveform can be generated based on the aggregated extracted events that characterizes a plurality of emotional arcs within the narrative text. Subsequently, a plurality of waveform elements are extracted from the waveform. The narrative quality (or other quality) of the narrative text is then scored based on the extracted plurality of waveform elements and using a machine learning model trained to correlate emotional arc waveforms with narrative quality scores. Related apparatus, systems, techniques and articles are also described.

Abstract: Data is received that includes a passage of text generated in response to a prompt which comprises a plurality of sentences. Thereafter, the passage of text is tokenized into a plurality of tokens each corresponding to a different word in the passage of text. A first classification head of an adaptive fine-tuned transforms classifies each of the tokens into one of a plurality of classes. A second classification head of the adaptive fine-tuned transformer model classifies each of the sentences as either including or not including an argument. Data can then be provided which characterizes the first and second classifications. Related apparatus, systems, techniques and articles are also described.

Abstract: Data is received by an automated spoken language learning and assessment system that includes a passage of text comprising a response to stimulus material. Thereafter, at least one machine learning model is used to detect absent key points within the passage of text and/or location spans of key points in the passage of text. The at least one machine learning model can be trained using a corpus with annotated key points and a span for each key point. In addition, each of the detected key points is scored by at least one key point quality model to result in a corresponding key point score. Diagnostic feedback targeting content development skills is then determined based on the detecting and using the key point scores. Data can then be provided which characterizes such diagnostic feedback. Related apparatus, systems, techniques and articles are also described.

Abstract: Data such as unstructured text is received that includes a sequence of sentences. This received data is then tokenized into a plurality of tokens. The received data is segmented using a hierarchical transformer network model including a token transformer, a sentence transformer, and a segmentation classifier. The token transformer contextualizes tokens within sentences and yields sentence embeddings. The sentences transformer contextualizes sentence representations based on the sentence embedddings. The segmentation classifier predicts segments of the received data based on the contextualized sentence representations. Data can be provided which characterizes the segmentation of the received data. Related apparatus, systems, techniques and articles are also described.

Abstract: Systems and methods are provided for automatic evaluation of argument critique essays written by young students in response to prompts. A transformer pre-trained for natural language processing is employed as a machine learning model, which is fine-tune with a first training dataset comprising unannotated argument critique essays written by college students, and then fine-tuned with a second training dataset comprising annotated argument critique essays written by middle school students, where each sentence in the second training dataset is annotated for the presence of valid critiques to prompts. The fine-tuned machine learning model is used to classify each sentence in an essay to be evaluated as either containing a valid critique or not.

Abstract: A computer-implemented technique for characterizing lexical concreteness in narrative includes receiving data encapsulating narrative text having a plurality of words. Thereafter, the function words can be removed from the narrative text to result in only content words. A concreteness score can then be assigned to each content word by polling a database to identify matching words and to use concreteness scores associated with such matching words as specified by the database. Data can then be provided which characterizes the assigned concreteness scores. Related apparatus, systems, techniques and articles are also described.

Abstract: Human-machine dialog is characterized by receiving data comprising a recording of an individual interacting with a dialog application simulating a conversation. Thereafter, the received data is parsed using automated speech recognition to result in text comprising a plurality of words. Features are extracted from the parsed data and then input an ensemble of different machine learning models each trained to generate a score characterizing a plurality of different dialog constructs. Thereafter, scores generated by the machine learning models for each of the dialog constructs are fused. A performance score is then generated based on the fused scores which characterizes a conversational proficiency of the individual interacting with the dialog application. Data can then be provided which includes or otherwise characterizes the generated score. Related apparatus, systems, techniques and articles are also described.

Abstract: Systems and methods are provided for processing a group of essays to develop a classifier that detects nonsensical computer-generated essays. A data structure associated with a group of essays is accessed, wherein the group of essays includes nonsensical computer-generated essays and good-faith essays. Both the nonsensical computer-generated essays and the good-faith essays are assigned feature values. The distribution of feature values between the nonsensical computer-generated essays and the good-faith essays is measured. A classifier that detects whether an essay is a nonsensical computer-generated essay is developed, wherein the classifier is developed using the distribution of feature values.

Abstract: An electronic audio file is received that comprises spontaneous speech responsive to a prompt in a non-native language of a speaker. Thereafter, the electronic audio file is parsed into a plurality of spoken words. The spoken words are then normalized to remove stop words and disfluencies. At least one trained content scoring model is then used to determine an absence of pre-defined key points associated with the prompt in the normalized spoken words. A list of the determined absent key points can be generated. This list can then be displayed/caused to be displayed in a graphical user interface along with feedback to improve content completeness. Related apparatus, systems, techniques and articles are also described.

Abstract: Systems and methods are disclosed for automatically scoring a constructed response using a neural network. In embodiments, a constructed response received by a processing system may be processed to divide the constructed response into multiple series of word tokens, wherein each word token includes a sequence of characters. The constructed response may be further processed to correct one or more spelling errors. The word tokens may be encoded to generate representation vectors for the constructed response. A set of nonlinear operations may be applied to the plurality of representation vectors in a neural network to generate a single vector output. A set of predetermined network weights may be applied to the vector output of the neural network to generate a scalar output for scoring the constructed response.

Abstract: A method comprising accessing a first data structure that is associated with a first product prepared by a student and that includes first process data associated with a process performed by the student in generating the first product, analyzing the first data structure to generate a first characterization score based on the first product and the first process data, accessing a second data structure that is associated with a second product prepared by the student and that includes second process data associated with a process performed by the student in generating the second product, analyzing the second data structure to generate a second characterization score based on the second product and the second process data, and calculating a skill level change metric based on the first characterization score and the second characterization score indicating a change in ability level of the student over a course of the scenario-based assessment.

Abstract: Data is received that encapsulates a document of text. The text is then segmented into a plurality of semantically coherent units using a coherence-aware text segmentation (CATS) machine learning model. Data is then provided that characterizes the segmenting. Related apparatus, systems, techniques and articles are also described.

Abstract: Reading comprehension of a user can be assessed by presenting, in a graphical user interface, sequential reading text comprising a plurality of passages. The graphical user interface can alternate between (i) automatically advancing through passages of the reading text and (ii) manually advancing through passages of the reading text within the graphical user interface which is in response to user-generated input received via the graphical user interface. An audio narration is provided during the automatic advancing of the reading text. An audio file is recorded during the manual advancing of the reading text which is used to automatically determine an estimated level of reading comprehension of the user. Data characterizing the determined level of reading comprehension of the user can then be provided (e.g., displayed, loaded into memory, stored on a hard drive, transmitted to a remote computing system, etc.). Related apparatus, systems, techniques and articles are also described.

Abstract: Biometric keystroke measure data derived from a computer-implemented long form examination taken by an examinee is received. Features are the extracted from the biometric keystroke measure data for the examinee. A similarity value is then determined, using one or more of a direct distance approach or a machine learning approach, for the extracted features relative to features extracted from biometric keystroke measure data derived from each of a plurality of other examinees while taking the long form examination. At least one of the determined similarity values is then identified having a value above a pre-defined threshold. The pre-defined threshold indicates a likelihood of the examinee being the same as one of the other examinees. Data can then be provided that characterizes the identification. Related apparatus, systems, techniques and articles are also described.

Abstract: Systems and methods are provided for the design and implementation of experiments that facilitate the investigation of process data. The experiments involve recording the completion of a task by participants and then playing back the video of task completion to automatically probe participants about their affective, behavioral, and cognitive experiences. As a result of this system, information about affective, behavioral, and cognitive processes can be more easily investigated by researchers without computer programming knowledge. Corresponding apparatuses, systems, and methods are also discussed.

Abstract: Systems and methods for computer-implemented evaluation of a performance are provided. In a first aspect, a computer-implemented method of evaluating an interaction generates a first temporal record of first behavior features exhibited by a first entity during an interaction between a first entity and a second entity. A second temporal record is generated including second behavior features exhibited by a second entity during an interaction with a first entity. A determination is made that a first feature of a first temporal record is associated with a second feature of a second temporal record. The length of time that passes between the first feature and second feature is evaluated, and a determination is made that the length of time satisfies a temporal condition. A co-occurrence record associated with a first feature and a second feature is generated and included in a co-occurrence record data-structure.

Abstract: Systems and methods are provided for automatically scoring a constructed response. The constructed response is processed to generate a plurality of numerical vectors that is representative of the constructed response. A model is applied to the plurality of numerical vectors. The model includes an input layer configured to receive the plurality of numerical vectors, the input layer being connected to a following layer of the model via a first plurality of connections. Each of the connections has a first weight. An intermediate layer of nodes is configured to receive inputs from an immediately-preceding layer of the model via a second plurality of connections, each of the connections having a second weight. An output layer is connected to the intermediate layer via a third plurality of connections, each of the connections having a third weight. The output layer is configured to generate a score for the constructed response.

Abstract: Data is received that encapsulates a spoken response to a prompt text comprising a string of words. Thereafter, the received data is transcribed into a string of words. The string of words is then compared with a prompt so that a similarity grid representation of the comparison can be generated that characterizes a level of similarity between the string of words in the spoken response and the string of words in the prompt text. The grid representation is then scored using at least one machine learning model. The score indicates a likelihood of the spoken response having been off-topic. Data providing the encapsulated score can then be provided. Related apparatus, systems, techniques and articles are also described.

Abstract: Systems and methods are provided for processing a drawing in a modeling prototype. A data structure associated with a visual model is accessed. The visual model is analyzed to extract construct-relevant features, where the construct-relevant features are extracted using a drawing object by identifying visual attributes of the visual model and populating a data structure for each object drawn. The visual model is analyzed to generate a statistical model, where the statistical model is generated using a multidimensional scoring rubric by targeting different constructs which compositely estimate learning progression levels, wherein the statistical model is based on features that are principally aligned with one or more of the constructs. An automated scoring is determined based on the construct-relevant features and the statistical model, where the automated scoring is stored in a computer readable medium. and is outputted for display, transmitted across a computer network, or printed.