Abstract: Systems and methods are provided for evaluating career interests through situational judgment test format. In embodiments, a career assessment is generated, using a first language based machine learning model and a set of criteria. The career assessment is administered to a user. A plurality of test answers for the user are received based on the career assessment. For a particular test answer a score is assigned for the particular test answer and at least one user career interest category based on the score is determined. The totals for the at least one user career interest category based on the determining is tabulated. At least one career suggestion is provided, using a second language based machine learning model, based on the totals for the at least one user career interest category.

Abstract: A speech in response to a prompt is accessed. The speech is provided to a speech recognition module that is configured to generate a text transcript of the speech. Speech features are extracted from the speech. Similarly, text features are extracted from the text transcript. Both speech features and text features are vector representations of the speech. The two features are concatenated into one vector representation that captures both perceptual and linguistic components of the speech. The concatenated vector is provided to a speech scoring model. The speech scoring model simultaneously provides a holistic score as well as fine-grained scores to the speech based on the concatenated features.

Abstract: Systems and methods are provided for generating language learning items. In embodiments, training data comprising a plurality of concept-sentence pairs received by a machine-learning based language model may be used to train the model to generate sentences based on an input. A stimulus may be received and used by the trained machine-learning based language model to generate a sentence. The sentence may be stored in a computer readable medium.

Abstract: Systems and methods are provided for modeling lexical experience for tracking of oral reading fluency. In embodiments, a background corpus and a text are received. A reading passage is selected from the text. A surprisal model is generated based on the background corpus and a portion of the text preceding the reading passage. An audio of a user reciting the reading passage is iteratively received, wherein a token of the audio from a plurality of tokens is received at a time. Oral reading fluency is evaluated based on the audio and the surprisal mode. The next reading passage is selected. An oral reading fluency report is stored in a computer readable medium.

Abstract: Computer-based argument mining can be implemented by accessing data comprising text encapsulating a plurality of debates relating to varying topics. Thereafter, the text is parsed into snippets of text containing arguments. Features are then extracted or otherwise derived from pairs of snippets. The features are then used by a transformer network to classify each of the snippets according to one of three categories, the pairs of snippets either being on a same side of a debate or on a different side of the debate. Data is then provided which characterizes the classification. Related apparatus, systems, techniques and articles are also described.

Abstract: Data is received that includes a passage of text. The passage of text can be tokenized so that features can be extracted from the resulting tokens. One or more machine learning models can detect one or more metaphors within the passage of text using the extracted features. The at least one machine learning model can be trained, for example, by interleaving data relating to metaphors with data relating to one or more auxiliary tasks associated with related figurative language constructs. Data can then be provided that identifies the detected metaphors. Related apparatus, systems, techniques and articles are also described.

Abstract: An ordered interaction task is initiated in a graphical user interface. A main region of the graphical user interface is segmented into a plurality of discrete sub-regions, each sub-region including content of the ordered interaction task. The user is then prompted to begin the ordered interaction task through a non-visual prompt that is provided concurrently with the graphical user interface. In response to a first user-initiated command received in the graphical user interface, a non-visual presentation of at least a portion of the content of at least one sub-region is provided concurrently with the graphical user interface.

Abstract: Data is received that encapsulates a video of a subject performing a task. This video is used to generate a transcript using an automatic speech recognition (ASR) system. A plurality of text segments are generated from the transcript and then tokenized. A textual representation of each segment is extracted by a transformer model using the tokenized text segment (i.e., the tokens corresponding to the text segment). Thereafter, for each segment, a fused representation derived from the textual representations and corresponding visual and audio features from the video is generated. A sparse attention machine learning model then selects an optimal slice of the video based on the fused representations. The optimal slice can then be input into one or more machine learning models trained to characterize performance of the task by the subject.

Abstract: Automatic content generation by one or more computing devices can include receiving data comprising an original sentence with a grammar artifact of interest. Thereafter, a plurality of distractor candidates are generated based on the original sentence with the grammar artifact of interest. At least one machine learning-based language model then scores each of the distractor candidates. These scores characterize a likelihood of the corresponding distractor candidate being selected as part of an assessment by a subject. The distractor candidates can be filtered to result in a filtered list of distractor candidates from which the x top scoring distractor candidates can be selected. A grammar practice item is then generated based on the original sentence and the x top scoring distractor candidates. The grammar practice can then be provided. Related apparatus, systems, and articles are also described.

Abstract: Quality of a narrative is characterized by receiving data that includes a narrative text. This narrative text is then tokenized and events are extracted from the tokenized words. The extraction can use, in parallel, two or more different extraction techniques. The extracted events are then extracted so that a waveform can be generated based on the aggregated extracted events that characterizes a plurality of emotional arcs within the narrative text. Subsequently, a plurality of waveform elements are extracted from the waveform. The narrative quality (or other quality) of the narrative text is then scored based on the extracted plurality of waveform elements and using a machine learning model trained to correlate emotional arc waveforms with narrative quality scores. Related apparatus, systems, techniques and articles are also described.

Abstract: Data is received that includes a passage of text generated in response to a prompt which comprises a plurality of sentences. Thereafter, the passage of text is tokenized into a plurality of tokens each corresponding to a different word in the passage of text. A first classification head of an adaptive fine-tuned transforms classifies each of the tokens into one of a plurality of classes. A second classification head of the adaptive fine-tuned transformer model classifies each of the sentences as either including or not including an argument. Data can then be provided which characterizes the first and second classifications. Related apparatus, systems, techniques and articles are also described.

Abstract: Data is received by an automated spoken language learning and assessment system that includes a passage of text comprising a response to stimulus material. Thereafter, at least one machine learning model is used to detect absent key points within the passage of text and/or location spans of key points in the passage of text. The at least one machine learning model can be trained using a corpus with annotated key points and a span for each key point. In addition, each of the detected key points is scored by at least one key point quality model to result in a corresponding key point score. Diagnostic feedback targeting content development skills is then determined based on the detecting and using the key point scores. Data can then be provided which characterizes such diagnostic feedback. Related apparatus, systems, techniques and articles are also described.

Abstract: Data such as unstructured text is received that includes a sequence of sentences. This received data is then tokenized into a plurality of tokens. The received data is segmented using a hierarchical transformer network model including a token transformer, a sentence transformer, and a segmentation classifier. The token transformer contextualizes tokens within sentences and yields sentence embeddings. The sentences transformer contextualizes sentence representations based on the sentence embedddings. The segmentation classifier predicts segments of the received data based on the contextualized sentence representations. Data can be provided which characterizes the segmentation of the received data. Related apparatus, systems, techniques and articles are also described.

Abstract: Systems and methods are provided for automatic evaluation of argument critique essays written by young students in response to prompts. A transformer pre-trained for natural language processing is employed as a machine learning model, which is fine-tune with a first training dataset comprising unannotated argument critique essays written by college students, and then fine-tuned with a second training dataset comprising annotated argument critique essays written by middle school students, where each sentence in the second training dataset is annotated for the presence of valid critiques to prompts. The fine-tuned machine learning model is used to classify each sentence in an essay to be evaluated as either containing a valid critique or not.

Abstract: Human-machine dialog is characterized by receiving data comprising a recording of an individual interacting with a dialog application simulating a conversation. Thereafter, the received data is parsed using automated speech recognition to result in text comprising a plurality of words. Features are extracted from the parsed data and then input an ensemble of different machine learning models each trained to generate a score characterizing a plurality of different dialog constructs. Thereafter, scores generated by the machine learning models for each of the dialog constructs are fused. A performance score is then generated based on the fused scores which characterizes a conversational proficiency of the individual interacting with the dialog application. Data can then be provided which includes or otherwise characterizes the generated score. Related apparatus, systems, techniques and articles are also described.

Abstract: A computer-implemented technique for characterizing lexical concreteness in narrative includes receiving data encapsulating narrative text having a plurality of words. Thereafter, the function words can be removed from the narrative text to result in only content words. A concreteness score can then be assigned to each content word by polling a database to identify matching words and to use concreteness scores associated with such matching words as specified by the database. Data can then be provided which characterizes the assigned concreteness scores. Related apparatus, systems, techniques and articles are also described.

Abstract: An electronic audio file is received that comprises spontaneous speech responsive to a prompt in a non-native language of a speaker. Thereafter, the electronic audio file is parsed into a plurality of spoken words. The spoken words are then normalized to remove stop words and disfluencies. At least one trained content scoring model is then used to determine an absence of pre-defined key points associated with the prompt in the normalized spoken words. A list of the determined absent key points can be generated. This list can then be displayed/caused to be displayed in a graphical user interface along with feedback to improve content completeness. Related apparatus, systems, techniques and articles are also described.

Abstract: Systems and methods are provided for processing a group of essays to develop a classifier that detects nonsensical computer-generated essays. A data structure associated with a group of essays is accessed, wherein the group of essays includes nonsensical computer-generated essays and good-faith essays. Both the nonsensical computer-generated essays and the good-faith essays are assigned feature values. The distribution of feature values between the nonsensical computer-generated essays and the good-faith essays is measured. A classifier that detects whether an essay is a nonsensical computer-generated essay is developed, wherein the classifier is developed using the distribution of feature values.

Abstract: Systems and methods are disclosed for automatically scoring a constructed response using a neural network. In embodiments, a constructed response received by a processing system may be processed to divide the constructed response into multiple series of word tokens, wherein each word token includes a sequence of characters. The constructed response may be further processed to correct one or more spelling errors. The word tokens may be encoded to generate representation vectors for the constructed response. A set of nonlinear operations may be applied to the plurality of representation vectors in a neural network to generate a single vector output. A set of predetermined network weights may be applied to the vector output of the neural network to generate a scalar output for scoring the constructed response.

Abstract: A method comprising accessing a first data structure that is associated with a first product prepared by a student and that includes first process data associated with a process performed by the student in generating the first product, analyzing the first data structure to generate a first characterization score based on the first product and the first process data, accessing a second data structure that is associated with a second product prepared by the student and that includes second process data associated with a process performed by the student in generating the second product, analyzing the second data structure to generate a second characterization score based on the second product and the second process data, and calculating a skill level change metric based on the first characterization score and the second characterization score indicating a change in ability level of the student over a course of the scenario-based assessment.