Abstract: Data is received that encapsulates a spoken response to a test question. Thereafter, the received data is transcribed into a string of words. The string of words is then compared with at least one source string so that a similarity grid representation of the comparison can be generated that characterizes a level of similarity between the string of words and the at least one source string. The grid representation is then scored using at least one machine learning model. The score indicates a likelihood of the spoken response having been plagiarized. Data providing the encapsulated score can then be provided. Related apparatus, systems, techniques and articles are also described.

Abstract: Systems and methods are provided for the detection of sentiment in writing. A plurality of texts is received from a larger collection of writing samples with a computer system. A set of seed words from the plurality of texts are labeled as being of positive sentiment or of negative sentiment with the computer system. The set of seed words is expanded in size with the computer system to provide an expanded set of seed words. Intensity values are assigned to words of the expanded set of seed words. Each of the words of the expanded set of seed words is assigned three intensity values: a value corresponding to the strength of the word's association with a positive polarity class, a value corresponding to the strength of the word's association with a negative polarity class, and a value corresponding to the strength of the word's association with a neutral polarity class.

Abstract: Computer-implemented systems and methods are described herein for detecting unusually frequent exactly matching and nearly matching test responses. A plurality of test responses for each of a plurality of test takers is received. Evidence of a circulated key from the plurality of test responses is identified by detecting two or more test takers with exactly matching test responses for a plurality of questions. The strength of the evidence of the circulated key is analyzed by determining the probability that the two or more test takers would produce exactly matching test responses, wherein the strength of the evidence is inversely related to the probability. Test takers with nearly matching test responses to the circulated key are then identified. All test takers with exactly matching and nearly matching test responses are tagged for further investigation.

Abstract: Systems and methods for identifying a person's native language and/or non-native language based on code-switched text and/or speech, are presented. The systems may be trained using various methods. For example, a language identification system may be trained using one or more code-switched corpora. Text and/or speech features may be extracted from the corpora and used, in combination with a per-word language identify of the text and/or speech, to train at least one machine learner. Code-switched text and/or speech may be received and processed by extracting text and/or speech features. These features may be fed into the at least one machine learner to identify the person's native language.

Abstract: Systems and methods are provided for conducting a simulated conversation with a language learner include determining a first dialog state of the simulated conversation. First audio data corresponding to simulated speech based on the dialog state is transmitted. Second audio data corresponding to a variable length utterance spoken in response to the simulated speech is received. A fixed dimension vector is generated based on the variable length utterance. A semantic label is predicted for the variable-length utterance based on the fixed dimension vector. A second dialog state of the simulated conversation is determined based on the semantic label, and third audio data corresponding to simulated speech is transmitted based on the second dialog state.

Abstract: Systems and methods for defining and using an optimal burst boundary threshold to assess the reliability of a manual/automatic writing score are presented. Keystroke data, including inter-key interval data, such as inter-word interval data, may be gathered from writings. Clustering analyses may be performed on the inter-key interval data to determine an optimal number of bursts for the writings. An optimal burst boundary may be determined from the optimal number of bursts. Other burst-related measures and statistics, including the average and maximum burst lengths, may be determined from the writings based on the optimal burst boundary threshold. A score may be received for each of the writings. A validation indication metric may be generated for each of the writings based on the received score and the optimal burst boundary threshold. The resulting measures and statistics may be used or applied in different ways and provide personalized feedback as learning analytics.

Abstract: Methods and systems for scoring an argument critique written essay, including identifying a discourse segment of the argument critique written essay, determining a position of each sentence in the discourse segment, classifying sentences into discourse units, evaluating an argumentative content of each sentence, and assigning an argumentative score to the essay based on the argumentative content of each sentence in the discourse segment of that essay. Methods and systems for training the scoring method are also disclosed. Corresponding apparatuses, systems, and methods are also disclosed.

Abstract: Systems and methods are provided for acquiring physical-world data indicative of interactions of a subject with an avatar for evaluation. An interactive avatar is provided for interaction with the subject. Speech from the subject to the avatar is captured, and automatic speech recognition is performed to determine content of the subject speech. Motion data from the subject interacting with the avatar is captured. A next action of the interactive avatar is determined based on the content of the subject speech or the motion data. The next action of the avatar is implemented, and a score for the subject is determined based on the content of the subject speech and the motion data.

Abstract: Systems and methods are provided for automatic detection of plagiarized spoken responses during standardized testing of language proficiency. A first spoken response to a speaking task that elicits spontaneous speech and a second spoken response to a source-based speaking task that is assumed to be non-plagiarized are digitally recorded. A first and second sets of speaking proficiency features are calculated for the first and the second spoken response, respectively. The first spoken response is classifying as plagiarized or non-plagiarized based on the comparison between the first and the second set of speaking proficiency features. Corresponding apparatuses, systems, and methods are also disclosed.

Abstract: Systems and methods are provided for automatically scoring a response and statistically revaluating whether it can be considered as aberrant. In one embodiment, a constructed response is evaluated via a pre-screening stage and a post-hoc screening stage. The pre-screening stage attempts to determine whether the constructed response is aberrant based on a variety of aberration metrics and criteria. If the constructed response is deemed not to be aberrant, then the post-hoc screening stage attempts to predict a discrepancy between what score an automated scoring system would assigned and what score a human rater would assign to the response. If the discrepancy is sufficiently low, then the constructed response may be scored by an automated scoring engine. On the other hand, if the constructed response failed to pass either of the two stages, then a flag may be raised to indicate that additional human review may be needed.

Abstract: Computer-implemented systems and methods are described herein for automatically generating questions from text. Text including one or more sentences is received. A sentence, comprising a predicate and one or more arguments associated with the predicate, is parsed from the text. Semantic role labels are assigned to the one or more arguments associated with the predicate. One or more questions are automatically generated relating to the predicate based on the assigned semantic role labels. Each answer to the generated questions is one of the one or more arguments associated with the predicate.

Abstract: Systems and methods of improving reading fluency by presenting a virtual reading partner that takes turns reading aloud with a reader, are presented. First audio data comprising speech corresponding to the reader reading aloud from a first portion of a written transcript is received. The first audio data is processed to identify spoken words from the speech. The reader's speaking of the words in the first portion of the written transcript are compared to correct speaking of words in the first portion of the written transcript. A reading fluency of the reader is assessed. Second audio data comprising a pre-recorded narration of a second portion of the written transcript or automatically generated text to speech of the second portion is generated. The virtual reading partner is presented to the reader and takes turns reading aloud with the reader until the end of the written transcript has been reached.

Abstract: Systems and methods are provided for providing feedback on a user's writing behavior in generating a constructed response. An electronic process log for the constructed response is received. The electronic process log is processed to generate a vector having a predetermined number of elements. The vector comprises information related to the user's actions in generating the constructed response and includes (i) data indicating types of actions performed by the user in generating the constructed response, (ii) time points associated with the actions, and (iii) locations associated with the actions, each location indicating whether an associated action occurred within a word, between words, within a phrase, between phrases, within a sentence, between sentences, within a paragraph, or between paragraph. The vector is compared to one or more vectors associated with other constructed responses, and feedback is generated based on the comparison.

Abstract: A system for end-to-end automated scoring is disclosed. The system includes a word embedding layer for converting a plurality of ASR outputs into input tensors; a neural network lexical model encoder receiving the input tensors; a neural network acoustic model encoder implementing AM posterior probability, word duration, mean value of pitch and mean value of intensity based on a plurality of cues; and a linear regression module, for receiving concatenated encoded features from the neural network lexical model encoder and the neural network acoustic model encoder.

Abstract: Systems and methods are provided for processing a response to essay prompts that request a narrative response. A data structure associated with a narrative essay is accessed. The essay is analyzed to generate an organization subscore, where the organization subscore is generated using a graph metric by identifying content words in each sentence of the essay and populating a data structure with links between related content words in neighboring sentences, wherein the organization subscore is determined based on the links. The essay is analyzed to generate a development subscore, where the development subscore is generated using a transition metric by accessing a transition cue data store and identifying transition words in the essay, wherein the development subscore is based on a number of words in the essay that match words in the transition cue data store. A narrative quality metric is determined based on the organization subscore and the development subscore.

Abstract: Computer-implemented systems and methods are provided for automatically generating recitation items. For example, a computer performing the recitation item generation can receive one or more text sets that each includes one or more texts. The computer can determine a value for each text set using one or more metrics, such as a vocabulary difficulty metric, a syntactic complexity metric, a phoneme distribution metric, a phonetic difficulty metric, and a prosody distribution metric. Then the computer can select a final text set based on the value associated with each text set. The selected final text set can be used as the recitation items for a speaking assessment test.

Abstract: Systems and methods are provided for scoring video clips using visual feature extraction. A signal including a video clip of a subject is received. For each frame of the video clip, physiological features of the subject visually rendered in the video clip are extracted. A plurality of visual words associated with the extracted physiological features are determined. A document including the plurality of visual words is generated. A plurality of feature vectors associated with the document are determined. The plurality of feature vectors to a regression model for scoring are provided.

Abstract: Systems and methods for training raters to rate constructed responses to tasks are described herein. In one embodiment, a plurality of trainee raters are selected without regard to their prior experience. The trainee raters are then train in individual training sessions, during which they are asked to rate responses to a task. Each session presents to the trainee rater the task, a rating rubric, and training responses to the task. The training program receives ratings assigned by the trainee rater to the training responses through a graphical user interface. Upon receiving the assigned rating, the training program presents feedback substantially immediately and determines a score for the trainee rater's assigned rating. Thereafter, qualified raters are selected from the plurality of trainee raters based upon their performance during the training sessions as compared with a statistical model. Operational constructed responses are then assigned to rated by the qualified raters.

Abstract: Systems and methods for identifying a person's native language, are presented. A native language identification system, comprising a plurality of artificial neural networks, such as time delay deep neural networks, is provided. Respective artificial neural networks of the plurality of artificial neural networks are trained as universal background models, using separate native language and non-native language corpora. The artificial neural networks may be used to perform voice activity detection and to extract sufficient statistics from the respective language corpora. The artificial neural networks may use the sufficient statistics to estimate respective T-matrices, which may in turn be used to extract respective i-vectors. The artificial neural networks may use i-vectors to generate a multilayer perceptron model, which may be used to identify a person's native language, based on an utterance by the person in his or her non-native language.

Abstract: Computer-implemented systems and methods for generating a personalization score for a constructed response are provided. A constructed response generated by a user is received. The constructed response is processed with a processing system to generate one or more feature values representative of one or more aspects of the constructed response. A personalization score for the constructed response is generated using the processing system by applying a computer scoring model to the one or more feature values. The personalization score provides a measure of a personalization of the constructed response.