Abstract: Methods and systems for scoring an argument critique written essay, including identifying a discourse segment of the argument critique written essay, determining a position of each sentence in the discourse segment, classifying sentences into discourse units, evaluating an argumentative content of each sentence, and assigning an argumentative score to the essay based on the argumentative content of each sentence in the discourse segment of that essay. Methods and systems for training the scoring method are also disclosed. Corresponding apparatuses, systems, and methods are also disclosed.

Abstract: Systems and methods are provided for acquiring physical-world data indicative of interactions of a subject with an avatar for evaluation. An interactive avatar is provided for interaction with the subject. Speech from the subject to the avatar is captured, and automatic speech recognition is performed to determine content of the subject speech. Motion data from the subject interacting with the avatar is captured. A next action of the interactive avatar is determined based on the content of the subject speech or the motion data. The next action of the avatar is implemented, and a score for the subject is determined based on the content of the subject speech and the motion data.

Abstract: Systems and methods are provided for automatic detection of plagiarized spoken responses during standardized testing of language proficiency. A first spoken response to a speaking task that elicits spontaneous speech and a second spoken response to a source-based speaking task that is assumed to be non-plagiarized are digitally recorded. A first and second sets of speaking proficiency features are calculated for the first and the second spoken response, respectively. The first spoken response is classifying as plagiarized or non-plagiarized based on the comparison between the first and the second set of speaking proficiency features. Corresponding apparatuses, systems, and methods are also disclosed.

Abstract: Systems and methods are provided for automatically scoring a response and statistically revaluating whether it can be considered as aberrant. In one embodiment, a constructed response is evaluated via a pre-screening stage and a post-hoc screening stage. The pre-screening stage attempts to determine whether the constructed response is aberrant based on a variety of aberration metrics and criteria. If the constructed response is deemed not to be aberrant, then the post-hoc screening stage attempts to predict a discrepancy between what score an automated scoring system would assigned and what score a human rater would assign to the response. If the discrepancy is sufficiently low, then the constructed response may be scored by an automated scoring engine. On the other hand, if the constructed response failed to pass either of the two stages, then a flag may be raised to indicate that additional human review may be needed.

Abstract: Computer-implemented systems and methods are described herein for automatically generating questions from text. Text including one or more sentences is received. A sentence, comprising a predicate and one or more arguments associated with the predicate, is parsed from the text. Semantic role labels are assigned to the one or more arguments associated with the predicate. One or more questions are automatically generated relating to the predicate based on the assigned semantic role labels. Each answer to the generated questions is one of the one or more arguments associated with the predicate.

Abstract: Systems and methods of improving reading fluency by presenting a virtual reading partner that takes turns reading aloud with a reader, are presented. First audio data comprising speech corresponding to the reader reading aloud from a first portion of a written transcript is received. The first audio data is processed to identify spoken words from the speech. The reader's speaking of the words in the first portion of the written transcript are compared to correct speaking of words in the first portion of the written transcript. A reading fluency of the reader is assessed. Second audio data comprising a pre-recorded narration of a second portion of the written transcript or automatically generated text to speech of the second portion is generated. The virtual reading partner is presented to the reader and takes turns reading aloud with the reader until the end of the written transcript has been reached.

Abstract: Systems and methods are provided for providing feedback on a user's writing behavior in generating a constructed response. An electronic process log for the constructed response is received. The electronic process log is processed to generate a vector having a predetermined number of elements. The vector comprises information related to the user's actions in generating the constructed response and includes (i) data indicating types of actions performed by the user in generating the constructed response, (ii) time points associated with the actions, and (iii) locations associated with the actions, each location indicating whether an associated action occurred within a word, between words, within a phrase, between phrases, within a sentence, between sentences, within a paragraph, or between paragraph. The vector is compared to one or more vectors associated with other constructed responses, and feedback is generated based on the comparison.

Abstract: A system for end-to-end automated scoring is disclosed. The system includes a word embedding layer for converting a plurality of ASR outputs into input tensors; a neural network lexical model encoder receiving the input tensors; a neural network acoustic model encoder implementing AM posterior probability, word duration, mean value of pitch and mean value of intensity based on a plurality of cues; and a linear regression module, for receiving concatenated encoded features from the neural network lexical model encoder and the neural network acoustic model encoder.

Abstract: Systems and methods are provided for processing a response to essay prompts that request a narrative response. A data structure associated with a narrative essay is accessed. The essay is analyzed to generate an organization subscore, where the organization subscore is generated using a graph metric by identifying content words in each sentence of the essay and populating a data structure with links between related content words in neighboring sentences, wherein the organization subscore is determined based on the links. The essay is analyzed to generate a development subscore, where the development subscore is generated using a transition metric by accessing a transition cue data store and identifying transition words in the essay, wherein the development subscore is based on a number of words in the essay that match words in the transition cue data store. A narrative quality metric is determined based on the organization subscore and the development subscore.

Abstract: Computer-implemented systems and methods are provided for automatically generating recitation items. For example, a computer performing the recitation item generation can receive one or more text sets that each includes one or more texts. The computer can determine a value for each text set using one or more metrics, such as a vocabulary difficulty metric, a syntactic complexity metric, a phoneme distribution metric, a phonetic difficulty metric, and a prosody distribution metric. Then the computer can select a final text set based on the value associated with each text set. The selected final text set can be used as the recitation items for a speaking assessment test.

Abstract: Systems and methods are provided for scoring video clips using visual feature extraction. A signal including a video clip of a subject is received. For each frame of the video clip, physiological features of the subject visually rendered in the video clip are extracted. A plurality of visual words associated with the extracted physiological features are determined. A document including the plurality of visual words is generated. A plurality of feature vectors associated with the document are determined. The plurality of feature vectors to a regression model for scoring are provided.

Abstract: Systems and methods for training raters to rate constructed responses to tasks are described herein. In one embodiment, a plurality of trainee raters are selected without regard to their prior experience. The trainee raters are then train in individual training sessions, during which they are asked to rate responses to a task. Each session presents to the trainee rater the task, a rating rubric, and training responses to the task. The training program receives ratings assigned by the trainee rater to the training responses through a graphical user interface. Upon receiving the assigned rating, the training program presents feedback substantially immediately and determines a score for the trainee rater's assigned rating. Thereafter, qualified raters are selected from the plurality of trainee raters based upon their performance during the training sessions as compared with a statistical model. Operational constructed responses are then assigned to rated by the qualified raters.

Abstract: Systems and methods for identifying a person's native language, are presented. A native language identification system, comprising a plurality of artificial neural networks, such as time delay deep neural networks, is provided. Respective artificial neural networks of the plurality of artificial neural networks are trained as universal background models, using separate native language and non-native language corpora. The artificial neural networks may be used to perform voice activity detection and to extract sufficient statistics from the respective language corpora. The artificial neural networks may use the sufficient statistics to estimate respective T-matrices, which may in turn be used to extract respective i-vectors. The artificial neural networks may use i-vectors to generate a multilayer perceptron model, which may be used to identify a person's native language, based on an utterance by the person in his or her non-native language.

Abstract: Computer-implemented systems and methods for generating a personalization score for a constructed response are provided. A constructed response generated by a user is received. The constructed response is processed with a processing system to generate one or more feature values representative of one or more aspects of the constructed response. A personalization score for the constructed response is generated using the processing system by applying a computer scoring model to the one or more feature values. The personalization score provides a measure of a personalization of the constructed response.

Abstract: Computer-implemented systems and methods are provided for scoring content of a spoken response to a prompt. A scoring model is generated for a prompt, where generating the scoring model includes generating a transcript for each of a plurality of training responses to the prompt, dividing the plurality of training responses into clusters based on the transcripts of the training responses, selecting a subset of the training responses in each cluster for scoring, scoring the selected subset of training responses for each cluster, and generating content training vectors using the transcripts from the scored subset. A transcript is generated for a received spoken response to be scored, and a similarity metric is computed between the transcript of the spoken response to be scored and the content training vectors. A score is assigned to the spoken response based on the determined similarity metric.

Abstract: Systems and methods are described for providing a multi-modal evaluation of a presentation. A system includes a motion capture device configured to detect motion an examinee giving a presentation and an audio recording device configured to capture audio of the examinee giving the presentation. One or more data processors are configured to extract a non-verbal feature of the presentation based on data collected by the motion capture device and an audio feature of the presentation based on data collected by the audio recording device. The one or more data processors are further configured to generate a presentation score based on the non-verbal feature and the audio feature.

Abstract: Systems and methods are provided for implementing a parallel Expectation Minimization algorithm for generalized latent variable models. Item response data that is based on responses to items from multiple respondents is accessed. The item response data includes data for multiple response variables. The item response data is analyzed using a generalized latent variable model, and the analysis includes an application of a Parallel-E Parallel-M (PEPM) algorithm. In a parallel Expectation step of the PEPM algorithm, the respondents are subdivided into N groups of respondents, and computations for the N groups are performed in parallel using the N processor cores. In a parallel Maximization step of the PEPM algorithm, the response variables are subdivided into N groups of response variables, and computations for the N groups of response variables are performed in parallel using the N processor cores.

Abstract: Systems and methods are provided for determining whether an examination prompt elicits responses of a type for which the examination prompt is designed. Word types for each of a plurality of words in a plurality of responses to the examination prompt are identified. An evaluation distribution of the identified word types is determined. Predetermined distributions stored on a computer-readable medium associated with a plurality of different response types are accessed, and the evaluation distribution is compared to each of the predetermined distributions to determine which predetermined distribution the evaluation distribution is most similar. A determination is made as to whether the most similar distribution is associated with a same response type as the type for which the examination prompt is designed. An indication is made as to whether the examination prompt is deemed to elicit responses of the type for which the examination prompt is designed.

Abstract: Systems and methods are provided for automatically scoring a constructed response. The constructed response is processed to generate a plurality of numerical vectors that is representative of the constructed response. A model is applied to the plurality of numerical vectors. The model includes an input layer configured to receive the plurality of numerical vectors, the input layer being connected to a following layer of the model via a first plurality of connections. Each of the connections has a first weight. An intermediate layer of nodes is configured to receive inputs from an immediately-preceding layer of the model via a second plurality of connections, each of the connections having a second weight. An output layer is connected to the intermediate layer via a third plurality of connections, each of the connections having a third weight. The output layer is configured to generate a score for the constructed response.

Abstract: Systems and methods are provided for implementing an educational dialog system. An initial task model is accessed that identifies a plurality of dialog states associated with a task, a language model configured to identify a response meaning associated with a received response, and a language understanding model configured to select a next dialog state based on the identified response meaning. The task is provided to a plurality of persons for training. The task model is updated by revising the language model and the language understanding model based on responses received to prompts of the provided task, and the updated task is provided to a student for development of speaking capabilities.