Abstract: An electronic audio file is received that comprises spontaneous speech responsive to a prompt in a non-native language of a speaker. Thereafter, the electronic audio file is parsed into a plurality of spoken words. The spoken words are then normalized to remove stop words and disfluencies. At least one trained content scoring model is then used to determine an absence of pre-defined key points associated with the prompt in the normalized spoken words. A list of the determined absent key points can be generated. This list can then be displayed/caused to be displayed in a graphical user interface along with feedback to improve content completeness. Related apparatus, systems, techniques and articles are also described.

Abstract: Systems and methods are provided for automatically scoring a constructed response. The constructed response is processed to generate a plurality of numerical vectors that is representative of the constructed response. A model is applied to the plurality of numerical vectors. The model includes an input layer configured to receive the plurality of numerical vectors, the input layer being connected to a following layer of the model via a first plurality of connections. Each of the connections has a first weight. An intermediate layer of nodes is configured to receive inputs from an immediately-preceding layer of the model via a second plurality of connections, each of the connections having a second weight. An output layer is connected to the intermediate layer via a third plurality of connections, each of the connections having a third weight. The output layer is configured to generate a score for the constructed response.

Abstract: Data is received that encapsulates a spoken response to a prompt text comprising a string of words. Thereafter, the received data is transcribed into a string of words. The string of words is then compared with a prompt so that a similarity grid representation of the comparison can be generated that characterizes a level of similarity between the string of words in the spoken response and the string of words in the prompt text. The grid representation is then scored using at least one machine learning model. The score indicates a likelihood of the spoken response having been off-topic. Data providing the encapsulated score can then be provided. Related apparatus, systems, techniques and articles are also described.

Abstract: Data is received that encapsulates a spoken response to a test question. Thereafter, the received data is transcribed into a string of words. The string of words is then compared with at least one source string so that a similarity grid representation of the comparison can be generated that characterizes a level of similarity between the string of words and the at least one source string. The grid representation is then scored using at least one machine learning model. The score indicates a likelihood of the spoken response having been plagiarized. Data providing the encapsulated score can then be provided. Related apparatus, systems, techniques and articles are also described.

Abstract: Computer-implemented systems and methods are provided for automatically generating recitation items. For example, a computer performing the recitation item generation can receive one or more text sets that each includes one or more texts. The computer can determine a value for each text set using one or more metrics, such as a vocabulary difficulty metric, a syntactic complexity metric, a phoneme distribution metric, a phonetic difficulty metric, and a prosody distribution metric. Then the computer can select a final text set based on the value associated with each text set. The selected final text set can be used as the recitation items for a speaking assessment test.

Abstract: Computer-implemented systems and methods are provided for scoring content of a spoken response to a prompt. A scoring model is generated for a prompt, where generating the scoring model includes generating a transcript for each of a plurality of training responses to the prompt, dividing the plurality of training responses into clusters based on the transcripts of the training responses, selecting a subset of the training responses in each cluster for scoring, scoring the selected subset of training responses for each cluster, and generating content training vectors using the transcripts from the scored subset. A transcript is generated for a received spoken response to be scored, and a similarity metric is computed between the transcript of the spoken response to be scored and the content training vectors. A score is assigned to the spoken response based on the determined similarity metric.

Abstract: Systems and methods are provided for automatically scoring a constructed response. The constructed response is processed to generate a plurality of numerical vectors that is representative of the constructed response. A model is applied to the plurality of numerical vectors. The model includes an input layer configured to receive the plurality of numerical vectors, the input layer being connected to a following layer of the model via a first plurality of connections. Each of the connections has a first weight. An intermediate layer of nodes is configured to receive inputs from an immediately-preceding layer of the model via a second plurality of connections, each of the connections having a second weight. An output layer is connected to the intermediate layer via a third plurality of connections, each of the connections having a third weight. The output layer is configured to generate a score for the constructed response.

Abstract: Systems and methods are provided for automatically scoring a constructed response. The constructed response is processed to generate a plurality of numerical vectors that is representative of the constructed response. A model is applied to the plurality of numerical vectors. The model includes an input layer configured to receive the plurality of numerical vectors, the input layer being connected to a following layer of the model via a first plurality of connections. Each of the connections has a first weight. An intermediate layer of nodes is configured to receive inputs from an immediately-preceding layer of the model via a second plurality of connections, each of the connections having a second weight. An output layer is connected to the intermediate layer via a third plurality of connections, each of the connections having a third weight. The output layer is configured to generate a score for the constructed response.

Abstract: Systems and methods are provided for a processor-implemented method of analyzing quality of sound acquired via a microphone. An input metric is extracted from a sound recording at each of a plurality of time intervals. The input metric is provided at each of the time intervals to a neural network that includes a memory component, where the neural network provides an output metric at each of the time intervals, where the output metric at a particular time interval is based on the input metric at a plurality of time intervals other than the particular time interval using the memory component of the neural network. The output metric is aggregated from each of the time intervals to generate a score indicative of the quality of the sound acquired via the microphone.

Abstract: Computer-implemented systems and methods are provided for automatically generating recitation items. For example, a computer performing the recitation item generation can receive one or more text sets that each includes one or more texts. The computer can determine a value for each text set using one or more metrics, such as a vocabulary difficulty metric, a syntactic complexity metric, a phoneme distribution metric, a phonetic difficulty metric, and a prosody distribution metric. Then the computer can select a final text set based on the value associated with each text set. The selected final text set can be used as the recitation items for a speaking assessment test.

Abstract: Computer-implemented systems and methods are provided for scoring content of a spoken response to a prompt. A scoring model is generated for a prompt, where generating the scoring model includes generating a transcript for each of a plurality of training responses to the prompt, dividing the plurality of training responses into clusters based on the transcripts of the training responses, selecting a subset of the training responses in each cluster for scoring, scoring the selected subset of training responses for each cluster, and generating content training vectors using the transcripts from the scored subset. A transcript is generated for a received spoken response to be scored, and a similarity metric is computed between the transcript of the spoken response to be scored and the content training vectors. A score is assigned to the spoken response based on the determined similarity metric.

Abstract: A method for scoring non-native speech includes receiving a speech sample spoken by a non-native speaker and performing automatic speech recognition and metric extraction on the speech sample to generate a transcript of the speech sample and a speech metric associated with the speech sample. The method further includes determining whether the speech sample is scorable or non-scorable based upon the transcript and speech metric, where the determination is based on an audio quality of the speech sample, an amount of speech of the speech sample, a degree to which the speech sample is off-topic, whether the speech sample includes speech from an incorrect language, or whether the speech sample includes plagiarized material. When the sample is determined to be non-scorable, an indication of non-scorability is associated with the speech sample. When the sample is determined to be scorable, the sample is provided to a scoring model for scoring.

Abstract: Systems and methods are provided for scoring non-native, spontaneous speech. A spontaneous speech sample is received, where the sample is of spontaneous speech spoken by a non-native speaker. Automatic speech recognition is performed on the sample using an automatic speech recognition system to generate a transcript of the sample, where a speech recognizer metric is determined by the automatic speech recognition system. A word accuracy rate estimate is determined for the transcript of the sample generated by the automatic speech recognition system based on the speech recognizer metric. The spontaneous speech sample is scored using a preferred scoring model when the word accuracy rate estimate satisfies a threshold, and the spontaneous speech sample is scored using an alternate scoring model when the word accuracy rate estimate fails to satisfy the threshold.

Abstract: Computer-implemented systems and methods are provided for automatically scoring the content of moderately predictable responses. For example, a computer performing the content scoring analysis can receive a response (either in text or spoken form) to a prompt. The computer can determine the content correctness of the response by analyzing one or more content features. One of the content features is analyzed by applying one or more regular expressions, determined based on training responses associated with the prompt. Another content feature is analyzed by applying one or more context free grammars, determined based on training responses associated with the prompt. Another content feature is analyzed by applying a keyword list, determined based on the test prompt eliciting the response and/or stimulus material. Another content feature is analyzed by applying one or more probabilistic n-gram models, determined based on training responses associated with the prompt.

Abstract: Systems and methods are provided for assigning a difficulty score to a speech sample. Speech recognition is performed on a digitized version of the speech sample using an acoustic model to generate word hypotheses for the speech sample. Time alignment is performed between the speech sample and the word hypotheses to associate the word hypotheses with corresponding sounds of the speech sample. A first difficulty measure is determined based on the word hypotheses, and a second difficulty measure is determined based on acoustic features of the speech sample. A difficulty score for the speech sample is generated based on the first difficulty measure and the second difficulty measure.

Abstract: Systems and methods are provided for scoring non-native speech. Two or more speech samples are received, where each of the samples are of speech spoken by a non-native speaker, and where each of the samples are spoken in response to distinct prompts. The two or more samples are concatenated to generate a concatenated response for the non-native speaker, where the concatenated response is based on the two or more speech samples that were elicited using the distinct prompts. A concatenated speech proficiency metric is computed based on the concatenated response, and the concatenated speech proficiency metric is provided to a scoring model, where the scoring model generates a speaking score based on the concatenated speech metric.

Abstract: Systems and methods are provided for scoring a non-scripted speech sample. A system includes one or more data processors and one or more computer-readable mediums. The computer-readable mediums are encoded with a non-scripted speech sample data structure, where the non-scripted speech sample data structure includes: a speech sample identifier that identifies a non-scripted speech sample, a content feature extracted from the non-scripted speech sample, and a content-based speech score for the non-scripted speech sample. The computer-readable mediums further include instructions for commanding the one or more data processors to extract the content feature from a set of words automatically recognized in the non-scripted speech sample and to score the non-scripted speech sample by providing the extracted content feature to a scoring model to generate the content-based speech score.

Abstract: Computer-implemented systems and methods are provided for assessing non-native spontaneous speech pronunciation. Speech recognition on digitized speech is performed using a non-native acoustic model trained with non-native speech to generate word hypotheses for the digitized speech. Time alignment is performed between the digitized speech and the word hypotheses using a reference acoustic model trained with native-quality speech. Statistics are calculated regarding individual words and phonemes in the word hypotheses based on the alignment. A plurality of features for use in assessing pronunciation of the speech are calculated based on the statistics, an assessment score is calculated based on one or more of the calculated features, and the assessment score is stored in a computer-readable memory.

Abstract: Systems and methods are provided for automatically scoring a constructed response. The constructed response is processed to generate a plurality of numerical vectors that is representative of the constructed response. A model is applied to the plurality of numerical vectors. The model includes an input layer configured to receive the plurality of numerical vectors, the input layer being connected to a following layer of the model via a first plurality of connections. Each of the connections has a first weight. An intermediate layer of nodes is configured to receive inputs from an immediately-preceding layer of the model via a second plurality of connections, each of the connections having a second weight. An output layer is connected to the intermediate layer via a third plurality of connections, each of the connections having a third weight. The output layer is configured to generate a score for the constructed response.

Abstract: Systems and methods are provided for scoring speech. A speech sample is received, where the speech sample is associated with a script. The speech sample is aligned with the script. An event recognition metric of the speech sample is extracted, and locations of prosodic events are detected in the speech sample based on the event recognition metric. The locations of the detected prosodic events are compared with locations of model prosodic events, where the locations of model prosodic events identify expected locations of prosodic events of a fluent, native speaker speaking the script. A prosodic event metric is calculated based on the comparison, and the speech sample is scored using a scoring model based upon the prosodic event metric.