Abstract: Computer-implemented systems and methods are provided for automatically scoring the content of moderately predictable responses. For example, a computer performing the content scoring analysis can receive a response (either in text or spoken form) to a prompt. The computer can determine the content correctness of the response by analyzing one or more content features. One of the content features is analyzed by applying one or more regular expressions, determined based on training responses associated with the prompt. Another content feature is analyzed by applying one or more context free grammars, determined based on training responses associated with the prompt. Another content feature is analyzed by applying a keyword list, determined based on the test prompt eliciting the response and/or stimulus material. Another content feature is analyzed by applying one or more probabilistic n-gram models, determined based on training responses associated with the prompt.

Abstract: Systems and methods are provided for scoring non-native, spontaneous speech. A spontaneous speech sample is received, where the sample is of spontaneous speech spoken by a non-native speaker. Automatic speech recognition is performed on the sample using an automatic speech recognition system to generate a transcript of the sample, where a speech recognizer metric is determined by the automatic speech recognition system. A word accuracy rate estimate is determined for the transcript of the sample generated by the automatic speech recognition system based on the speech recognizer metric. The spontaneous speech sample is scored using a preferred scoring model when the word accuracy rate estimate satisfies a threshold, and the spontaneous speech sample is scored using an alternate scoring model when the word accuracy rate estimate fails to satisfy the threshold.

Abstract: Systems and methods are provided for determining a test item rendering anomaly. A test item is provided to a first test item rendering system that is configured to generate a first graphical representation of the test item. The test item is provided to a second test item rendering system that is configured to generate a second graphical representation of the test item. A first digital pixel image of the first graphical representation is captured, and a second digital pixel image of the second graphical representation is captured. Pixels of the digital pixel images are processed, where the processing includes comparing the digital pixel images to determine an extent to which the second digital pixel image differs from the first digital pixel image. An indication for the test item is provided when the second digital pixel image differs from the first digital pixel image by more than a threshold amount.

Abstract: Systems and methods are provided for generating an intelligibility score for speech of a non-native speaker. Words in a speech recording are identified using an automated speech recognizer, where the automated speech recognizer provides a string of words identified in the speech recording, and where the automated speech recognizer further provides an acoustic model likelihood score for each word in the string of words. For a particular word in the string of words, a context metric value is determined based upon a usage of the particular word within the string of words. An acoustic score for the particular word is determined based on the acoustic model likelihood score for the particular word from the automated speech recognizer. An intelligibility score is determined for the particular word based on the acoustic score for the particular word and the context metric value for the particular word.

Abstract: In accordance with the teachings described herein, systems and methods are provided for measuring a user's comprehension of subject matter of a text. A summary generated by the user is received, where the summary summarizes the text. The summary is processed to determine a first numerical measure indicative of a similarity between the summary and a reference summary. The summary is processed to determine a second numerical measure indicative of a degree to which a single sentence of the summary summarizes an entirety of the text. The summary is processed to determine a third numerical measure indicative of a degree of copying in the summary of multi-word sequences present in the text. A numerical model is applied to the first numerical measure, the second numerical measure and the third numerical measure to determine a score for the summary indicative of the user's comprehension of the subject matter of the text.

Abstract: Systems and methods are provided for administering a test using an electronic device. The electronic device is registered to a test-taker, where the registering includes receiving identifying information from the test-taker and associating the electronic device with the test-taker using the identifying information. The test is stored in encrypted form on the electronic device, and the test includes a test question. The test is decrypted prior to test administration. The test is administered to the test-taker via the electronic device, and the administering includes displaying the test question on the electronic device and receiving an answer to the test question on the electronic device. The administered test is removed from the electronic device after transferring the answer to a testing service.

Abstract: Systems and methods are provided for identifying one or more target words of a corpus that have a lexical relationship to a plurality of provided cue words. The cue words and statistical lexical information derived from a corpus of documents are analyzed to determine candidate words that have a lexical association with the cue words. The statistical information includes numerical values indicative of probabilities of word pairs appearing together as adjacent words in a well-formed text or appearing together within a paragraph of a well-formed text. For each candidate word, a statistical association score between the candidate word and each of the cue words is determined. An aggregate score for each of the candidate words is determined based on the statistical association scores. One or more of the candidate words are selected to be the one or more target words based on the aggregate scores.

Abstract: Systems and methods are provided for scoring a speech sample. Automatic speech recognition is performed on the speech sample using an automatic speech recognition system to generate a transcription of the sample. Words in the transcription are associated with parts of speech, and part of speech sequences are extracted from the parts of speech associations. A grammar metric is generated based on the part of speech sequences, and the speech sample is scored based on the grammar metric.

Abstract: Computer-implemented systems and methods are provided for assessing non-native speech proficiency. a non-native speech sample is processed to identify a plurality of vowel sound boundaries in the non-native speech sample. Portions of the non-native speech sample are analyzed within the vowel sound boundaries to extract vowel characteristics associated with a first vowel sound and a second vowel sound represented in the non-native speech sample. The vowel characteristics are processed to identify a first vowel pronunciation metric for the first vowel sound and a second vowel pronunciation metric for the second vowel sound, and the first vowel pronunciation metric and the second vowel pronunciation metric are processed to determine whether the non-native speech sample exhibits a distinction in pronunciation of the first vowel sound and the second vowel sound.

Abstract: Systems and methods are provided for scoring a response to a character-by-character highlighting task. A similarity value for the response is calculated by comparing the response to one or more correct responses to the task to determine the similarity or dissimilarity of the response to the one or more correct responses to the task. A threshold similarity value is calculated for the task, where the threshold similarity value is indicative of an amount of similarity or dissimilarity to the one or more correct responses required for the response to be scored at a certain level. The similarity value for the response is compared to the threshold similarity value. A score is assigned at, above, or below the certain level based on the comparison.

Abstract: Systems and methods described herein automate imposture detection in, e.g., test settings based on voice samples. Based on user instructions, a processing system may determine at least one set of appointments, each having voice samples and a voice print, and a comparison plan for comparing the appointments. The comparison plan defines a plurality of appointment pairs. For each appointment pair, the system compares the associated first and second appointments by, e.g., comparing the first appointment's voice samples to the second appointment's voice print and generating corresponding raw scores, which may be used to compute a composite score. If the composite score satisfies a predetermined threshold condition for fraud, the system may determine whether flagging/holding criteria are satisfied by the raw scores. If the criteria are satisfied, a flag or hold notice may be associated with the appointment pair to trigger an appropriate system/human response (e.g., withholding the appointments' test results).

Abstract: A computer-based testing system includes testing stations connected to a testing service center and backend via the Internet for providing testing services. The system is operable to perform state management to implement fault recovery due to a computing device failure while a test is being administered. The system is also operable to utilize multiple caching techniques for mitigating network latency while administering tests.

Abstract: Systems and methods are provided for assigning a difficulty score to a speech sample. Speech recognition is performed on a digitized version of the speech sample using an acoustic model to generate word hypotheses for the speech sample. Time alignment is performed between the speech sample and the word hypotheses to associate the word hypotheses with corresponding sounds of the speech sample. A first difficulty measure is determined based on the word hypotheses, and a second difficulty measure is determined based on acoustic features of the speech sample. A difficulty score for the speech sample is generated based on the first difficulty measure and the second difficulty measure.

Abstract: Systems and methods are described for generating a scoring model for responses. A computer-implemented method of calibrating a scoring model using a processing system for scoring examinee responses includes accessing a plurality of training responses for training the scoring model. The plurality of training responses are analyzed to derive values of multiple features (variables) of the training responses. The scoring model is trained based on the values of the multiple features of the training responses and one or more external measures of proficiency for each individual associated with a training response utilized in the training. The one or more external measures are not derived from the training responses. Based on the training, a weight for each of the multiple features is determined. The scoring model is calibrated to include the weights for at least some of the features for scoring examinee responses.

Abstract: Systems and methods are provided for automated detection of plagiarized spoken responses. A spoken response is processed to generate a text that is representative of the spoken response. The text is processed to remove disfluencies in the text and to identify a plurality of sentences in the text. A first numerical measure indicative of a number of words and phrases of the text that are included verbatim in a source text is determined. The source text has been designated as a source of plagiarized content. A second numerical measure indicative of an amount of the text that paraphrases portions of the source text is determined. A third numerical measure indicative of a similarity between sentences of the text and sentences of the source text is determined. A model is applied to the first, second, and third numerical measures to classify the spoken response as being plagiarized or non-plagiarized.

Abstract: Systems and methods are provided for detecting punctuation errors in a text including one or more sentences. A sentence including a plurality of words is received, the sentence including one or more preexisting punctuation marks. One or more punctuation marks are determined with a statistical classifier based on a set of rules, to be inserted in the sentence. The determined punctuation marks are compared with the preexisting punctuation marks. A report of punctuation errors is output based on the comparison.

Abstract: Systems and methods are provided for scoring non-native speech. Two or more speech samples are received, where each of the samples are of speech spoken by a non-native speaker, and where each of the samples are spoken in response to distinct prompts. The two or more samples are concatenated to generate a concatenated response for the non-native speaker, where the concatenated response is based on the two or more speech samples that were elicited using the distinct prompts. A concatenated speech proficiency metric is computed based on the concatenated response, and the concatenated speech proficiency metric is provided to a scoring model, where the scoring model generates a speaking score based on the concatenated speech metric.

Abstract: Systems and methods are provided for correcting a grammatical error in a text sequence. A first text sequence in a first language is received. The first text sequence is translated to a second language to provide a first translated text. The first text sequence is translated to a third language to provide a second translated text. The third language is different from the second language. The first translated text is translated to the first language to provide a first back translation. The second translated text is translated to the first language to provide a second back translation. A plurality of candidate text sequences that include features of the first back translation and the second back translation are determined. The plurality of candidate text sequences include alternative grammatical options for the first text sequence. The plurality of candidate text sequences are scored with the processing system.

Abstract: A computer-implemented method, system, and computer program product for generating vector-based similarity scores in text document comparisons considering confounding effects of document length. Vector-based methods for comparing the semantic similarity between texts (such as Content Vector Analysis and Random Indexing) have a characteristic which may reduce their usefulness for some applications: the similarity estimates they produce are strongly correlated with the lengths of the texts compared. The statistical basis for this confound is described, and suggests the application of a pivoted normalization method from information retrieval to correct for the effect of document length. In two text categorization experiments, Random Indexing similarity scores using pivoted normalization are shown to perform significantly better than standard vector-based similarity estimation methods.

Abstract: Computer-implemented systems and methods are provided for assessing non-native speech proficiency. A non-native speech sample is processed to identify a plurality of vowel sound boundaries in the non-native speech sample. Portions of the non-native speech sample are analyzed within the vowel sound boundaries to extract vowel characteristics. The vowel characteristics are used to identify a plurality of vowel space metrics for the non-native speech sample, and the vowel space metrics are used to determine a non-native speech proficiency score for the non-native speech sample.