Abstract: A method and system for using a natural language generator for automatic assessment item generation is disclosed. The natural language generator includes a document structure generator that produces an abstract document specification defining a structure for an assessment item based on user input. The abstract document specification is input into a logical schema generator, which produces a logical schema specification that creates a more detailed specification for an assessment item. Finally, a sentence generator receives the logical schema specification and creates natural language for the assessment item based on the variables defined in the logical schema specification.

Abstract: Systems and methods are provided for selecting a proposed test item for inclusion in an examination where a non-multiple choice response to the proposed test item will be automatically scored. A proposed test item is analyzed to generate a proposed test item metric, where the proposed test item is a non-multiple choice test item. The proposed test item metric is provided to a proposed test item scoring model, where the proposed test item scoring model outputs a likelihood score indicative of a likelihood that automated scoring of a response to the proposed test item would be at or above a quality level. The proposed test item is selected for inclusion in the examination based on the likelihood score.

Abstract: Systems and methods are described for providing a multi-modal evaluation of a presentation. A system includes a motion capture device configured to detect motion an examinee giving a presentation and an audio recording device configured to capture audio of the examinee giving the presentation. One or more data processors are configured to extract a non-verbal feature of the presentation based on data collected by the motion capture device and an audio feature of the presentation based on data collected by the audio recording device. The one or more data processors are further configured to generate a presentation score based on the non-verbal feature and the audio feature.

Abstract: Systems and methods are provided for a processor-implemented method of analyzing quality of sound acquired via a microphone. An input metric is extracted from a sound recording at each of a plurality of time intervals. The input metric is provided at each of the time intervals to a neural network that includes a memory component, where the neural network provides an output metric at each of the time intervals, where the output metric at a particular time interval is based on the input metric at a plurality of time intervals other than the particular time interval using the memory component of the neural network. The output metric is aggregated from each of the time intervals to generate a score indicative of the quality of the sound acquired via the microphone.

Abstract: Systems and methods are provided for scoring a constructed response generated by a user and providing information on the user's writing behavior. A constructed response and associated electronic process log are received. The constructed response is processed to generate first feature values representative of aspects of the constructed response. The electronic process log is processed to generate second feature values related to the user's actions in generating the constructed response. A score for the constructed response is generated using the processing system by applying a computer scoring model to the first and second feature values. A rule of a rule engine that is satisfied is identified, the rule being satisfied when one or more feature values of the second feature values meet a condition associated with the rule. Information on the user's actions in generating the constructed response is provided based on the satisfied rule.

Abstract: Systems and methods are provided for determining a susceptibility of a computer-implemented automated scoring engine to gaming strategies. A plurality of responses to a prompt are provided to a computer-implemented automated scoring engine to receive a first set of scores. A first transformation is performed on each of the plurality of responses to generate a first set of transformed responses. The first set of transformed responses is provided to the computer-implemented automatic scoring engine to receive a second set of scores, and a gaming susceptibility metric is determined based on the first set of scores and the second set of scores.

Abstract: Computer-implemented systems and methods for evaluating a performance are provided. Motion of a user in a performance is detected using a motion capture device. Data collected by the motion capture device is processed with a processing system to identify occurrences of first and second types of actions by the user. The data collected by the motion capture device is processed with the processing system to determine values indicative of amounts of time between the occurrences. A non-verbal feature of the performance is determined based on the identified occurrences and the values. A score for the performance is generated using the processing system by applying a computer scoring model to the non-verbal feature.

Abstract: Computer-implemented systems and methods are provided for determining a document's complexity. For example, a computer performing the complexity analysis can receive a document. The computer can determine the content words within the document and determine an association measure for each group of content words. An association profile can be created for the document using the association measures. The computer can use the association profile to determine the complexity of the document. The complexity of the document may correspond to the document's suitable reading level or, if the document is an essay, an essay score.

Abstract: Systems and methods are provided for providing voice authentication of a candidate speaker. Training data sets are accessed, where each training data set comprises data associated with a training speech sample of a speaker and a plurality of speaker metrics, where the plurality of speaker metrics include a native language of the speaker. The training data sets are used to train a neural network, where the data associated with each training speech sample is a training input to the neural network, and each of the plurality of speaker metrics is a training output to the neural network. Data associated with a speech sample is provided to the neural network to generate a vector that contains values for the plurality of speaker metrics, and the values contained in the vector are compared to values contained in a reference vector associated with a known person to determine whether the candidate speaker is the known person.

Abstract: Provide automatic assessment of oral recitations during computer based language assessments using a trained neural network to automate the scoring and feedback processes without human transcription and scoring input by automatically generating a score of a language assessment. Providing an automatic speech recognition (“ASR”) scoring system. Training multiple scoring reference vectors associated with multiple possible scores of an assessment, and receiving an acoustic language assessment response to an assessment item. Based on the acoustic language assessment automatically generating a transcription, and generating an individual word vector from the transcription. Generating an input vector by concatenating an individual word vector with a transcription feature vector, and supplying an input vector as input to a neural network. Generating an output vector based on weights of a neural network; and generating a score by comparing an output vector with scoring vectors.

Abstract: Systems and methods are provided for measuring a user's English language proficiency. A constructed response generated by a user is received, the constructed response being based on a picture. The constructed response is processed to determine a first numerical measure indicative of a presence of one or more grammar errors in the constructed response. The constructed response is processed to determine a second numerical measure indicative of a degree to which the constructed response describes a subject matter of the picture. The constructed response is processed to determine a third numerical measure indicative of a degree of awkward word usage in the constructed response. A model is applied to the first, second, and third numerical measures to determine a score for the constructed response indicative of the user's English language proficiency. The model includes first, second, and third variables with associated first, second, and third weighting factors, respectively.

Abstract: Computer-implemented systems and methods are provided for automatically generating recitation items. For example, a computer performing the recitation item generation can receive one or more text sets that each includes one or more texts. The computer can determine a value for each text set using one or more metrics, such as a vocabulary difficulty metric, a syntactic complexity metric, a phoneme distribution metric, a phonetic difficulty metric, and a prosody distribution metric. Then the computer can select a final text set based on the value associated with each text set. The selected final text set can be used as the recitation items for a speaking assessment test.

Abstract: Systems and methods described herein utilize supervised machine learning to generate a figure-of-speech prediction model for classify content words in a running text as either being figurative (e.g., as a metaphor, simile, etc.) or non-figurative (i.e., literal). The prediction model may extract and analyze any number of features in making its prediction, including a topic model feature, unigram feature, part-of-speech feature, concreteness feature, concreteness difference feature, literal context feature, non-literal context feature, and off-topic feature, each of which are described in detail herein. Since uses of figure of speech in writings may signal content sophistication, the figure-of-speech prediction model allows scoring engines to further take into consideration a text's use of figure of speech when generating a score.

Abstract: Systems and methods are provided for measuring a user's English language proficiency. A constructed response generated by a user is received, the constructed response being based on a picture. The constructed response is processed to determine a first numerical measure indicative of a presence of one or more grammar errors in the constructed response. The constructed response is processed to determine a second numerical measure indicative of a degree to which the constructed response describes a subject matter of the picture. The constructed response is processed to determine a third numerical measure indicative of a degree of awkward word usage in the constructed response. A model is applied to the first, second, and third numerical measures to determine a score for the constructed response indicative of the user's English language proficiency. The model includes first, second, and third variables with associated first, second, and third weighting factors, respectively.

Abstract: Computer-implemented systems and methods are provided for scoring content of a spoken response to a prompt. A scoring model is generated for a prompt, where generating the scoring model includes generating a transcript for each of a plurality of training responses to the prompt, dividing the plurality of training responses into clusters based on the transcripts of the training responses, selecting a subset of the training responses in each cluster for scoring, scoring the selected subset of training responses for each cluster, and generating content training vectors using the transcripts from the scored subset. A transcript is generated for a received spoken response to be scored, and a similarity metric is computed between the transcript of the spoken response to be scored and the content training vectors. A score is assigned to the spoken response based on the determined similarity metric.

Abstract: Systems and methods are provided for evaluating a user's performance. A first mobile communications device is configured to capture first audio and visual data of a user's performance. The first audio and video data are analyzed by the first device during the user's performance to determine if audio and video quality requirements are met. A second mobile communications device is configured to capture second audio and visual data of the user's performance, the second visual data capturing a different field of view than the first visual data. The second audio and visual data are transmitted by the second device to a remote computing system. The transmitted data is analyzed at the remote computing system to determine if the user is receiving utilizing unauthorized material in the performance. A computer-based scoring system is configured to determine a score for the user's performance based on the first audio and visual data.

Abstract: Systems and methods are provided for providing a computer-based test to be taken at a computer. A determination is made as to whether a geolocation of the computer is within an approved geolocation range. A determination is made as to whether a time at which the computer-based test is to be provided to the computer is within an approved time range. A hardware profile of the testing center is compared to an expected hardware profile of the testing center. Biometric data of a test center administrator is compared to expected biometric data of the test center administrator, and a confidence score is modified based on results of the comparison. If the confidence score meets or exceeds a threshold, the computer-based test is provided to the computer over a network. If the confidence score is less than the threshold, the computer-based test is prevented from being provided to the computer.

Abstract: Systems and methods are provided for rendering an assessment item. A document encoded in a markup language is received. The document includes data that relates to at least one of an appearance and content of an assessment item, and the document includes a reference to a content file. The content file is retrieved, where the content file includes computer-executable instructions for generating an interactive graphical representation for display on a display device. The interactive graphical representation includes one or more features with which a user interacts to respond to the assessment item. The computer-executable instructions are executed based on the data of the document to generate the interactive graphical representation. Data generated based on user manipulation of the one or more features is received. The data generated from the user manipulation is processed to generate a modified graphical representation responsive to the assessment item.

Abstract: A method for scoring non-native speech includes receiving a speech sample spoken by a non-native speaker and performing automatic speech recognition and metric extraction on the speech sample to generate a transcript of the speech sample and a speech metric associated with the speech sample. The method further includes determining whether the speech sample is scorable or non-scorable based upon the transcript and speech metric, where the determination is based on an audio quality of the speech sample, an amount of speech of the speech sample, a degree to which the speech sample is off-topic, whether the speech sample includes speech from an incorrect language, or whether the speech sample includes plagiarized material. When the sample is determined to be non-scorable, an indication of non-scorability is associated with the speech sample. When the sample is determined to be scorable, the sample is provided to a scoring model for scoring.

Abstract: Systems and methods are provided for automatically generating a coherence score for a text using a scoring model. A lexical chain is identified within a text to be scored, where the lexical chain comprises a set of words spaced within the text. A discourse element is identified within the text, where the discourse element comprises a word within the text. A coherence metric is determined based on a relationship between the lexical chain and the discourse element. A coherence score is generated using a scoring model by providing the coherence metric to the scoring model.