Patents Assigned to Educational Testing Service
  • Patent number: 11417339
    Abstract: Data is received that encapsulates a spoken response to a test question. Thereafter, the received data is transcribed into a string of words. The string of words is then compared with at least one source string so that a similarity grid representation of the comparison can be generated that characterizes a level of similarity between the string of words and the at least one source string. The grid representation is then scored using at least one machine learning model. The score indicates a likelihood of the spoken response having been plagiarized. Data providing the encapsulated score can then be provided. Related apparatus, systems, techniques and articles are also described.
    Type: Grant
    Filed: November 26, 2019
    Date of Patent: August 16, 2022
    Assignee: Educational Testing Service
    Inventors: Xinhao Wang, Keelan Evanini, Yao Qian, Klaus Zechner
  • Patent number: 11410072
    Abstract: Systems and methods are provided for the detection of sentiment in writing. A plurality of texts is received from a larger collection of writing samples with a computer system. A set of seed words from the plurality of texts are labeled as being of positive sentiment or of negative sentiment with the computer system. The set of seed words is expanded in size with the computer system to provide an expanded set of seed words. Intensity values are assigned to words of the expanded set of seed words. Each of the words of the expanded set of seed words is assigned three intensity values: a value corresponding to the strength of the word's association with a positive polarity class, a value corresponding to the strength of the word's association with a negative polarity class, and a value corresponding to the strength of the word's association with a neutral polarity class.
    Type: Grant
    Filed: October 19, 2012
    Date of Patent: August 9, 2022
    Assignee: Educational Testing Service
    Inventors: Jill Burstein, Beata Beigman Klebanov, Joel Tetreault, Nitin Madnani, Adam Faulkner
  • Patent number: 11398161
    Abstract: Computer-implemented systems and methods are described herein for detecting unusually frequent exactly matching and nearly matching test responses. A plurality of test responses for each of a plurality of test takers is received. Evidence of a circulated key from the plurality of test responses is identified by detecting two or more test takers with exactly matching test responses for a plurality of questions. The strength of the evidence of the circulated key is analyzed by determining the probability that the two or more test takers would produce exactly matching test responses, wherein the strength of the evidence is inversely related to the probability. Test takers with nearly matching test responses to the circulated key are then identified. All test takers with exactly matching and nearly matching test responses are tagged for further investigation.
    Type: Grant
    Filed: May 10, 2019
    Date of Patent: July 26, 2022
    Assignee: Educational Testing Service
    Inventors: Shelby J. Haberman, Yi-Hsuan Lee, Paul Papierman, Yuxing Zhou, Ruchi Subhedar
  • Patent number: 11238844
    Abstract: Systems and methods for identifying a person's native language and/or non-native language based on code-switched text and/or speech, are presented. The systems may be trained using various methods. For example, a language identification system may be trained using one or more code-switched corpora. Text and/or speech features may be extracted from the corpora and used, in combination with a per-word language identify of the text and/or speech, to train at least one machine learner. Code-switched text and/or speech may be received and processed by extracting text and/or speech features. These features may be fed into the at least one machine learner to identify the person's native language.
    Type: Grant
    Filed: January 23, 2019
    Date of Patent: February 1, 2022
    Assignee: Educational Testing Service
    Inventors: Vikram Ramanarayanan, Robert Pugh, Yao Qian, David Suendermann-Oeft
  • Patent number: 11222627
    Abstract: Systems and methods are provided for conducting a simulated conversation with a language learner include determining a first dialog state of the simulated conversation. First audio data corresponding to simulated speech based on the dialog state is transmitted. Second audio data corresponding to a variable length utterance spoken in response to the simulated speech is received. A fixed dimension vector is generated based on the variable length utterance. A semantic label is predicted for the variable-length utterance based on the fixed dimension vector. A second dialog state of the simulated conversation is determined based on the semantic label, and third audio data corresponding to simulated speech is transmitted based on the second dialog state.
    Type: Grant
    Filed: November 21, 2018
    Date of Patent: January 11, 2022
    Assignee: Educational Testing Service
    Inventors: Yao Qian, Rutuja Ubale, Vikram Ramanarayanan, Patrick Lange, David Suendermann-Oeft, Keelan Evanini, Eugene Tsuprun
  • Patent number: 11195431
    Abstract: Systems and methods for defining and using an optimal burst boundary threshold to assess the reliability of a manual/automatic writing score are presented. Keystroke data, including inter-key interval data, such as inter-word interval data, may be gathered from writings. Clustering analyses may be performed on the inter-key interval data to determine an optimal number of bursts for the writings. An optimal burst boundary may be determined from the optimal number of bursts. Other burst-related measures and statistics, including the average and maximum burst lengths, may be determined from the writings based on the optimal burst boundary threshold. A score may be received for each of the writings. A validation indication metric may be generated for each of the writings based on the received score and the optimal burst boundary threshold. The resulting measures and statistics may be used or applied in different ways and provide personalized feedback as learning analytics.
    Type: Grant
    Filed: March 12, 2019
    Date of Patent: December 7, 2021
    Assignee: Educational Testing Service
    Inventors: Mo Zhang, Jiangang Hao, Paul Deane, Chen Li
  • Patent number: 11151894
    Abstract: Methods and systems for scoring an argument critique written essay, including identifying a discourse segment of the argument critique written essay, determining a position of each sentence in the discourse segment, classifying sentences into discourse units, evaluating an argumentative content of each sentence, and assigning an argumentative score to the essay based on the argumentative content of each sentence in the discourse segment of that essay. Methods and systems for training the scoring method are also disclosed. Corresponding apparatuses, systems, and methods are also disclosed.
    Type: Grant
    Filed: February 2, 2018
    Date of Patent: October 19, 2021
    Assignee: Educational Testing Service
    Inventors: Beata Beigman Klebanov, Binod Gyawali
  • Patent number: 11132913
    Abstract: Systems and methods are provided for acquiring physical-world data indicative of interactions of a subject with an avatar for evaluation. An interactive avatar is provided for interaction with the subject. Speech from the subject to the avatar is captured, and automatic speech recognition is performed to determine content of the subject speech. Motion data from the subject interacting with the avatar is captured. A next action of the interactive avatar is determined based on the content of the subject speech or the motion data. The next action of the avatar is implemented, and a score for the subject is determined based on the content of the subject speech and the motion data.
    Type: Grant
    Filed: April 20, 2016
    Date of Patent: September 28, 2021
    Assignee: Educational Testing Service
    Inventors: Vikram Ramanarayanan, Mark Katz, Eric Steinhauer, Ravindran Ramaswamy, David Suendermann-Oeft
  • Patent number: 11094335
    Abstract: Systems and methods are provided for automatic detection of plagiarized spoken responses during standardized testing of language proficiency. A first spoken response to a speaking task that elicits spontaneous speech and a second spoken response to a source-based speaking task that is assumed to be non-plagiarized are digitally recorded. A first and second sets of speaking proficiency features are calculated for the first and the second spoken response, respectively. The first spoken response is classifying as plagiarized or non-plagiarized based on the comparison between the first and the second set of speaking proficiency features. Corresponding apparatuses, systems, and methods are also disclosed.
    Type: Grant
    Filed: July 21, 2017
    Date of Patent: August 17, 2021
    Assignee: Educational Testing Service
    Inventors: Xinhao Wang, Keelan Evanini, James Bruno, Matthew Mulholland
  • Patent number: 11049409
    Abstract: Systems and methods are provided for automatically scoring a response and statistically revaluating whether it can be considered as aberrant. In one embodiment, a constructed response is evaluated via a pre-screening stage and a post-hoc screening stage. The pre-screening stage attempts to determine whether the constructed response is aberrant based on a variety of aberration metrics and criteria. If the constructed response is deemed not to be aberrant, then the post-hoc screening stage attempts to predict a discrepancy between what score an automated scoring system would assigned and what score a human rater would assign to the response. If the discrepancy is sufficiently low, then the constructed response may be scored by an automated scoring engine. On the other hand, if the constructed response failed to pass either of the two stages, then a flag may be raised to indicate that additional human review may be needed.
    Type: Grant
    Filed: December 18, 2015
    Date of Patent: June 29, 2021
    Assignee: Educational Testing Service
    Inventors: Mo Zhang, Jing Chen, Andre Alexander Rupp, David Michael Williamson
  • Patent number: 11024194
    Abstract: Systems and methods of improving reading fluency by presenting a virtual reading partner that takes turns reading aloud with a reader, are presented. First audio data comprising speech corresponding to the reader reading aloud from a first portion of a written transcript is received. The first audio data is processed to identify spoken words from the speech. The reader's speaking of the words in the first portion of the written transcript are compared to correct speaking of words in the first portion of the written transcript. A reading fluency of the reader is assessed. Second audio data comprising a pre-recorded narration of a second portion of the written transcript or automatically generated text to speech of the second portion is generated. The virtual reading partner is presented to the reader and takes turns reading aloud with the reader until the end of the written transcript has been reached.
    Type: Grant
    Filed: November 16, 2017
    Date of Patent: June 1, 2021
    Assignee: Educational Testing Service
    Inventors: Beata Beigman Klebanov, Rene R. Lawless, John Sabatini, Markku T. Hakkinen
  • Patent number: 11023684
    Abstract: Computer-implemented systems and methods are described herein for automatically generating questions from text. Text including one or more sentences is received. A sentence, comprising a predicate and one or more arguments associated with the predicate, is parsed from the text. Semantic role labels are assigned to the one or more arguments associated with the predicate. One or more questions are automatically generated relating to the predicate based on the assigned semantic role labels. Each answer to the generated questions is one of the one or more arguments associated with the predicate.
    Type: Grant
    Filed: March 19, 2019
    Date of Patent: June 1, 2021
    Assignee: Educational Testing Service
    Inventors: Michael Flor, Brian W. Riordan
  • Patent number: 10964224
    Abstract: Systems and methods are provided for providing feedback on a user's writing behavior in generating a constructed response. An electronic process log for the constructed response is received. The electronic process log is processed to generate a vector having a predetermined number of elements. The vector comprises information related to the user's actions in generating the constructed response and includes (i) data indicating types of actions performed by the user in generating the constructed response, (ii) time points associated with the actions, and (iii) locations associated with the actions, each location indicating whether an associated action occurred within a word, between words, within a phrase, between phrases, within a sentence, between sentences, within a paragraph, or between paragraph. The vector is compared to one or more vectors associated with other constructed responses, and feedback is generated based on the comparison.
    Type: Grant
    Filed: March 13, 2017
    Date of Patent: March 30, 2021
    Assignee: Educational Testing Service
    Inventors: Mo Zhang, Jiangang Hao, Paul Deane
  • Patent number: 10937444
    Abstract: A system for end-to-end automated scoring is disclosed. The system includes a word embedding layer for converting a plurality of ASR outputs into input tensors; a neural network lexical model encoder receiving the input tensors; a neural network acoustic model encoder implementing AM posterior probability, word duration, mean value of pitch and mean value of intensity based on a plurality of cues; and a linear regression module, for receiving concatenated encoded features from the neural network lexical model encoder and the neural network acoustic model encoder.
    Type: Grant
    Filed: November 20, 2018
    Date of Patent: March 2, 2021
    Assignee: Educational Testing Service
    Inventors: David Suendermann-Oeft, Lei Chen, Jidong Tao, Shabnam Ghaffarzadegan, Yao Qian
  • Patent number: 10885274
    Abstract: Systems and methods are provided for processing a response to essay prompts that request a narrative response. A data structure associated with a narrative essay is accessed. The essay is analyzed to generate an organization subscore, where the organization subscore is generated using a graph metric by identifying content words in each sentence of the essay and populating a data structure with links between related content words in neighboring sentences, wherein the organization subscore is determined based on the links. The essay is analyzed to generate a development subscore, where the development subscore is generated using a transition metric by accessing a transition cue data store and identifying transition words in the essay, wherein the development subscore is based on a number of words in the essay that match words in the transition cue data store. A narrative quality metric is determined based on the organization subscore and the development subscore.
    Type: Grant
    Filed: June 21, 2018
    Date of Patent: January 5, 2021
    Assignee: Educational Testing Service
    Inventors: Swapna Somasundaran, Michael Flor, Martin Chodorow, Binod Gyawali, Hillary Molloy, Laura McCulla
  • Patent number: 10867525
    Abstract: Computer-implemented systems and methods are provided for automatically generating recitation items. For example, a computer performing the recitation item generation can receive one or more text sets that each includes one or more texts. The computer can determine a value for each text set using one or more metrics, such as a vocabulary difficulty metric, a syntactic complexity metric, a phoneme distribution metric, a phonetic difficulty metric, and a prosody distribution metric. Then the computer can select a final text set based on the value associated with each text set. The selected final text set can be used as the recitation items for a speaking assessment test.
    Type: Grant
    Filed: February 13, 2018
    Date of Patent: December 15, 2020
    Assignee: Educational Testing Service
    Inventors: Su-Youn Yoon, Lei Chen, Keelan Evanini, Klaus Zechner
  • Patent number: 10803318
    Abstract: Systems and methods are provided for scoring video clips using visual feature extraction. A signal including a video clip of a subject is received. For each frame of the video clip, physiological features of the subject visually rendered in the video clip are extracted. A plurality of visual words associated with the extracted physiological features are determined. A document including the plurality of visual words is generated. A plurality of feature vectors associated with the document are determined. The plurality of feature vectors to a regression model for scoring are provided.
    Type: Grant
    Filed: May 17, 2017
    Date of Patent: October 13, 2020
    Assignee: Educational Testing Service
    Inventors: Lei Chen, Gary Feng, Chee Wee Leong, Chong Min Lee
  • Patent number: 10796595
    Abstract: Systems and methods for training raters to rate constructed responses to tasks are described herein. In one embodiment, a plurality of trainee raters are selected without regard to their prior experience. The trainee raters are then train in individual training sessions, during which they are asked to rate responses to a task. Each session presents to the trainee rater the task, a rating rubric, and training responses to the task. The training program receives ratings assigned by the trainee rater to the training responses through a graphical user interface. Upon receiving the assigned rating, the training program presents feedback substantially immediately and determines a score for the trainee rater's assigned rating. Thereafter, qualified raters are selected from the plurality of trainee raters based upon their performance during the training sessions as compared with a statistical model. Operational constructed responses are then assigned to rated by the qualified raters.
    Type: Grant
    Filed: November 16, 2015
    Date of Patent: October 6, 2020
    Assignee: Educational Testing Service
    Inventor: Yigal Attali
  • Patent number: 10783873
    Abstract: Systems and methods for identifying a person's native language, are presented. A native language identification system, comprising a plurality of artificial neural networks, such as time delay deep neural networks, is provided. Respective artificial neural networks of the plurality of artificial neural networks are trained as universal background models, using separate native language and non-native language corpora. The artificial neural networks may be used to perform voice activity detection and to extract sufficient statistics from the respective language corpora. The artificial neural networks may use the sufficient statistics to estimate respective T-matrices, which may in turn be used to extract respective i-vectors. The artificial neural networks may use i-vectors to generate a multilayer perceptron model, which may be used to identify a person's native language, based on an utterance by the person in his or her non-native language.
    Type: Grant
    Filed: December 17, 2018
    Date of Patent: September 22, 2020
    Assignee: Educational Testing Service
    Inventors: Yao Qian, Keelan Evanini, Patrick Lange, Robert A. Pugh, Rutuja Ubale
  • Patent number: 10769962
    Abstract: Computer-implemented systems and methods for generating a personalization score for a constructed response are provided. A constructed response generated by a user is received. The constructed response is processed with a processing system to generate one or more feature values representative of one or more aspects of the constructed response. A personalization score for the constructed response is generated using the processing system by applying a computer scoring model to the one or more feature values. The personalization score provides a measure of a personalization of the constructed response.
    Type: Grant
    Filed: May 3, 2016
    Date of Patent: September 8, 2020
    Assignee: Educational Testing Service
    Inventors: Beata Beigman Klebanov, Jill Burstein