Patents Assigned to Educational Testing Service
  • Patent number: 10339826
    Abstract: Systems and methods are provided for automatically scoring essay responses to a prompt using a scoring model. A relevant word corpus and an irrelevant word corpus are accessed. A scoring model is generated by, for each of a plurality of words in the relevant word corpus, determining a topic signature score based on a number of appearances of that word in the relevant word corpus and a number of appearances of that word in the irrelevant word corpus. For each of a plurality of words in an essay response, a topic signature score is determined for that word. A score for the essay response is determined based on the identified topic signature scores.
    Type: Grant
    Filed: October 12, 2016
    Date of Patent: July 2, 2019
    Assignee: Educational Testing Service
    Inventors: Swapna Somasundaran, Martin Chodorow, Jill Burstein
  • Patent number: 10332411
    Abstract: Systems and methods are provided for selecting a proposed test item for inclusion in an examination where a non-multiple choice response to the proposed test item will be automatically scored. A proposed test item is analyzed to generate a proposed test item metric, where the proposed test item is a non-multiple choice test item. The proposed test item metric is provided to a proposed test item scoring model, where the proposed test item scoring model outputs a likelihood score indicative of a likelihood that automated scoring of a response to the proposed test item would be at or above a quality level. The proposed test item is selected for inclusion in the examination based on the likelihood score.
    Type: Grant
    Filed: November 9, 2012
    Date of Patent: June 25, 2019
    Assignee: Educational Testing Service
    Inventor: Jana Sukkarieh
  • Patent number: 10332416
    Abstract: A method and system for using a natural language generator for automatic assessment item generation is disclosed. The natural language generator includes a document structure generator that produces an abstract document specification defining a structure for an assessment item based on user input. The abstract document specification is input into a logical schema generator, which produces a logical schema specification that creates a more detailed specification for an assessment item. Finally, a sentence generator receives the logical schema specification and creates natural language for the assessment item based on the variables defined in the logical schema specification.
    Type: Grant
    Filed: April 12, 2004
    Date of Patent: June 25, 2019
    Assignee: Educational Testing Service
    Inventors: Paul Deane, Derrick Higgins
  • Patent number: 10311743
    Abstract: Systems and methods are described for providing a multi-modal evaluation of a presentation. A system includes a motion capture device configured to detect motion an examinee giving a presentation and an audio recording device configured to capture audio of the examinee giving the presentation. One or more data processors are configured to extract a non-verbal feature of the presentation based on data collected by the motion capture device and an audio feature of the presentation based on data collected by the audio recording device. The one or more data processors are further configured to generate a presentation score based on the non-verbal feature and the audio feature.
    Type: Grant
    Filed: April 8, 2014
    Date of Patent: June 4, 2019
    Assignee: Educational Testing Service
    Inventors: Lei Chen, Gary Feng, Chee Wee Leong, Christopher Kitchen, Chong Min Lee
  • Patent number: 10283142
    Abstract: Systems and methods are provided for a processor-implemented method of analyzing quality of sound acquired via a microphone. An input metric is extracted from a sound recording at each of a plurality of time intervals. The input metric is provided at each of the time intervals to a neural network that includes a memory component, where the neural network provides an output metric at each of the time intervals, where the output metric at a particular time interval is based on the input metric at a plurality of time intervals other than the particular time interval using the memory component of the neural network. The output metric is aggregated from each of the time intervals to generate a score indicative of the quality of the sound acquired via the microphone.
    Type: Grant
    Filed: July 21, 2016
    Date of Patent: May 7, 2019
    Assignee: Educational Testing Service
    Inventors: Zhou Yu, Vikram Ramanarayanan, David Suendermann-Oeft, Xinhao Wang, Klaus Zechner, Lei Chen, Jidong Tao, Yao Qian
  • Patent number: 10262547
    Abstract: Systems and methods are provided for scoring a constructed response generated by a user and providing information on the user's writing behavior. A constructed response and associated electronic process log are received. The constructed response is processed to generate first feature values representative of aspects of the constructed response. The electronic process log is processed to generate second feature values related to the user's actions in generating the constructed response. A score for the constructed response is generated using the processing system by applying a computer scoring model to the first and second feature values. A rule of a rule engine that is satisfied is identified, the rule being satisfied when one or more feature values of the second feature values meet a condition associated with the rule. Information on the user's actions in generating the constructed response is provided based on the satisfied rule.
    Type: Grant
    Filed: November 10, 2015
    Date of Patent: April 16, 2019
    Assignee: Educational Testing Service
    Inventors: Paul Deane, Gary Feng, Mo Zhang, Jiangang Hao, Yoav Bergner, Michael Flor, Michael E. Wagner, Nathan Lederer, Yigal Attali
  • Patent number: 10255820
    Abstract: Systems and methods are provided for determining a susceptibility of a computer-implemented automated scoring engine to gaming strategies. A plurality of responses to a prompt are provided to a computer-implemented automated scoring engine to receive a first set of scores. A first transformation is performed on each of the plurality of responses to generate a first set of transformed responses. The first set of transformed responses is provided to the computer-implemented automatic scoring engine to receive a second set of scores, and a gaming susceptibility metric is determined based on the first set of scores and the second set of scores.
    Type: Grant
    Filed: March 26, 2014
    Date of Patent: April 9, 2019
    Assignee: Educational Testing Service
    Inventors: Derrick Higgins, Isaac Bejar, Michael Heilman, Yoko Futagi, Michael Flor
  • Patent number: 10176365
    Abstract: Computer-implemented systems and methods for evaluating a performance are provided. Motion of a user in a performance is detected using a motion capture device. Data collected by the motion capture device is processed with a processing system to identify occurrences of first and second types of actions by the user. The data collected by the motion capture device is processed with the processing system to determine values indicative of amounts of time between the occurrences. A non-verbal feature of the performance is determined based on the identified occurrences and the values. A score for the performance is generated using the processing system by applying a computer scoring model to the non-verbal feature.
    Type: Grant
    Filed: April 20, 2016
    Date of Patent: January 8, 2019
    Assignee: Educational Testing Service
    Inventors: Vikram Ramanarayanan, Lei Chen, Chee Wee Leong, Gary Feng, David Suendermann-Oeft
  • Patent number: 10134297
    Abstract: Computer-implemented systems and methods are provided for determining a document's complexity. For example, a computer performing the complexity analysis can receive a document. The computer can determine the content words within the document and determine an association measure for each group of content words. An association profile can be created for the document using the association measures. The computer can use the association profile to determine the complexity of the document. The complexity of the document may correspond to the document's suitable reading level or, if the document is an essay, an essay score.
    Type: Grant
    Filed: February 14, 2014
    Date of Patent: November 20, 2018
    Assignee: Educational Testing Service
    Inventors: Beata Beigman Klebanov, Michael Flor
  • Patent number: 10008209
    Abstract: Systems and methods are provided for providing voice authentication of a candidate speaker. Training data sets are accessed, where each training data set comprises data associated with a training speech sample of a speaker and a plurality of speaker metrics, where the plurality of speaker metrics include a native language of the speaker. The training data sets are used to train a neural network, where the data associated with each training speech sample is a training input to the neural network, and each of the plurality of speaker metrics is a training output to the neural network. Data associated with a speech sample is provided to the neural network to generate a vector that contains values for the plurality of speaker metrics, and the values contained in the vector are compared to values contained in a reference vector associated with a known person to determine whether the candidate speaker is the known person.
    Type: Grant
    Filed: September 23, 2016
    Date of Patent: June 26, 2018
    Assignee: Educational Testing Service
    Inventors: Yao Qian, Jidong Tao, David Suendermann-Oeft, Keelan Evanini, Alexei V. Ivanov, Vikram Ramanarayanan
  • Patent number: 9984682
    Abstract: Provide automatic assessment of oral recitations during computer based language assessments using a trained neural network to automate the scoring and feedback processes without human transcription and scoring input by automatically generating a score of a language assessment. Providing an automatic speech recognition (“ASR”) scoring system. Training multiple scoring reference vectors associated with multiple possible scores of an assessment, and receiving an acoustic language assessment response to an assessment item. Based on the acoustic language assessment automatically generating a transcription, and generating an individual word vector from the transcription. Generating an input vector by concatenating an individual word vector with a transcription feature vector, and supplying an input vector as input to a neural network. Generating an output vector based on weights of a neural network; and generating a score by comparing an output vector with scoring vectors.
    Type: Grant
    Filed: March 30, 2017
    Date of Patent: May 29, 2018
    Assignee: Educational Testing Service
    Inventors: Jidong Tao, Lei Chen, Chong Min Lee
  • Patent number: 9959776
    Abstract: Systems and methods are provided for measuring a user's English language proficiency. A constructed response generated by a user is received, the constructed response being based on a picture. The constructed response is processed to determine a first numerical measure indicative of a presence of one or more grammar errors in the constructed response. The constructed response is processed to determine a second numerical measure indicative of a degree to which the constructed response describes a subject matter of the picture. The constructed response is processed to determine a third numerical measure indicative of a degree of awkward word usage in the constructed response. A model is applied to the first, second, and third numerical measures to determine a score for the constructed response indicative of the user's English language proficiency. The model includes first, second, and third variables with associated first, second, and third weighting factors, respectively.
    Type: Grant
    Filed: October 31, 2017
    Date of Patent: May 1, 2018
    Assignee: Educational Testing Service
    Inventors: Swapna Somasundaran, Martin Chodorow, Joel Tetreault
  • Patent number: 9928754
    Abstract: Computer-implemented systems and methods are provided for automatically generating recitation items. For example, a computer performing the recitation item generation can receive one or more text sets that each includes one or more texts. The computer can determine a value for each text set using one or more metrics, such as a vocabulary difficulty metric, a syntactic complexity metric, a phoneme distribution metric, a phonetic difficulty metric, and a prosody distribution metric. Then the computer can select a final text set based on the value associated with each text set. The selected final text set can be used as the recitation items for a speaking assessment test.
    Type: Grant
    Filed: March 17, 2014
    Date of Patent: March 27, 2018
    Assignee: Educational Testing Service
    Inventors: Su-Youn Yoon, Lei Chen, Keelan Evanini, Klaus Zechner
  • Patent number: 9852379
    Abstract: Systems and methods described herein utilize supervised machine learning to generate a figure-of-speech prediction model for classify content words in a running text as either being figurative (e.g., as a metaphor, simile, etc.) or non-figurative (i.e., literal). The prediction model may extract and analyze any number of features in making its prediction, including a topic model feature, unigram feature, part-of-speech feature, concreteness feature, concreteness difference feature, literal context feature, non-literal context feature, and off-topic feature, each of which are described in detail herein. Since uses of figure of speech in writings may signal content sophistication, the figure-of-speech prediction model allows scoring engines to further take into consideration a text's use of figure of speech when generating a score.
    Type: Grant
    Filed: March 6, 2015
    Date of Patent: December 26, 2017
    Assignee: Educational Testing Service
    Inventors: Beata Beigman Klebanov, Chee Wee Leong, Michael Flor, Michael Heilman
  • Patent number: 9836985
    Abstract: Systems and methods are provided for measuring a user's English language proficiency. A constructed response generated by a user is received, the constructed response being based on a picture. The constructed response is processed to determine a first numerical measure indicative of a presence of one or more grammar errors in the constructed response. The constructed response is processed to determine a second numerical measure indicative of a degree to which the constructed response describes a subject matter of the picture. The constructed response is processed to determine a third numerical measure indicative of a degree of awkward word usage in the constructed response. A model is applied to the first, second, and third numerical measures to determine a score for the constructed response indicative of the user's English language proficiency. The model includes first, second, and third variables with associated first, second, and third weighting factors, respectively.
    Type: Grant
    Filed: February 27, 2015
    Date of Patent: December 5, 2017
    Assignee: Educational Testing Service
    Inventors: Swapna Somasundaran, Martin Chodorow, Joel Tetreault
  • Patent number: 9799228
    Abstract: Computer-implemented systems and methods are provided for scoring content of a spoken response to a prompt. A scoring model is generated for a prompt, where generating the scoring model includes generating a transcript for each of a plurality of training responses to the prompt, dividing the plurality of training responses into clusters based on the transcripts of the training responses, selecting a subset of the training responses in each cluster for scoring, scoring the selected subset of training responses for each cluster, and generating content training vectors using the transcripts from the scored subset. A transcript is generated for a received spoken response to be scored, and a similarity metric is computed between the transcript of the spoken response to be scored and the content training vectors. A score is assigned to the spoken response based on the determined similarity metric.
    Type: Grant
    Filed: January 10, 2014
    Date of Patent: October 24, 2017
    Assignee: Educational Testing Service
    Inventors: Lei Chen, Klaus Zechner, Anastassia Loukina
  • Patent number: 9754503
    Abstract: Systems and methods are provided for evaluating a user's performance. A first mobile communications device is configured to capture first audio and visual data of a user's performance. The first audio and video data are analyzed by the first device during the user's performance to determine if audio and video quality requirements are met. A second mobile communications device is configured to capture second audio and visual data of the user's performance, the second visual data capturing a different field of view than the first visual data. The second audio and visual data are transmitted by the second device to a remote computing system. The transmitted data is analyzed at the remote computing system to determine if the user is receiving utilizing unauthorized material in the performance. A computer-based scoring system is configured to determine a score for the user's performance based on the first audio and visual data.
    Type: Grant
    Filed: March 24, 2015
    Date of Patent: September 5, 2017
    Assignee: Educational Testing Service
    Inventor: Gary Feng
  • Patent number: 9720890
    Abstract: Systems and methods are provided for rendering an assessment item. A document encoded in a markup language is received. The document includes data that relates to at least one of an appearance and content of an assessment item, and the document includes a reference to a content file. The content file is retrieved, where the content file includes computer-executable instructions for generating an interactive graphical representation for display on a display device. The interactive graphical representation includes one or more features with which a user interacts to respond to the assessment item. The computer-executable instructions are executed based on the data of the document to generate the interactive graphical representation. Data generated based on user manipulation of the one or more features is received. The data generated from the user manipulation is processed to generate a modified graphical representation responsive to the assessment item.
    Type: Grant
    Filed: August 6, 2014
    Date of Patent: August 1, 2017
    Assignee: Educational Testing Service
    Inventors: Mankit Lee, Sherry Wang, Kenneth H. Berger, Jenny Gray
  • Patent number: 9721080
    Abstract: Systems and methods are provided for providing a computer-based test to be taken at a computer. A determination is made as to whether a geolocation of the computer is within an approved geolocation range. A determination is made as to whether a time at which the computer-based test is to be provided to the computer is within an approved time range. A hardware profile of the testing center is compared to an expected hardware profile of the testing center. Biometric data of a test center administrator is compared to expected biometric data of the test center administrator, and a confidence score is modified based on results of the comparison. If the confidence score meets or exceeds a threshold, the computer-based test is provided to the computer over a network. If the confidence score is less than the threshold, the computer-based test is prevented from being provided to the computer.
    Type: Grant
    Filed: August 20, 2015
    Date of Patent: August 1, 2017
    Assignee: Educational Testing Service
    Inventors: James B. Moran, Michael J. Cripps, Kenneth H. Berger, Robert E. Thornton
  • Patent number: 9704413
    Abstract: A method for scoring non-native speech includes receiving a speech sample spoken by a non-native speaker and performing automatic speech recognition and metric extraction on the speech sample to generate a transcript of the speech sample and a speech metric associated with the speech sample. The method further includes determining whether the speech sample is scorable or non-scorable based upon the transcript and speech metric, where the determination is based on an audio quality of the speech sample, an amount of speech of the speech sample, a degree to which the speech sample is off-topic, whether the speech sample includes speech from an incorrect language, or whether the speech sample includes plagiarized material. When the sample is determined to be non-scorable, an indication of non-scorability is associated with the speech sample. When the sample is determined to be scorable, the sample is provided to a scoring model for scoring.
    Type: Grant
    Filed: March 23, 2015
    Date of Patent: July 11, 2017
    Assignee: Educational Testing Service
    Inventors: Su-Youn Yoon, Derrick Higgins, Klaus Zechner, Shasha Xie, Je Hun Jeon, Keelan Evanini, Guangming Ling, Isaac Bejar