Abstract: A Portal Assessment Design System includes a system for designing assessment models and assessments (e.g., standardized tests). In assessment design, specifications are organized around three basic elements: 1) what knowledge, skill or abilities are important to measure; 2) what is necessary evidence of that knowledge, skill or ability; 3) what performance situations are needed to elicit the required evidence from students and other assessment users. The system includes three modules (i.e., Domain Analysis, Domain Modeling, Conceptual Assessment Framework), each of which targets a specific phase of the design process, for designing an assessment in accordance with the three basic elements. The sequence of modules represents the structure of a design process that helps assessment developers work through successive stages or phases of assessment design, from the gathering of raw information to the creation of a complete set of specifications for an assessment product.

Abstract: Systems and methods are provided for assigning a difficulty score to a speech sample. Speech recognition is performed on a digitized version of the speech sample using an acoustic model to generate word hypotheses for the speech sample. Time alignment is performed between the speech sample and the word hypotheses to associate the word hypotheses with corresponding sounds of the speech sample. A first difficulty measure is determined based on the word hypotheses, and a second difficulty measure is determined based on acoustic features of the speech sample. A difficulty score for the speech sample is generated based on the first difficulty measure and the second difficulty measure.

Abstract: An on-line teaching and learning system with rapid change-by-change or real time reinforcement signals for students and simultaneous monitoring by the teacher of the actual responses as well as their correctness is disclosed. The system may include at least one teacher computer, a plurality of student computers operably connected to the at least one teacher computer by a communications network. The system may be used in a classroom setting or in a distance-learning environment.

Abstract: A Portal Assessment Design System includes a system for designing assessment models and assessments (e.g., standardized tests). In assessment design, specifications are organized around three basic elements: 1) what knowledge, skill or abilities are important to measure; 2) what is necessary evidence of that knowledge, skill or ability; 3) what performance situations are needed to elicit the required evidence from students and other assessment users. The system includes three modules (i.e., Domain Analysis, Domain Modeling, Conceptual Assessment Framework), each of which targets a specific phase of the design process, for designing an assessment in accordance with the three basic elements. The sequence of modules represents the structure of a design process that helps assessment developers work through successive stages or phases of assessment design, from the gathering of raw information to the creation of a complete set of specifications for an assessment product.

Abstract: A method and system for customizing an automated essay scoring system are disclosed. Relative weights may be assigned to a plurality of features for a plurality of benchmark essay responses. Based on the relative weights, automated scores may be determined for one or more first essay responses to a first prompt. The automated scores may be scaled based on one or more scoring standards for the first prompt. A scaled automated score may be assigned for a second essay response to the first prompt based on the scaled automated scores for the one or more first essay responses.

Abstract: To automatically annotate an essay, a sentence of the essay is identified and a feature associated with the sentence is determined. In addition, a probability of the sentence being a discourse element is determined by mapping the feature to a model. The model having been generated by a machine learning application based on at least one annotated essay. Furthermore, the essay is annotated based on the probability.

Abstract: Systems and methods are provided for non-monotonic recognition of phrasal terms. Phrasal terms are identified from a corpus of written materials and ranked based on, for example, a mutual rank ratio. The phrasal terms are sequentially selected and a determination is made as to whether to accept or reject the selected phrasal term based on at least one predetermined criteria. The ranking of the phrasal terms may also rely on linguistic support to reduce duplication of phrasal terms and to distinguish different confidence levels for identified and accepted phrasal terms.

Abstract: A computer memory stores a data structure representing a ternary search tree (TST) representing multiple word n-grams for a corpus of documents. The data structure includes plural records in a first memory, each record representing a node of the TST and comprising plural fields. At least some n-grams have a sequence of units. The plurality of fields includes one for identifying a given unit of the sequence for a given node, one reserved for storing payload information for the given node, and plural child fields reserved for storing information for a first, second and third child nodes of the given node. The child fields store a null value indicating the absence of the child node or an identifier identifying a memory location of the child node. For at least one record, at least one of the child fields stores an identifier identifying a memory location of a memory different than the first memory.

Abstract: Systems and methods are provided for identifying factual information in a written document. Named entities and corresponding noun phrases are identified in the written document. A query is built by combining one of the named entities with a respective one of the noun phrases. The query represents an assertion of a potential fact. The query is submitted for comparison with a fact repository which assesses whether the query presents a factual assertion. If the query presents a factual assertion (e.g., it matches a fact within the fact repository), a match is returned. Various modifications may be made to the queries to return additional matches and various combinations of filters may be applied to the matches to filter out less relevant matches.

Abstract: A method of determining a mastery level for an examinee from an assessment is disclosed. The method includes receiving one or more of an overall skill level for an examinee, a weight for the overall skill level, a covariate vector for an examinee, and a weight for the covariate vector. An examinee attribute value is computed using one or more of the received values for each examinee and each attribute. The computation of the examinee attribute values can include estimating the value using a Markov Chain Monte Carlo estimation technique. Examinee mastery levels are then assigned based on each examinee attribute level. Dichotomous or polytomous levels can be assigned based on requirements for the assessment.

Abstract: A computer-based testing system includes testing stations connected to a testing service center and backend via the Internet for providing testing services. The system is operable to perform state management to implement fault recovery due to a computing device failure while a test is being administered. The system is also operable to utilize multiple caching techniques for mitigating network latency while administering tests.

Abstract: Systems and methods are provided for generating a transcript of a speech sample response to a test question. The speech sample response to the test question is provided to a language model, where the language model is configured to perform an automated speech recognition function. The language model is adapted to the test question to improve the automated speech recognition function by providing to the language model automated speech recognition data related to the test question, Internet data related to the test question, or human-generated transcript data related to the test question. The transcript of the speech sample is generated using the adapted language model.

Abstract: Systems and methods are provided for scoring a response to a character-by-character highlighting task. A similarity value for the response is calculated by comparing the response to one or more correct responses to the task to determine the similarity or dissimilarity of the response to the one or more correct responses to the task. A threshold similarity value is calculated for the task, where the threshold similarity value is indicative of an amount of similarity or dissimilarity to the one or more correct responses required for the response to be scored at a certain level. The similarity value for the response is compared to the threshold similarity value. A score is assigned at, above, or below the certain level based on the comparison.

Abstract: A computer-implemented method, system, and computer program product for automatically assessing text difficulty. Text reading difficulty predictions are expressed on a scale that is aligned with published reading standards. Two distinct difficulty models are provided for informational and literary texts. A principal components analysis implemented on a large collection of texts is used to develop independent variables accounting for strong intercorrelations exhibited by many important linguistic features. Multiple dimensions of text variation are addressed, including new dimensions beyond syntactic complexity and semantic difficulty. Feedback about text difficulty is provided in a hierarchically structured format designed to support successful text adaptation efforts.

Abstract: Systems and methods are provided for scoring a speech sample. Automatic speech recognition is performed on the speech sample using an automatic speech recognition system to generate a transcription of the sample. Words in the transcription are associated with parts of speech, and part of speech sequences are extracted from the parts of speech associations. A grammar metric is generated based on the part of speech sequences, and the speech sample is scored based on the grammar metric.

Abstract: In accordance with the teachings described herein, systems and methods are provided for identifying organizational elements in argumentative or persuasive discourse. A text that has been annotated is received. The annotated text includes argumentative or persuasive discourse that includes claims and evidence and organizational elements configured to organize the claims and evidence. Annotations of the annotated text distinguish the organizational elements from the claims and evidence. A rule set or a feature set is identified from the annotated text, where the rule set or the feature set includes textual patterns or word frequency features related to the organizational elements of the annotated text. A model is built based on the annotations and on the rule set or the feature set. The model is configured to identify organizational elements in a new text. The model is applied to the new text.

Abstract: Systems and methods are provided for detecting punctuation errors in a text including one or more sentences. A sentence including a plurality of words is received, the sentence including one or more preexisting punctuation marks. One or more punctuation marks are determined with a statistical classifier based on a set of rules, to be inserted in the sentence. The determined punctuation marks are compared with the preexisting punctuation marks. A report of punctuation errors is output based on the comparison.

Abstract: The present disclosure presents a useful metric for assessing the relative difficulty which non-native speakers face in pronouncing a given utterance and a method and systems for using such a metric in the evaluation and assessment of the utterances of non-native speakers. In an embodiment, the metric may be based on both known sources of difficulty for language learners and a corpus-based measure of cross-language sound differences. The method may be applied to speakers who primarily speak a first language speaking utterances in any non-native second language.

Abstract: Systems and methods for building a model for use in grading an essay are provided. A plurality of human graded essays are evaluated using a processor to generate a set of features. A score category is determined for each of the plurality of human graded essays. A weight is produced using a processor for each feature based on the score category for each of the plurality of human graded essays and the set of features. A model is generated using a processor based on the weights for the set of features for evaluating an essay. The set of features includes n features, where the first k features have optimized weights, and the last n?k features have fixed predetermined weights.

Abstract: Systems and methods for detecting collocation errors in a text sample using a reference database from a corpus are provided. Collocation candidates are identified within the text sample based upon syntactic patterns in the text sample. Whether a given collocation candidate contains a collocation error is detected, the detecting including: determining a first association measure using the reference database for the given collocation candidate; determining whether the first association measure satisfies a predetermined condition and identifying the given collocation candidate as proper if the first association measure satisfies the predetermined condition; determining an additional association measure for a variation of the given collocation candidate using the reference database; and determining whether or not the collocation candidate contains an error based upon the additional association measure of the variation.