Abstract: Methods and systems for generating and processing an assessment examination answer sheet are disclosed. An answer sheet is formatted by question, column or section using a layout tool and includes calibration marks, and examinee and examination identifiers. The format is stored in a computer-readable form, such as an XML file. A conventional printer prints the answer sheet prior to a test administration. Upon completion, a scanned image of the answer sheet is produced using a conventional scanner. The calibration marks are used to determine relative positions of answer sheet elements and to determine light and dark answer values. The examinee and examination identifiers identify the examinee and the proper stored format. Automatic grading ensues. A graphical user interface (“GUI”) assists grading of the answer sheet. The GUI may use confidence level indicators to denote answer choices or questions for which the grading system cannot precisely determine the intended response.

Abstract: A method and system for performing automatic text analysis is described. A local ranking for one or more contexts with respect to a word and a global ranking for one or more contexts are generated. The rankings are based on the frequency with which the contexts appear in a corpus. A statistic may be generated using the local and global rankings, such as a log ratio rank statistic equal to the logarithm of the global rank divided by local rank, to measure the similarity of contexts with respect to words with which they combine. A source matrix of word to context values is then created. Singular value decomposition is used to create sub-matrices from the source matrix. Vectors from the sub-matrices corresponding to context(s) and/or word(s) are then selected to determine term-term or context-context similarity or term-context correspondence.

Abstract: A tool, method, and system for use in the development of sentence-based test items are disclosed. The tool may include a user interface that may include a database selection field, a sentence pattern entry field, an option pane, and an output pane. The tool may search a database for one or more sentences and may generate one or more responses to the one or more sentences. The one or more sentences and one or more responses may be used to produce the sentence-based test items. The tool may allow test items to be developed more quickly and easily than manual test item authoring. Accordingly, test item development costs may be lowered and test security may be enhanced.

Abstract: A computer memory stores a data structure representing a ternary search tree (TST) representing multiple word n-grams for a corpus of documents. The data structure includes plural records in a first memory, each record representing a node of the TST and comprising plural fields. At least some n-grams have a sequence of units. The plurality of fields includes one for identifying a given unit of the sequence for a given node, one reserved for storing payload information for the given node, and plural child fields reserved for storing information for a first, second and third child nodes of the given node. The child fields store a null value indicating the absence of the child node or an identifier identifying a memory location of the child node. For at least one record, at least one of the child fields stores an identifier identifying a memory location of a memory different than the first memory.

Abstract: A method and system for determining a lexical association of phrasal terms are described. A corpus having a plurality of words is received, and a plurality of contexts including one or more context words proximate to a word in the corpus is determined. An occurrence count for each context is determined, and a global rank is assigned based on the occurrence count. Similarly, a number of occurrences of a word being used in a context is determined, and a local rank is assigned to the word-context pair based on the number of occurrences. A rank ratio is then determined for each word-context pair. A rank ratio is equal to the global rank divided by the local rank for a word-context pair. A mutual rank ratio is determined by multiplying the rank ratios corresponding to a phrase. The mutual rank ratio is used to identify phrasal terms in the corpus.

Abstract: A method and system for constructing a test using a computer system that performs specification matching during the test creation process is disclosed. A test developer determines one or more test item databases from which to select test items. The test item databases are organized based on psychometric and/or content specifications. The developer can examine the textual passages, artwork or statistical information pertaining to a test item before selecting it by clicking on a designation of the test item in a database. The developer can then add the test item to a list of test items for the test. The test development system updates pre-designated psychometric and content specification information as the developer adds each test item to the test. The test developer can use the specification information to determine whether to add to, subtract from, or modify the list of test items selected for the test.

Abstract: Disclosed herein is a method of analyzing large scale survey results comprising obtaining a sparse data set representing a subset of an original data set comprising a plurality of individuals' responses to a plurality of questions, wherein the sparse data set comprises less than ninety percent of the responses in the original data set; analyzing the sparse data set using a general diagnostic model; and obtaining estimated person parameters using the general diagnostic model.

Abstract: The present invention provides a method of doing cognitive diagnosis of mental skills, medical and psychiatric diagnosis of diseases and disorders, and in general the diagnosing of latent properties of a set of objects, usually people, for which multiple pieces of binary (dichotomous) information about the objects are available, for example testing examinees using right/wrong scored test questions. Settings where the present invention can be applied but are not limited to include classrooms at all levels, web-based instruction, corporate in-house training, large scale standardized tests, and medical and psychiatric settings. Uses include but are not limited to individual learner feedback, learner remediation, group level educational assessment, and medical and psychiatric treatment.

Abstract: A tool, method, and system for use in the development of sentence-based test items are disclosed. The tool may include a user interface that may include a database selection field, a sentence pattern entry field, an option pane, and an output pane. The tool may search a database for one or more sentences and may generate one or more responses to the one or more sentences. The one or more sentences and one or more responses may be used to produce the sentence-based test items. The tool may allow test items to be developed more quickly and easily than manual test item authoring. Accordingly, test item development costs may be lowered and test security may be enhanced.

Abstract: Systems and methods for parallel conversion, compilation, and distribution of content are disclosed. An application interface may be provided that enables a user to request the conversion of content items into formats well suited for presentation. The request may include a set of deliverables. Each deliverable may include an associated content item and a corresponding associated format to which to convert the associated content item. The request may be processed by a “pipeline” application. The pipeline application calls existing applications or proprietary applications associated with a delivery channel, and the pipeline application may customize the content items for conversion into the corresponding formats. The pipeline application may convert the associated content items using a plurality of parallel processing threads, each thread corresponding to an associated deliverable, whereby each thread converts the associated content item to the corresponding associated format.

Abstract: A method for evaluating at least one cognitive and/or non-cognitive ability of a user is disclosed herein The method comprises receiving performance data from at least one associated activity, transmitting said performance data to an analysis engine, receiving, from the analysis engine and based on the new and pre-existing performance, an evaluation and diagnostic feedback, wherein the evaluation and diagnostic feedback are the product of at least one mathematical model, and wherein the evaluation describes the at least one cognitive and/or non-cognitive ability, and transmitting to the user report text, wherein the report is based on both the evaluation and the diagnostic feedback.

Abstract: A computer-based test creation system and method for generating test items. The test creation process includes creating items in electronic form and allowing access by a plurality of reviewers via a network. The reviewers may review the item for content, fairness and format, and collaborate with each other before indicating that the item is acceptable for inclusion in a test. After the administration of the items to test takers, item performance may be evaluated on-line by reviewing electronic feedback from the administrations and creating electronic evaluation reports. Since the items are created in an electronic form, the items may further include new forms of stimuli such as graphics and multimedia.

Abstract: A method and system for providing immediate diagnostic feedback on speech samples of non-native speakers are disclosed. A scoring model is generated based on speech features extracted from one or more training speech samples. An evaluation speech sample is received and speech features of the evaluation speech sample are determined. Based on the scoring model and the speech features, diagnostic feedback is provided to the speaker. In an alternate embodiment, speech features are extracted from an evaluation speech sample. The speech features are then compared with optimal values, ranges of values, or norms for those features. Based on the result of the comparison, diagnostic feedback is provided to the speaker.

Abstract: A method and system for administering an assessment to a student are disclosed. The expected weight of evidence may be calculated for each of one or more tasks based on a student model pertaining to a student. A task may be selected based on the calculated expected weights of evidence. The selected task may be administered to the student, and evidence may be collected regarding the selected task. The student model pertaining to the student may be updated based on the evidence. A determination of whether additional information is required to assess the student may be made. If additional information is required to assess the student, the above steps may be repeated. Otherwise, a proficiency status may be assigned to the student based on the student model.

Abstract: A method of grading an essay using an automated essay scoring system is provided. The method comprises the steps of deriving a set of predetermined features from the essay, wherein the predetermined feature set comprises one or more features that are independent from the test prompt, scoring the feature set with a scoring equation, wherein a multiple regression analysis with graded essay data produces weights for the scoring equation, generating a raw score for the essay; and processing the raw score for the essay into a score category based on an adaptive cutoff algorithm. Also provided is a method of generating a model in which to grade essays, wherein the data used to generate the model is independent from the test prompt or essay topic.

Abstract: To automatically annotate an essay, a sentence of the essay is identified and a feature associated with the sentence is determined. In addition, a probability of the sentence being a discourse element is determined by mapping the feature to a model. The model having been generated by a machine learning application based on at least one annotated essay. Furthermore, the essay is annotated based on the probability.

Abstract: The present disclosure presents a useful metric for assessing the relative difficulty which non-native speakers face in pronouncing a given utterance and a method and systems for using such a metric in the evaluation and assessment of the utterances of non-native speakers. In an embodiment, the metric may be based on both known sources of difficulty for language learners and a corpus-based measure of cross-language sound differences. The method may be applied to speakers who primarily speak a first language speaking utterances in any non-native second language.

Abstract: To automatically evaluate an essay, the essay is applied to a plurality of trait models and a plurality of trait scores are determined based on the plurality of trait models. Each of these trait scores having been generated from a respective trait model. In addition, a score is determined based on the plurality of trait scores.

Abstract: A method of developing a translation exercise is disclosed. A grammatical structure may be received. Each of one or more text segments in a first language may be translated into a corresponding text segment in a second language. Each text segment in the first language may include one or more sentences. A text segment may be selected as a prompt for a translation exercise. The selected text segment may include a text segment in the second language having the grammatical structure and/or a text segment in the first language corresponding to a text segment in the second language having the grammatical structure.

Abstract: A method and system for determining a lexical association of phrasal terms are described. A corpus having a plurality of words is received, and a plurality of contexts including one or more context words proximate to a word in the corpus is determined. An occurrence count for each context is determined, and a global rank is assigned based on the occurrence count. Similarly, a number of occurrences of a word being used in a context is determined, and a local rank is assigned to the word-context pair based on the number of occurrences. A rank ratio is then determined for each word-context pair. A rank ratio is equal to the global rank divided by the local rank for a word-context pair. A mutual rank ratio is determined by multiplying the rank ratios corresponding to a phrase. The mutual rank ratio is used to identify phrasal terms in the corpus.