Abstract: A computer-based test creation system and method for generating test items. The test creation process includes creating items in electronic form and allowing access by a plurality of reviewers via a network. The reviewers may review the item for content, fairness and format, and collaborate with each other before indicating that the item is acceptable for inclusion in a test. After the administration of the items to test takers, item performance may be evaluated on-line by reviewing electronic feedback from the administrations and creating electronic evaluation reports. Since the items are created in an electronic form, the items may further include new forms of stimuli such as graphics and multimedia.

Abstract: A method and system for automatically generating a scoring model for scoring a speech sample are disclosed. One or more training speech samples are received in response to a prompt. One or more speech features are determined for each of the training speech samples. A scoring model is then generated based on the speech features. At least one of the training speech samples may be a high entropy speech sample. An evaluation speech sample is received and a score is assigned to the evaluation speech sample using the scoring model. The evaluation speech sample may be a high entropy speech sample.

Abstract: A computerized method and system for creating test items by generating variants from a test item model, comprising the steps of creating a new test item model by identifying elements of an initial test item or test item model to be variabilized, variabilizing the elements thereby creating test item variables, indicating values the variables can assume, defining the variables, and generating test item variants utilizing a simultaneous constraint solver. The initial test item can be a pre-existing test item or test item model, a newly created test item or even a conceptual template in the mind of the test item creator. The generated test item variants are displayed to the test item creator. The test item creator can store and forward acceptable test item variants for later use as test items. Test item models can be stored for later use in generating new test item variants.

Abstract: A method and system for customizing an automated essay scoring system are disclosed. Relative weights may be assigned to a plurality of features for a plurality of benchmark essay responses. Based on the relative weights, automated scores may be determined for one or more first essay responses to a first prompt. The automated scores may be scaled based on one or more scoring standards for the first prompt. A scaled automated score may be assigned for a second essay response to the first prompt based on the scaled automated scores for the one or more first essay responses.

Abstract: Systems and methods are provided for determining a difficulty level of a text. A determination is made as to a number of cohesive devices present in a text. A further determination is made as to a number of cohesive devices expected in the text. A cohesiveness metric is calculated based on the number of cohesive devices present in the text and the number of cohesive devices expected in the text, where the cohesiveness metric is used to identify a difficulty level of the text.

Abstract: A method and system for determining text coherence in an essay is disclosed. A method of evaluating the coherence of an essay includes receiving an essay having one or more discourse elements and text segments. The one or more discourse elements are annotated either manually or automatically. A text segment vector is generated for each text segment in a discourse element using sparse random indexing vectors. The method or system then identifies one or more essay dimensions and measures the semantic similarity of each text segment based on the essay dimensions. Finally, a coherence level is assigned to the essay based on the measured semantic similarities.

Abstract: Methods and systems for generating and processing an assessment examination answer sheet are disclosed. An answer sheet is formatted by question, column or section using a layout tool and includes calibration marks, and examinee and examination identifiers. The format is stored in a computer-readable form, such as an XML file. A conventional printer prints the answer sheet prior to a test administration. Upon completion, a scanned image of the answer sheet is produced using a conventional scanner. The calibration marks are used to determine relative positions of answer sheet elements and to determine light and dark answer values. The examinee and examination identifiers identify the examinee and the proper stored format. Automatic grading ensues. A graphical user interface (“GUI”) assists grading of the answer sheet. The GUI may use confidence level indicators to denote answer choices or questions for which the grading system cannot precisely determine the intended response.

Abstract: A method and system for performing automatic text analysis is described. A local ranking for one or more contexts with respect to a word and a global ranking for one or more contexts are generated. The rankings are based on the frequency with which the contexts appear in a corpus. A statistic may be generated using the local and global rankings, such as a log ratio rank statistic equal to the logarithm of the global rank divided by local rank, to measure the similarity of contexts with respect to words with which they combine. A source matrix of word to context values is then created. Singular value decomposition is used to create sub-matrices from the source matrix. Vectors from the sub-matrices corresponding to context(s) and/or word(s) are then selected to determine term-term or context-context similarity or term-context correspondence.

Abstract: A tool, method, and system for use in the development of sentence-based test items are disclosed. The tool may include a user interface that may include a database selection field, a sentence pattern entry field, an option pane, and an output pane. The tool may search a database for one or more sentences and may generate one or more responses to the one or more sentences. The one or more sentences and one or more responses may be used to produce the sentence-based test items. The tool may allow test items to be developed more quickly and easily than manual test item authoring. Accordingly, test item development costs may be lowered and test security may be enhanced.

Abstract: A computer memory stores a data structure representing a ternary search tree (TST) representing multiple word n-grams for a corpus of documents. The data structure includes plural records in a first memory, each record representing a node of the TST and comprising plural fields. At least some n-grams have a sequence of units. The plurality of fields includes one for identifying a given unit of the sequence for a given node, one reserved for storing payload information for the given node, and plural child fields reserved for storing information for a first, second and third child nodes of the given node. The child fields store a null value indicating the absence of the child node or an identifier identifying a memory location of the child node. For at least one record, at least one of the child fields stores an identifier identifying a memory location of a memory different than the first memory.

Abstract: A method and system for determining a lexical association of phrasal terms are described. A corpus having a plurality of words is received, and a plurality of contexts including one or more context words proximate to a word in the corpus is determined. An occurrence count for each context is determined, and a global rank is assigned based on the occurrence count. Similarly, a number of occurrences of a word being used in a context is determined, and a local rank is assigned to the word-context pair based on the number of occurrences. A rank ratio is then determined for each word-context pair. A rank ratio is equal to the global rank divided by the local rank for a word-context pair. A mutual rank ratio is determined by multiplying the rank ratios corresponding to a phrase. The mutual rank ratio is used to identify phrasal terms in the corpus.

Abstract: A method and system for constructing a test using a computer system that performs specification matching during the test creation process is disclosed. A test developer determines one or more test item databases from which to select test items. The test item databases are organized based on psychometric and/or content specifications. The developer can examine the textual passages, artwork or statistical information pertaining to a test item before selecting it by clicking on a designation of the test item in a database. The developer can then add the test item to a list of test items for the test. The test development system updates pre-designated psychometric and content specification information as the developer adds each test item to the test. The test developer can use the specification information to determine whether to add to, subtract from, or modify the list of test items selected for the test.

Abstract: Disclosed herein is a method of analyzing large scale survey results comprising obtaining a sparse data set representing a subset of an original data set comprising a plurality of individuals' responses to a plurality of questions, wherein the sparse data set comprises less than ninety percent of the responses in the original data set; analyzing the sparse data set using a general diagnostic model; and obtaining estimated person parameters using the general diagnostic model.

Abstract: The present invention provides a method of doing cognitive diagnosis of mental skills, medical and psychiatric diagnosis of diseases and disorders, and in general the diagnosing of latent properties of a set of objects, usually people, for which multiple pieces of binary (dichotomous) information about the objects are available, for example testing examinees using right/wrong scored test questions. Settings where the present invention can be applied but are not limited to include classrooms at all levels, web-based instruction, corporate in-house training, large scale standardized tests, and medical and psychiatric settings. Uses include but are not limited to individual learner feedback, learner remediation, group level educational assessment, and medical and psychiatric treatment.

Abstract: Systems and methods for parallel conversion, compilation, and distribution of content are disclosed. An application interface may be provided that enables a user to request the conversion of content items into formats well suited for presentation. The request may include a set of deliverables. Each deliverable may include an associated content item and a corresponding associated format to which to convert the associated content item. The request may be processed by a “pipeline” application. The pipeline application calls existing applications or proprietary applications associated with a delivery channel, and the pipeline application may customize the content items for conversion into the corresponding formats. The pipeline application may convert the associated content items using a plurality of parallel processing threads, each thread corresponding to an associated deliverable, whereby each thread converts the associated content item to the corresponding associated format.

Abstract: A tool, method, and system for use in the development of sentence-based test items are disclosed. The tool may include a user interface that may include a database selection field, a sentence pattern entry field, an option pane, and an output pane. The tool may search a database for one or more sentences and may generate one or more responses to the one or more sentences. The one or more sentences and one or more responses may be used to produce the sentence-based test items. The tool may allow test items to be developed more quickly and easily than manual test item authoring. Accordingly, test item development costs may be lowered and test security may be enhanced.

Abstract: A method for evaluating at least one cognitive and/or non-cognitive ability of a user is disclosed herein The method comprises receiving performance data from at least one associated activity, transmitting said performance data to an analysis engine, receiving, from the analysis engine and based on the new and pre-existing performance, an evaluation and diagnostic feedback, wherein the evaluation and diagnostic feedback are the product of at least one mathematical model, and wherein the evaluation describes the at least one cognitive and/or non-cognitive ability, and transmitting to the user report text, wherein the report is based on both the evaluation and the diagnostic feedback.

Abstract: A computer-based test creation system and method for generating test items. The test creation process includes creating items in electronic form and allowing access by a plurality of reviewers via a network. The reviewers may review the item for content, fairness and format, and collaborate with each other before indicating that the item is acceptable for inclusion in a test. After the administration of the items to test takers, item performance may be evaluated on-line by reviewing electronic feedback from the administrations and creating electronic evaluation reports. Since the items are created in an electronic form, the items may further include new forms of stimuli such as graphics and multimedia.

Abstract: A method and system for providing immediate diagnostic feedback on speech samples of non-native speakers are disclosed. A scoring model is generated based on speech features extracted from one or more training speech samples. An evaluation speech sample is received and speech features of the evaluation speech sample are determined. Based on the scoring model and the speech features, diagnostic feedback is provided to the speaker. In an alternate embodiment, speech features are extracted from an evaluation speech sample. The speech features are then compared with optimal values, ranges of values, or norms for those features. Based on the result of the comparison, diagnostic feedback is provided to the speaker.

Abstract: A method of grading an essay using an automated essay scoring system is provided. The method comprises the steps of deriving a set of predetermined features from the essay, wherein the predetermined feature set comprises one or more features that are independent from the test prompt, scoring the feature set with a scoring equation, wherein a multiple regression analysis with graded essay data produces weights for the scoring equation, generating a raw score for the essay; and processing the raw score for the essay into a score category based on an adaptive cutoff algorithm. Also provided is a method of generating a model in which to grade essays, wherein the data used to generate the model is independent from the test prompt or essay topic.