Abstract: A process for using a computer to assess problem-solving skill includes a computer program that grades multi-step problems and is made aware of several different but equally valid formulas or methods that use answers to one or more prior steps to arrive at a correct answer to any problem step for which such choices exist. The user is allowed to submit answers, repeatedly if necessary, to any or all of a problem's steps, in any desired order, until all steps are either graded as correct or the correct answer was requested and given. A user's problem-solving skill level is assessed by compiling how many times answers to each step were submitted and on which Tries; if graded as correct, the Try on which credit was given and whether credit was given for a submitted answer, a submitted formula, or an unsubmitted answer for which prior-answer credit was given; if the correct answer was requested, the Try for which the request was made; and if hints are provided, a summary of their use.

Abstract: The current invention is designed to improve the certainty of test-taker classifications in testing programs utilizing open-ended items. The current invention discloses methods to improve certainty in estimates of test-taker performance for all test-takers while using a minimum of scoring resources. The invention selectively allocates resources to scoring by ceasing to allocate scoring resources to test-takers whose performance has been determined with sufficient certainty, and by having an additional reader or readers score responses from test-takers whose scores are close to cutoff levels. The invention further increases precision of scores for test-takers near a cutoff by selectively allocating more reliable and accurate readers to score responses most likely to influence a test-taker's performance classification. Other scoring resources are also shown.

Abstract: A natural interactions academic course designer system and method that allows an educator to integrate a natural interactions forum (such as discussion boards, blogs, and collaborative wikis) into a formal curriculum. Embodiments of the system and method include a natural interactions integration module, which allows an educator to select a type of natural interaction forum and attaches a rubric, and a natural interactions provisioning module that creates the forum, determines the participants, and the learners to be assessed. Embodiments of the system and method also include an assessment module that assesses a validated learner using one or more of three different metrics. These metrics include a participation metric, a tagging metric, and a natural language processing metric. The assessment module then creates a raw score from the metrics used. Embodiments of the system and method also include a learner tracking module that generates a single final score for the assignment.

Abstract: A system and method for generating a Flash-based educational training course, wherein course content may be received from a user generating the training course. The course content may be parsed into segments. An XML file may be generated from the parsed course content. An XML instance may be created. A function that includes actions to be performed by the XML instance, once the XML file has been read, may be initialized. The function may be assigned to the onLoad property of the XML instance, causing the function to be performed upon the completion of the data loading. The XML load may be started using a Flash module. The course content may be presented using the Flash module.

Abstract: A human test that is based on a human conceptual capability. A human test according to the present techniques includes posing a question that is selected to exercise a human conceptual capability, obtaining an answer to the question, and comparing the answer to a correct answer that would be rendered by a human being.

Abstract: An improved system for administering a computer and paper-based test, including a multifunction peripheral having a printing function and a scanning function for operating on the test, and a host remotely connected to the peripheral over a network. The multifunction peripheral is configured to receive the test from the host and transmit a scanned copy of a modified paper test to the host, and the host is configured to transmit a result of an evaluation of the modified test to the peripheral. A method for administering a test including transmitting a registration code, receiving a print job including a test associated with the registration code and printing the test. The method further includes digitizing the test and transmitting the digitized test.

Abstract: A method of grading an essay using an automated essay scoring system is provided. The method comprises the steps of deriving a set of predetermined features from the essay, wherein the predetermined feature set comprises one or more features that are independent from the test prompt, scoring the feature set with a scoring equation, wherein a multiple regression analysis with graded essay data produces weights for the scoring equation, generating a raw score for the essay; and processing the raw score for the essay into a score category based on an adaptive cutoff algorithm. Also provided is a method of generating a model in which to grade essays, wherein the data used to generate the model is independent from the test prompt or essay topic.

Abstract: To automatically annotate an essay, a sentence of the essay is identified and a feature associated with the sentence is determined. In addition, a probability of the sentence being a discourse element is determined by mapping the feature to a model. The model having been generated by a machine learning application based on at least one annotated essay. Furthermore, the essay is annotated based on the probability.

Abstract: The present disclosure presents a useful metric for assessing the relative difficulty which non-native speakers face in pronouncing a given utterance and a method and systems for using such a metric in the evaluation and assessment of the utterances of non-native speakers. In an embodiment, the metric may be based on both known sources of difficulty for language learners and a corpus-based measure of cross-language sound differences. The method may be applied to speakers who primarily speak a first language speaking utterances in any non-native second language.

Abstract: To automatically evaluate an essay, the essay is applied to a plurality of trait models and a plurality of trait scores are determined based on the plurality of trait models. Each of these trait scores having been generated from a respective trait model. In addition, a score is determined based on the plurality of trait scores.

Abstract: A material processing apparatus has a reader unit, an information extracting unit and a grade determination unit. The reader unit reads a material. The information extracting unit extracts information from the material, the information having information written by a user and information printed on the material in advance. The grade determination unit grades the information written by the user based on the information printed on the material in advance.

Abstract: A writing instrument for use by a grader comprises an elongate, tubular housing, a writing component secured within the housing and operative to make marks on a surface, and a calculation assembly comprised of electronic components. The calculation assembly is retained within the housing and includes a plurality of data entry devices for entering data used to calculate a grade, the data entry devices being accessible by the grader through an exterior surface of the housing; a processing device programmed to receive data from the plurality of data entry devices and calculate the grade based on the entered data by performing division; and a display component for displaying to the grader the grade calculated by the processing device. A method of grading comprises using the writing instrument.

Abstract: A question item comprising: textual, visual or auditory subject matter; a query or instruction pertaining to the subject matter; and a representation of a response mechanism for receiving a user response to the query or instruction, may be retrieved from a data store. The subject matter, query or instruction and response mechanism may be presented in a presentation sequence, wherein a presentation duration of each of the subject matter, query or instruction and response mechanism is controlled by user input. Based on the user input, a presentation duration of each of the subject matter, query or instruction, and response mechanism may be determined. Each of the three presentation durations may be stored. A completion time may be determined based on one or more of the presentation durations and may be stored along with response accuracy, for use in calculating a user competency measure for the question item.

Abstract: An essay is analyzed automatically by accepting the essay and determining whether each of a predetermined set of features is present or absent in each sentence of the essay. For each sentence in the essay a probability that the sentence is a member of a certain discourse element category is calculated. The probability is based on the determinations of whether each feature in the set of features is present or absent. Furthermore, based on the calculated probabilities, a sentence is chosen as the choice for the discourse element category.

Abstract: A system, method, and related techniques are disclosed for scoring user responses to constructed response test items. The system includes a scoring engine for receiving a user response to a test question and evaluating the response against a scoring rubric. The scoring rubric may include a binding stage, an assertion stage, and a scoring stage. Furthermore, the system includes a database for referencing elements used by the scoring engine which may comprise objects, object sets, attributes of objects, and transformations of any elements.

Abstract: Self-scoring and self-analyzing test answer sheet which facilitates grading and provides the grader with data concerning the nature of each question and the skill being tested. A skills summary area is provided for recording the total test score and also the number of incorrect responses for each of the specific skills tested.

Abstract: Methods and systems for detecting off-topic essays are described that do not require training using human-scored essays. The methods can detect different types of off-topic essays, such as unexpected topic essays and bad faith essays. Unexpected topic essays are essays that address an incorrect topic. Bad faith essays address no topic. The methods can use content vector analysis to determine the similarity between the essay and one or more prompts. If the essay prompt with which an essay is associated is among the most similar to the essay, the essay is on-topic. Otherwise, the essay is considered to be an unexpected topic essay. Similarly, if the essay is sufficiently dissimilar to all essay prompts, the essay is considered to be a bad faith essay.

Abstract: A method and system for automatically helping a teacher/educator evaluate assessments administered to students for determining student's attributes. The teacher/educator reviews stored assessment forms at a digital user interface (DUI) at a multifunction device (MFD) and selects the desired forms and creates an Assessment Batch which includes a List of Students to be given the forms for marking. The system automatically codes each form with personalized student information and prints the individualized assessment forms. The system may include an assessment repository for storing assessment definitions, rubrics, and administered assessments (e.g., results sheets), an may additionally or alternatively include an assessment analyzer for interpreting scanned imaged of administered assessments.

Abstract: A method and system for automatically helping a teacher/educator evaluate assessments administered to students for determining student's attributes. The teacher/educator reviews stored assessment forms at a digital user interface (DUI) at a multifunction device (MFD) and selects the desired forms and creates an Assessment Batch which includes a List of Students to be given the forms for marking. The system automatically codes each form with personalized student information and prints the individualized assessment forms. The teacher/educator administers the assessment and assessment forms are manually marked, collected and scanned at the MFD, entered into storage and the marked images automatically analyzed and the assessments automatically evaluated from stored rubrics and the teacher/educator is automatically notified by email that the evaluation has been performed.

Abstract: A computer-based test creation system and method for generating test items. The test creation process includes creating items in electronic form and allowing access by a plurality of reviewers via a network. The reviewers may review the item for content, fairness and format, and collaborate with each other before indicating that the item is acceptable for inclusion in a test. After the administration of the items to test takers, item performance may be evaluated on-line by reviewing electronic feedback from the administrations and creating electronic evaluation reports. Since the items are created in an electronic form, the items may further include new forms of stimuli such as graphics and multimedia.

Abstract: A method for mapping locations of a plurality of denotable areas on an image, the method performed in a computer having a memory and a processor. According to one aspect of the present invention, the method comprises the steps of receiving data corresponding to the image containing the plurality of denotable areas; displaying the image containing the plurality of denotable areas; retrieving information corresponding to the plurality of denotable areas; prompting a user to identify a particular one of the plurality of denotable areas using the retrieved information; receiving input from the user corresponding to the location of the particular one of the plurality of denotable areas; and repeating the steps of prompting the user and receiving input from the user for one or more of the remaining denotable areas.

Abstract: A method for processing an assessment document. In one aspect, the method may include generating the assessment document having layout information, a test area, and an identifier corresponding to a student, receiving a scanned image of the assessment document after the assessment document has been administered to the student, identifying the test area in the scanned image using the layout information, identifying the student using the identifier in the scanned image, and displaying the test area in response to a request to display the test area for the student.

Abstract: The invented apparatus, system and method for examining a student's oral answer capability includes a presentation mechanism for presenting a program test including one or more questions and soliciting one or more answers as oral inputs; a voice recognition mechanism for receiving a voice stream from a student oral recitation and for converting the voice stream to text-based essay form and for recording the same as a first given answer; answer key data stored in memory, the answer key data representing an ideal answer; a correlation mechanism coupled with the first given answer and with the stored answer key data for comparing the given answer to the stored answer key; and a grading mechanism coupled with the correlation mechanism for assigning a point score representing the merit of the given answer relative to the stored answer key data, wherein the point score represents the level of correlation therebetween.

Abstract: A method for adaptive scoring of responses to constructed response questions is disclosed. Adaptive scoring may be used to apply evaluator time in such a way that a predetermined reliability level is reached with the least possible use of evaluator time, including adjusting the number of response graded and/or the number of evaluators grading each response. A score may be calculated after grading a subset of a test taker's responses to the constructed response questions. A probability or an error estimate is calculated and compared to a threshold value. Grading may be discontinued based on the comparison. A score may be calculated based on a predetermined number of ratings for the test taker's response to a constructed response. A probability that the score is within a predetermined range of what the score would be if all the responses are graded is calculated. If the probability is less than a threshold value, the number of ratings is increased.

Abstract: A pointing information extraction unit extracts pointing information indicating a pointing position and a pointing time on a slide from a slide file used in a lecture and a video file of a lecture video using a pointing device. A word information generation unit analyzes a text sentence extracted from the slide file to generate a word information file indicating a word and a position thereof. A word pointing information generation unit estimates a word closest to the pointing position on the slide to generate a word pointing information file with the pointing time assigned. A fill-in-the-blank word extraction unit extracts a word having a pointing time equal to or longer than a predetermined time from the word pointing information as a fill-in-the-blank word file. A fill-in-the-blank test question is generated by setting the fill-in-the-blank word of the slide information as a blank region.

Abstract: A method for implementing a course hub can include generating a three dimensional activity grading object. Furthermore, an assessable learner process journal can be generated as part of implementing the course hub. In addition, the implementing of the course hub can include generating a plurality of communication zones. Moreover, a “what if” assignment grade modeling can be generated as part of implementing the course hub.

Abstract: Embodiments of the present invention disclose a method for adapting a lesson. The method comprises for a given learner, forming an expectation of the learner's performance in answering questions of a lesson; adapting the lesson a first time based on the expectation; evaluating the learner's actual performance in answering questions of the adapted lesson; and selectively adapting the lesson a second time if a difference between the expectation and the actual performance is greater than a threshold.

Abstract: A computerized system for scoring constructed responses to one or more prompts. The system receives a plurality of constructed responses in an electronic font-based format and separates the plurality of constructed responses into a first group of constructed responses that are scorable by the system and a second group of constructed responses that are not scorable by the system. The constructed responses in the first group are assigned scores based on predetermined rules, and the scores are sent to a score database. In a preferred embodiment, the first group includes constructed responses that do not answer the prompt and constructed responses that match pre-scored responses. The second group of constructed responses are sent by the system to a hand-scorer for manual scoring.

Abstract: A scenario based training or assessment method and/or system including user consequences. The method being performed on a processing system and including providing the user with scenario information and providing the user with a question and a plurality of answers. The user provides a selected answer to be received by the processing system, the selected answer being selected from the plurality of answers by the user. The user is then provided with a consequence based on the selected answer. If the user selects an incorrect answer to the question in a particular scenario, a sequence of multiple scenarios loops back and returns to the particular scenario at a later point in time.

Abstract: Described is a method using a computer for grading student work on a problem when the student's own steps are shown in detail. A reference trace is created representing a best solution path to the problem.

Abstract: Techniques for real-time observation assessment are provided. The techniques, which are designed for educators, take advantage of handheld computers, desktop/laptop computers and Internet access in order to reduce the paperwork associated with conventional educational assessments. An array of instructional assessment applications are designed to run on handheld computers. The instructional assessment applications may be based on existing and widely used paper methodologies. A common Web-based platform for assessment application distribution, selection, download, data management and reporting is also provided. Users can then periodically synchronize instructional data (assessments, diagnostic results, notes and/or schedules) to the Web site. At the Web site, browser-based reports and analysis can be viewed, administered and shared via electronic mail.

Abstract: The present invention is directed to a method of operating a knowledge capture program. The knowledge capture program has the steps of starting the knowledge capture program wherein a user can access content that is either existing content or creating new content. Once the content has been selected then a source subject matter can be selected and displayed, and then captured and incorporated into the content. The source subject matter can be edited and saved into the content. The content can then be retrieved and played in a desired mode of learning.

Abstract: An optical mark recognition tabulation device includes a guide channel including opposing integral sidewalls configured to laterally constrain a test sheet as the test sheet is fed past a set of optical sensors. The optical sensors are aligned with optical entry indicia on the test sheet to sense marks on the optical entry indicia as the test sheet is fed past the optical sensors. The optical mark recognition tabulation device further includes a recessed channel located along at least a substantial portion of a side of the guide channel. The optical mark recognition tabulation device further includes a spacer strip configured to be received within the recessed channel and effectively reduce the width of the guide channel such that the guide channel is reconfigured to laterally constrain a different size of test sheet when the spacer strip is received within the recessed channel.

Abstract: The invention generally concerns a method of generating a real number score for a response, such as a written essay response. The method comprises providing a scoring model having one or more concepts; determining for each concept a probability that the concept is present in the response; creating a scoring rule or scoring rule function; determining an expected value function for the scoring rule; and generating a real number score for the response based on the scoring rule, the expected value function, and the probabilities that the concepts are present in the response (or a combination thereof). The real number score for the response may then be displayed or output, for instance, where the method is implemented as a computer system or application. Concept-based scoring provides improved scoring accuracy and individualized feedback for students and reports for teachers and parents.

Abstract: Methods and apparatuses are disclosed for computer-based evaluation of a test-taker's performance with respect to a reward threshold. A performance assessment of the test taker can be standardized or customized, as well as relative or absolute. The system allows a parentally-controlled reward system such that children who reach specified objectives can claim an award that parents are confident was fairly and honestly earned without the parent being required to proctor the testing.

Abstract: One or more response bubbles are used on a scannable form to be read by a scanning system. An unambiguous symbology of marks is defined to fill into each response bubble, thereby allowing at least one of a plurality of different non-empty intended responses having different output values to be entered into each response bubble. One or more response bubbles are provided on the scannable form to be filled in according to the defined symbology of unambiguous marks. A filled in form is scanned with the scanning system and an image of the scanned form is processed by identifying pixels associated with each response bubble, interpreting the pixels associated with each response bubble to define a raw score according to the defined unambiguous symbology of marks used to mark each response bubble, translating the raw score for each response bubble into one of the plurality of different intended non-empty responses, and outputting the value of the non-empty response for each response bubble.

Abstract: MCMEd is an institutional level Business Method claim for a new design of the structure and processes of learning to facilitate mass production/customization of High School and Higher Education. The present education system operates in shop-mode, in which the single teacher is in charge of the complete cycle of education. No technology so far, including on-line and distant education has changed the shop-mode of the learning process. MCMEd is a large-scale distributed business method that offers education in a customizable hybrid mode: online and in class in flexible combinations that maximize student's actual learning outcomes. In MCMEd design, MCMEd Central consolidates, coordinates, and computerizes the repetitive and duplicated activities, saving magnitudes, while producing higher and consistent quality content, exams, and grading. MCMEd further reduces duplication by adapting a modularized-structure of knowledge.

Abstract: A teaching material selection method, a computer program and a teaching material selection device for selecting the set of most suitable teaching materials on the basis of learning assessment data on an examinee. Provided is a teaching material selection method allowing a computer to execute the steps of: calculating achievement of an examinee in an academic skill area on the basis of learning assessment data of the examinee and a difficulty level of the academic skill area; calculating importance of the academic skill area on the basis of the academic skill area interrelationship structure indicating an interrelationship between the achievement and the academic skill area; calculating a value of a teaching material on the basis of the importance of the academic skill area; and performing calculation for selecting a set of teaching materials on the basis of the value of the teaching materials.

Abstract: A system and method for assessing a student's performance in a course in an online course environment, the method comprising creating rubrics that correlate with course materials for an online course, the rubrics including a scoring schema which is stored in an educational management system, distributing the rubrics from the educational system to a user's computer through a network connection, displaying the rubrics on the user's computer together with their correlating course materials, recording the outcome of a student's performance on the rubrics, the outcome including a grade associated with a student, and transferring the recorded outcome to the educational management system.

Abstract: A device and method for evaluating learning including a level examining module, a course planning module and a course evaluation module are provided. The level examining module provides a level examination for a user to determine a current capability level of the user. The course planning module plans a learning course for the use according to the current capability level as determined by the level examining module. The course evaluation module generates an evaluation quiz according to the learning course. After the user finishes the evaluation quiz, the course evaluation module also generates learning information to evaluate the user's learning status.

Abstract: A method and apparatus comprising at least one reward retainer for retaining an article, and a controller for presenting at least one query, for receiving at least one answer to the at least one query, and for providing access to the reward retainer in response to the correct answer to the at least one query.

Abstract: A system and method enable processing and displaying of test results in accordance with information and specifications provided by the client requiring such services. Skills examined by a test are identified, and skill-by-skill analysis of test results is provided. Performance within constituent skill categories are compared to applicable standards, thereby providing criterion-referenced conclusions about a student's performance within each skill category tested. Evaluation and recommendation statements are generated for students or groups of students based on their test performance and procedures for automatically generating such statements are provided. Test results are also used to identify skills in which individual students and groups have the greatest potential for growth. Test reports are produced in print and electronically using the same electronic document structures and data source files to ensure consistency between the two display mechanisms.

Abstract: The present invention provides a method of doing cognitive diagnosis of mental skills, medical and psychiatric diagnosis of diseases and disorders, and in general the diagnosing of latent properties of a set of objects, usually people, for which multiple pieces of binary (dichotomous) information about the objects are available, for example testing examinees using right/wrong scored test questions. Settings where the present invention can be applied but are not limited to include classrooms at all levels, web-based instruction, corporate in-house training, large scale standardized tests, and medical and psychiatric settings. Uses include but are not limited to individual learner feedback, learner remediation, group level educational assessment, and medical and psychiatric treatment.

Abstract: A method and system for monitoring the movement of academic test materials and/or their shipping containers while providing a complete chain of custody of those test materials and shipping containers. Test materials include paper based and electronic based tests and related items such as answer keys, teacher notes, and study guides. The providers of the test materials include, but may not be limited to, test publishers, state and federal agencies, specialized test developers, and distributors of tests created by other parties. Users include, but are not limited to, school districts, individual schools, students, corporations, and other individuals, as well as the test publisher, state and federal agencies, and other test producers. The tracking is done through the scanning of bar codes or RFID devices printed on or otherwise affixed to test materials and the test material shipping containers.

Abstract: The “Naughty or Nice Meter” is a visual novelty toy designed to let Children know where they stand on a “Naughty or Nice Meter”. Originally conceived for Christmas, the “Naughty or Nice Meter” has widespread applications and can be used for other seasons of the year or events. The device has twelve (12) behavioral questions children will be graded on. The grading system for each question is a point scale from zero thru five. Zero (0) being the worst (Naughty) score and five (5) being the best (Nice) score. A calculator is used to add/subtract the total number of points accumulated for all twelve (12) questions. The meter is categorized in five number ranges (0-35; 36-41; 42-47; 48-53; 54-60). Each range signifies a different percentage grade of naughty to nice. The “Meter” lights up/down to the numbered range displayed according to the total number accumulated by the calculator.

Abstract: The present invention provides a method of doing cognitive diagnosis of mental skills, medical and psychiatric diagnosis of diseases and disorders, and in general the diagnosing of latent properties of a set of objects, usually people, for which multiple pieces of binary (dichotomous) information about the objects are available, for example testing examinees using right/wrong scored test questions. Settings where the present invention can be applied but are not limited to include classrooms at all levels, web-based instruction, corporate in-house training, large scale standardized tests, and medical and psychiatric settings. Uses include but are not limited to individual learner feedback, learner remediation, group level educational assessment, and medical and psychiatric treatment.

Abstract: A computerized method and/or system for training over a network is provided, the method and/or system comprising: sending an item, over the network for evaluation related to a trainee; requiring an evaluator to evaluate the item in an evaluation, the evaluation including information on why the evaluation was given; receiving the evaluation over the network; generating a report comparing the evaluation to other evaluations performed by other evaluators, the report including the information on why each evaluation was given; determining needed change(s) based on the report; determining how to achieve the needed change; and providing materials over the network on how to achieve the needed change to the evaluator.

Abstract: The invention relates to methods for capturing, grading, and recording student workflow for both computerized and paper-based input. The workflow is analyzed and graded against a key irrespective of whether the workflow was captured on paper or through the use of computer input. A method of capturing workflow of a solution of a task having an answer by providing a task having a solution to a user or student is provided. The solution is collected and stored to an electronic storage device by extracting the input solution from either a computer interface or a worksheet. The solution is then posted to a user-account.

Abstract: Methods and systems for evaluating the fit of raw data to model data are disclosed. Relationships between an assessment item and a tested attribute, responses from examinees to an assessment item, mastery states for an examinee for a tested attribute, one or more parameters based on expected assessment item performance, estimates for non-tested attributes for each examinee, likelihoods that an examinee that has mastered attributes pertaining to an assessment item will answer the item correctly, likelihoods that an examinee that has not mastered an attribute for an assessment item will answer the item correctly, and/or other variables may be received. For each item and/or for each examinee, a determination of a class for the examinee and/or item may be determined. Statistics may also be generated for each examinee, each item, an examination and/or any other basis.

Abstract: A computer system configured to provide multi-day training to users. The computer system includes a program storage device. The program storage device includes a database that stores a number-of training-days value and sequence data. The program storage device also includes computer readable instructions, that when executed by the computer system. utilize the number-of-training-days value and the sequence data to control the sequence of training events.