Abstract: A portable storage apparatus is configured to interface with a computer having a processor. The portable storage apparatus comprises a wireless transceiver for transmitting and receiving data, an operating system, and a memory containing data representing a plurality of questions. The portable storage apparatus is configured to receive data representative of a question identification number, to communicate a question, from the plurality of questions, corresponding to the question identification number, to receive data representative of a response to the question, and to transmit the data representative of the response.

Abstract: The present subject matter relates to a method for customized question paper generation. The method includes assigning at least one key value to each of a plurality of questions, and storing the questions along with the at least one associated key value in a question bank. The method further includes fetching the stored questions based on rules in a question paper template, where the rules are based on the at least one key value. Furthermore, the method includes validating the fetched questions based on at least one constraint and generating the question paper based on the validating.

Abstract: A system and method assess educational development of a child. Indication of accomplishment by the child of a plurality of tests is entered into the system. A report is generated to indicate educational areas of concern and strength for the child.

Abstract: In order to provide questions suited to skill levels of a learner, it is provided a learning support system which consists of a computer including a processor and a memory, comprising: a question database; a question generator; an answer acquisition module; a scoring module and an achievement degree estimator. The achievement degree estimator compares a number of types of perturbation within the questions to which a correct answer has been given with a predetermined second threshold value. The question generator generates a word changed in accordance with the perturbation of a different type in a case where the number of types of perturbation within the questions to which the correct answer has been given is smaller than the predetermined second threshold value, and sets a question relating to the generated word as a candidate for the question.

Abstract: Systems and methods are provided for improved processing of forms using color. One aspect is a method for processing a plurality of test answer sheets in a standardized test system. In various embodiments, each of the plurality of completed test answer sheets is recorded. The completed test answer sheet is scanned, an identifier for the completed test answer sheet is determined, and three or more colors are electronically recorded to form a multicolor image on a computer readable medium. The multicolor image is representative of the scanned completed test answer sheet. The multicolor image is associated with the identifier in a database. Each of the plurality of completed test answer sheets is scored. Scoring the test answer sheets includes digitally modifying at least one color in the multicolor image at an image use point to assist with scoring the completed test answer sheets. Other aspects are provided herein.

Abstract: A computer-based assessment driven cognition system includes, in embodiments, a generator configured to present questions, where each question presented is associated with a stacking weight and has at least one correct response, and a question property evaluator configured to, for each question presented, determine a likelihood of randomly selecting the at least one correct response. The system also includes a question response receiver configured to receive at least one response to each question presented. Additionally, the system includes a question response evaluator configured to determine, for each question presented, when the at least one response is the at least one associated correct response, a likelihood that the at least one response is a false positive. The question response evaluator is further configured to adjust the stacking weight of each question presented depending at least upon the likelihood that the at least one response to the question is a false positive.

Abstract: A concept rater module is utilized to automatically grade or score constructed responses based on a model answer. The concept rater module may be configured to accept a model answer as input. The model answer may be used as a grading key by the concept rater module. The concept rater module may be further configured to accept student responses in a file format. The file format may be ASCII text, a formatted word processing (e.g., WORDPERFECT, MICROSOFT WORD, etc.) and the like. The concept rater module may be further configured to process a student response into a canonical representation of the student response. The canonical representation of the student response is compared against the model answer by the concept rater module. From the comparison, a score is generated which represents that student's ability to cover all the key concepts.

Abstract: A Consolidated Online Assessment System (COLA System) that creates, manipulates, and distributes an objected-oriented paradigm that represents the scoring and related activities as a unified and integrated family of loosely coupled objects, most notably a “Case” representing a state-machine that replaces the constructed response from the test taker as the unit of work. The Scoring Model and associated Properties that encapsulate the business rules associated with what actions are appropriate or required for a unit of work and the Responses to the unit of work which represent the test-taker-contributed content (e.g., essays or other text-based responses, audio responses, digitized video responses, scanned images, diagrams, lessons plans, etc.) and ties that content to its creator are linked to a Distinct Scorable Unit (DSU) which represents a tree-based mechanism that connects and provides inheritability for the other primary system objects.

Abstract: Implementations feature scannable form techniques and systems. Some aspects feature an apparatus that includes a pen-camera device, an electronic clipboard to store the pen-camera device, and a docking station for the clipboard. The clipboard includes a retention clip to hold a form on a substantially planar surface, and a pen-camera storage compartment to store the pen-camera device. The clipboard includes a power supply interface for charging the pen-camera device, a communication interface to transmit image and location data from the pen-camera device, and electrical interfaces for the clipboard docking station. The pen-camera device can mark response data on the form, optically scan response data and location data on the form, and associate time-stamped information for the marked responses on the form for error checking and fraud detection. The scannable form can include an area for encoded information that identifies a location and arrangement of a position-coding pattern on the form.

Abstract: Methods and apparatus for displaying data on a multi-user display device are disclosed. A first subset of the data is displayed in a first orientation, such that the displayed data is in a proper orientation with respect to a first user located at a first side of the display device. Upon receiving a user input requesting to share the data, a second subset of the data is displayed in a second different orientation, such that the displayed data is in the proper orientation with respect to a second user located at a second different side of the display device. Which subset of the data is displayed to which user is based at least in part on identifying information associated with each user.

Abstract: The method of computerized grading of formula-based multi-step problems includes the steps of providing a memory for storing problems and data to enable and facilitate assessment of correctness of problems, storing at least one formula-based multi-step problem in the memory, a complete solution to the problem requiring calculation of at least one intermediate answer and a subsequent calculation using said intermediate answer to arrive at said complete solution, providing a computer program that can grade submitted answers to the steps of said problem, providing a user interface with selective access to the memory and the computer program, and providing a global computer network with which the user interface can be connected to the memory and grading program.

Abstract: Systems and methods are provided for scoring speech. A speech sample is received, where the speech sample is associated with a script. The speech sample is aligned with the script. An event recognition metric of the speech sample is extracted, and locations of prosodic events are detected in the speech sample based on the event recognition metric. The locations of the detected prosodic events are compared with locations of model prosodic events, where the locations of model prosodic events identify expected locations of prosodic events of a fluent, native speaker speaking the script. A prosodic event metric is calculated based on the comparison, and the speech sample is scored using a scoring model based upon the prosodic event metric.

Abstract: A dynamic content distribution system and method that uses a unique Item Response Function IRF method to provide digital content to end-users based on psychometric reasoning. The system and method gathers psychometric attributes of Registered Users and Registered Subscribers and combines the psychometric attributes with Item Response Function methods, including characterizing content by corresponding psychometric attributes, to determine how and when digital content is presents to the Registered Users.

Abstract: The method and system include presenting a user with a plurality of questions, timing the user's response time for the plurality of questions, identifying an optimal answer time where the user is able to answer a maximum percentage of questions correctly, and informing the user of the optimal answer time. An embodiment of the method may also include providing the user with a suggested change in an actual time taken to answer a question. Certain embodiments may also include weighting the chance of receiving a question in a particular topic based on the user's past performance in answering questions in that topic.

Abstract: A method for creating an educational product, or training experience, is herein described. The training experience includes a learning experience to impart knowledge on training experience participants, a networking experience to allow training experience participants to interact with a relevant peer group during training, a guest experience providing a defined level of interest and focus by participants during the training experience, and an enculturation experience to integrate training experience participants into the culture of the sponsoring organization. The training experience may be created through an assessment phase, a planning phase, a design phase, and an implementation stage, with subsequent product evaluation and revisions. Various milestones and checkpoints may be used to ensure that a level of phenomenality is achieved by the final training experience.

Abstract: A method of capturing student performance can be provided that includes enabling a course creator to generate assessment data related to learning assessments for an online course and to generate learning outcomes for the online course, associating the assessment data with the learning outcomes, such that at least some of the learning assessments correspond to at least some of the learning outcomes, and generating an online classroom. The method may further include providing the learning assessments to students enrolled in the online course using the online classroom, wherein the students can generate completed learning assessments, and determining, for each of the completed learning assessments, whether one or more of the learning outcomes were met by each of the students.

Abstract: Methods and systems of clustering a plurality of students are disclosed. A computing device may receive assessment data for each of a plurality of students. The assessment data includes information pertaining to each of a plurality of questions in an assessment. The computing device may also receive a number of clusters into which to organize the plurality of students. The computing device may determine a similarity value between each pair of students in the plurality of students based on the assessment data associated with each student. The computing device may organize the plurality of students into the number of clusters based on the similarity values.

Abstract: A system and method of communication based on the Reverse Multiple-Choice Method of teaching and testing is disclosed where at least one communicant is a machine. The method is applicable for training a machine for knowledge engineering and artificial intelligence oriented applications, as well as for a trained machine to assist a human being engaged in the activity of teaching or testing.

Abstract: A computerized method comprises creating an answer key for an assessment comprising one or more questions to be delivered to one or more participants, where the answer key comprises assessment information and question information; delivering the assessment to the participants; collecting responses from the participants; and saving question descriptions, any annotations made thereon and the collected responses.

Abstract: A method, apparatus and computer software for automatically analysing a filled in questionnaire, are provided. The method includes automatically analysing a filled in questionnaire, the method comprising the steps of: providing a questionnaire having answer locations; scanning at least two copies of the questionnaire with the answer locations filled in differently and complementarily in a predetermined way, to produce a first scan and a second scan; automatically determining answer locations by comparing the first and second scans; providing a filled in questionnaire having certain answers; scanning the provided filled in questionnaire as a third scan; and automatically judging the certain answers with the determined answer locations.

Abstract: A method, system and processor-readable storage medium for evaluating vocabulary similarity are disclosed. A generic rate may be determined for each word in a plurality of first responses. Each first response may respond to one of a plurality of first prompts. At least one first response may respond to each of the first prompts. A specific rate may be determined for each word in a plurality of second responses, which each respond to a second prompt. A target response may be received that is associated with the second prompt and has a plurality of words. A vocabulary similarity index may be computed for the target response based on one or more generic rates and on or more specific rates. A determination of whether the target response is off-topic may be made based on the vocabulary similarity index for the target response.

Abstract: A computer implemented method, apparatus, and computer usable program code for developing training for a client. Training delivery requirements for a client are identified. Assessment information is gathered. A training curriculum is developed based on the training delivery requirements and the assessment information. An amount of time required to complete the training is estimated to form a delivery estimate. The delivery estimate is translated into an estimate of resources required for development of training materials for the training. Verification is performed to verify that the client receives the training materials for delivery to an end user.

Abstract: A method and system for customizing an automated essay scoring system are disclosed. Relative weights may be assigned to a plurality of features for a plurality of benchmark essay responses. Based on the relative weights, automated scores may be determined for one or more first essay responses to a first prompt. The automated scores may be scaled based on one or more scoring standards for the first prompt. A scaled automated score may be assigned for a second essay response to the first prompt based on the scaled automated scores for the one or more first essay responses.

Abstract: A services-oriented system structure for knowledge assessment and learning comprises a display device for displaying to a learner at a client terminal a plurality of multiple-choice questions and two-dimensional answers, an administration server adapted to administer one or more users of the system, a content management system server adapted to provide an interface for the one or more users to create and maintain a library of learning resources, a learning system server comprising a database of learning materials, wherein the plurality of multiple-choice questions and two-dimensional answers are stored in the database for selected delivery to the client terminal, and a registration and data analytics server adapted to create and maintain registration information about the learners.

Abstract: A system and method of knowledge assessment comprises displaying to a learner a plurality of multiple-choice questions and two-dimensional answers, accessing a database of learning materials, and transmitting to the learner the plurality of multiple-choice questions and two-dimensional answers. The answers includes plurality of full-confidence answers consisting of single-choice answers, a plurality of partial-confidence answers consisting of one or more sets of multiple single-choice answers, and an unsure answer. The method further comprises scoring a confidence-based assessment (CBA) administered to the learner by assigning various knowledge state designations based on the learner's responses to the two-dimensional questions.

Abstract: Techniques for statistically estimating a rating or other “figure of merit” for a user are disclosed. According to one such technique, a first quantity of submissions that were submitted by a user is determined. A second quantity of submissions that (a) were submitted by the user and (b) obtained a particular rating from a rating mechanism also is determined. A user rating for the user is determined based at least in part on the first quantity, the second quantity, and a factor that is independent of both the first quantity and the second quantity—such as the probability that an answer submitted by any answerer in a population will obtain the particular rating from the rating mechanism. The influence that the second quantity has on the user rating relative to the influence that the factor has on the user rating may depend at least in part on the first quantity.

Abstract: Techniques for statistically estimating a rating or other “figure of merit” for a user are disclosed. According to one such technique, a first quantity of submissions that were submitted by a user is determined. A second quantity of submissions that (a) were submitted by the user and (b) obtained a particular rating from a rating mechanism also is determined. A user rating for the user is determined based at least in part on the first quantity, the second quantity, and a factor that is independent of both the first quantity and the second quantity—such as the probability that an answer submitted by any answerer in a population will obtain the particular rating from the rating mechanism. The influence that the second quantity has on the user rating relative to the influence that the factor has on the user rating may depend at least in part on the first quantity.

Abstract: Disclosed is a discussion solution providing system. The discussion solution providing system opens a discussion room with a particular topic, and transmits image information of each discussion participant participating in the opened discussion room to terminals of each discussion participant, so that the transmitted image information may be displayed on discussion UIs corresponding to each discussion participant. The discussion solution providing system progresses discussion in a tournament type, thereby inspiring the participants to have competitive spirits and increasing the educational effect.

Abstract: A computer-based test creation system and method for generating test items. The test creation process includes creating items in electronic form and allowing access by a plurality of reviewers via a network. The reviewers may review the item for content, fairness and format, and collaborate with each other before indicating that the item is acceptable for inclusion in a test. After the administration of the items to test takers, item performance may be evaluated on-line by reviewing electronic feedback from the administrations and creating electronic evaluation reports. Since the items are created in an electronic form, the items may further include new forms of stimuli such as graphics and multimedia.

Abstract: A hardware and software implemented, standardize testing, scoring and reporting system has a network interface, being either LAN or the Internet. A test is selected from a library, assigned to a group of examinees, and a record is made of the assignment. The test is available on-line, or downloaded, or both. Downloaded tests are saved as PDF files and printed into booklets. Examinees record answers on bubble sheets, which are scanned at the customer site or shipped to a processing plant. Individual scanned answer sheets provided data for individual scanned records. Scanning software performs an initial auto-correct and then performs an initial check for errors in scanned records. Tests are batched for later inspection. Scanned records with errors are classified as invalid and the test papers are separated. Prompts are generated for a human test paper inspection and scanned record editing.

Abstract: A method and system for customizing an automated essay scoring system are disclosed. Relative weights may be assigned to a plurality of features for a plurality of benchmark essay responses. Based on the relative weights, automated scores may be determined for one or more first essay responses to a first prompt. The automated scores may be scaled based on one or more scoring standards for the first prompt. A scaled automated score may be assigned for a second essay response to the first prompt based on the scaled automated scores for the one or more first essay responses.

Abstract: A phonics training system provides immediate, audible and virtual answers to questions regarding various images such as objects, animals and people, posed by a user when the user views such images on a video display terminal of the system. The system can provide virtual answers to questions without the need for an instruction or teacher and includes a computer having a video output terminal and an electronic library containing common answers to basic questions. This system can also include an artificial intelligence system.

Abstract: The present invention comprises methods of developing and implementing educational products or materials, including testing products or materials, by utilizing multiple-choice questions. Starting with a given set of facts to be presumed in a question, the method may require one to examine each answer choice, and inquire how a set of facts to be presumed could be modified to make that answer choice the “correct” or “best” answer choice. If a given answer choice is already correct, no modification is needed. If a given answer choice is not correct, various modifications may be employed, including changing some of the words or phrases, or other symbols or objects within the set of facts to be presumed.

Abstract: A system including electronic data collection for school officials, staff, and administrators is described. The data collection system can be integrated onto school management systems to interconnect information regarding data such as student academic records, student merits, student progress, student and teacher biographies, etc. The system assesses student, teacher, classroom, individual school, school district, and state performance in a variety of educational area; associates imputed educational data; and permits the cross-application querying of the educational data and access of the educational data in real-time. The system is intended to help school systems accomplish tasks such as managing work, monitoring student progress in real-time, and determining education trends, via data linkage in real-time.

Abstract: An e-learning system that provides a customized e-learning experience for a user. Information is received from a user identifying the user's learning objectives. Based upon the user's objectives, a study plan that is customized for the user is developed. The study plan may comprise one or more course units that the user can access through a server using a client system used by the user. User activity is monitored, including the user's activity within the e-learning system and outside the e-learning system. Information is recorded regarding the user's progress and performance. This information is used to modify the study plan, as appropriate for the user.

Abstract: Systems, program products, and methods of curriculum planning are provided. An embodiment of a system includes instructional course instructions, responsive to user input commands, to associate one of several instructional courses with one of several teachers, calendar instructions, responsive to user input commands, to create a calendar schedule and assign the instructional course to a date and time for display on the calendar schedule, lesson-plan assignment instructions, responsive to the calendar schedule and user input commands, to assign one of several lesson-plans to the instructional course at the date and time on the calendar schedule, and lesson-plan indicator instructions, responsive to the calendar schedule and lesson-plan data, indicating whether the instructional course for the date and time has been assigned a completed lesson.

Abstract: A system and method for automated assessment of constrained constructed responses provides for automatic, e.g., programmatic, assessing of a test subject response to a constrained constructed response item or question. In one embodiment, paper test materials with test subject markings thereon are converted to electronic form. Test subject markings are further isolated, for example, by registration, sizing and removal of markings other than a graphic response. The subject markings are further compared with a baseline response, correctness deviation criteria and incorrectness deviation criteria (e.g., correctness, incorrectness, substantial correctness, substantial incorrectness, and so on) from which one or more of a score, further learning or other assessment results may be determined. Cluster analysis or other post-evaluation or post-assessment processing may also be conducted, or re-evaluation or re-assessment may also be conducted.

Abstract: Methods and systems for generating and processing an assessment examination answer sheet are disclosed. An answer sheet is formatted by question, column or section using a layout tool and includes calibration marks, and examinee and examination identifiers. The format is stored in a computer-readable form, such as an XML file. A conventional printer prints the answer sheet prior to a test administration. Upon completion, a scanned image of the answer sheet is produced using a conventional scanner. The calibration marks are used to determine relative positions of answer sheet elements and to determine light and dark answer values. The examinee and examination identifiers identify the examinee and the proper stored format. Automatic grading ensues. A graphical user interface (“GUI”) assists grading of the answer sheet. The GUI may use confidence level indicators to denote answer choices or questions for which the grading system cannot precisely determine the intended response.

Abstract: Systems and methods are provided for evaluating an automatic content scoring technology. A first text and a second text are received. A determination by the first automatic content scoring technology on whether at least a portion of the first text entails the second text, does not entail the second text, or refutes the second text is received. A determination by a human rater on whether at least a portion of the first text, entails the second text, does not entail the second text, or refutes the second text, and a reason for the determination by the human rater are received. The determination by the first automatic content scoring technology and the determination by the human rater are compared. A report is output indicating quality of the determination by the first automatic content scoring technology and showing disagreement between the determination by the first automatic content scoring technology and the determination by the human rater.

Abstract: The present invention is directed to an interactive computer tool for presenting and presenting documents directed to decision-making. The computer tool requires users to identify choice options and, for each choice option, relevant consequences, outcomes, and/or goals, for evaluating outcomes therefor.

Abstract: An apparatus and method for monitoring an interaction between a caller and an automated voice response (AVR) system is provided. An audio communication from a caller is processed by executing an AVR script, which includes a plurality of instructions. A visual representation of the audio communication is presented substantially simultaneously with the audio communication to an agent based on the AVR script. The visual representation includes at least one field to be populated with information obtained from the caller and the information populated in the field can be updated by the agent.

Abstract: Described are computer-based methods and apparatuses, including computer program products, for student performance assessment. In some examples, a method includes automatically generating an assessment for a plurality of students based on a selection of assessment information; receiving a plurality of assessment responses from the plurality of students in response to the generated assessment; transmitting requests for at least two preliminary assessment scores for each of the plurality of assessment responses; receiving at least two preliminary assessment scores for each of the plurality of assessment responses; determining if each of the at least two preliminary assessment scores for each of the assessment responses match a criteria; transmitting a request for an additional preliminary assessment score for the assessment response if the at least two preliminary assessment scores match the criteria; and generating a final assessment score for each of the assessment responses.

Abstract: The methodology of the present invention is implemented by a system which defines concepts about which a respondent is required to demonstrate knowledge of a relationship. The concepts are represented by respondent-manipulable devices that are uniquely associated with the concepts, and the respondent is able to manipulate attributes of the devices to create an arrangement of the devices representative of the respondent's understanding of the relationship of the concepts with which the devices are associated. The attributes of the manipulated devices are recorded and converted into a response pattern based on the cognitive relationships of the devices as manipulated by the respondent. One or more predefined rubric patterns are associated with the constructed response item and define possible cognitive patterns of the concepts as represented by the devices. The response pattern is compared to the rubric pattern to identify matches between the response pattern and the rubric pattern.

Abstract: A system, method, and related techniques are disclosed for scoring user responses to constructed response test items. The system includes a scoring engine for receiving a user response to a test question and evaluating the response against a scoring rubric. The scoring rubric may include a binding stage, an assertion stage, and a scoring stage. Furthermore, the system includes a database for referencing elements used by the scoring engine which may comprise objects, object sets, attributes of objects, and transformations of any elements.

Abstract: A system, method, and related techniques are disclosed for scoring user responses to constructed response test items. The system includes a scoring engine for receiving a user response to a test question and evaluating the response against a scoring rubric. The scoring rubric may include a binding stage, an assertion stage, and a scoring stage. Furthermore, the system includes a database for referencing elements used by the scoring engine which may comprise objects, object sets, attributes of objects, and transformations of any elements.

Abstract: Systems and methods are provided for distributing constructed responses to scorers to score while reducing an undesirable statistical metric. A constructed response scoring plan is generated, where the scoring plan includes distributing a plurality of constructed responses to scorers for scoring, and where a scoring effectiveness metric is calculated for the scoring plan. An undesirable statistical aspect is identified that has a negative effect on the scoring effectiveness metric. A distribution rule is generated that will reduce the effect of the undesirable statistical aspect on the scoring effectiveness metric. A constructed response queue is generated for a particular scorer based on the distribution rule, and a next constructed response is provided from the constructed response queue to the particular scorer for scoring.

Abstract: The invention relates to a method for testing the accuracy of answers to quiz questions, wherein a quiz master and a participant use, respectively, an identical quiz computer, wherein an answer code sequence is subsequently compared by means of a solution code sequence when a button on the quiz computer of the participant is actuated, and the correct and incorrect answers of the participant, associated with the question, are displayed on the display either successively or simultaneously as “correct” or “incorrect”.

Abstract: The present invention provides a method of doing cognitive diagnosis of mental skills, medical and psychiatric diagnosis of diseases and disorders, and in general the diagnosing of latent properties of a set of objects, usually people, for which multiple pieces of binary (dichotomous) information about the objects are available, for example testing examinees using right/wrong scored test questions. Settings where the present invention can be applied but are not limited to include classrooms at all levels, web-based instruction, corporate in-house training, large scale standardized tests, and medical and psychiatric settings. Uses include but are not limited to individual learner feedback, learner remediation, group level educational assessment, and medical and psychiatric treatment.

Abstract: A method for facilitating assessment of an answer to a coursework question and moderating the assessment, the method comprising the steps of displaying at least part of a representation of a coursework answer on a display of a computer, displaying a list of learning outcome statements, selecting a learning outcome statement from the list of learning outcome statements and placing the learning outcome statement on the representation of the coursework answer at a location evidencing fulfillment of the learning outcome statement.

Abstract: An apparatus, system, and method for monitoring test question response time from a remote location are provided. More specifically, the present invention provides a mechanism by which tests may be proctored to users from a remote location as a test administration service and providing alert notifications to the users based on the amount of time that has elapsed during the test taking process as well as the amount of time remaining to take the test. With the present invention, test progress data is obtained from the user and forwarded to a proctor workstation. A proctor device may monitor the user's test progress to determine if expected progress on the test is being attained. The testing progress data may be recorded along with test input data from the user's client device for later use. Moreover, the administering of the test may be done by a third party as a test administration service to which a test developer may subscribe.