Abstract: Systems and methods may involve processing of entity data by machine learning models to produce one or more entity and/aggregate risk scores and/or aggregate anticipated risk scores, which may be compared to one or more thresholds to determine when one or more predefined actions should be taken. The entity data may be collected for various entities related to an exam registration and delivery process, which may include a candidate, an exam, a test center, an exam registration event, a proctor, and an exam delivery event. The exam registration and delivery process may include multiple states—each being associated with a different set of entities. Aggregate risk scores for a given state may be calculated using only entity data for the set of entities associated with that state. The predetermined actions taken may also be dependent on the current state.

Abstract: Systems and methods may involve processing of entity data by machine learning models to produce one or more entity and/aggregate risk scores and/or aggregate anticipated risk scores, which may be compared to one or more thresholds to determine when one or more predefined actions should be taken. The entity data may be collected for various entities related to an exam registration and delivery process, which may include a candidate, an exam, a test center, an exam registration event, a proctor, and an exam delivery event. The exam registration and delivery process may include multiple states—each being associated with a different set of entities. Aggregate risk scores for a given state may be calculated using only entity data for the set of entities associated with that state. The predetermined actions taken may also be dependent on the current state.

Abstract: Systems and methods may involve processing of entity data by nested machine learning models to produce one or more aggregate risk scores, which may be compared to one or more thresholds to determine when one or more predefined actions should be taken. The entity data may be collected for various entities related to an exam registration and delivery process, which may include a candidate, an exam, a test center, an exam registration event, a proctor, and an exam delivery event. Entity data for each entity may be separately processed by entity-specific machine learning models to generate intermediate entity risk scores. The intermediate entity risk scores may be input to an aggregate machine learning model, which may output an aggregate risk score. A resource management server may cause the predefined actions to be taken after comparing the aggregate risk score to the one or more thresholds.

Abstract: Systems and methods may involve processing of entity data by machine learning models to produce one or more aggregate risk scores, which may be compared to one or more thresholds to determine when one or more predefined actions should be taken. The entity data may be collected for various entities related to an exam registration and delivery process, which may include a candidate, an exam, a test center, an exam registration event, a proctor, and an exam delivery event. The exam registration and delivery process may include multiple states—each being associated with a different set of entities. Aggregate risk scores for a given state may be calculated using only entity data for the set of entities associated with that state. The predetermined actions taken may also be dependent on the current state.

Abstract: Methods are disclosed for distributing digitized electronic representations of representative responses to non-objective assessment tasks and established scores for the responses to readers in order to train the readers to determine scores for responses in accordance with a scoring criteria, to qualify the readers to determine scores for responses in accordance with the scoring criteria, and to monitor how well a reader is determining scores for such responses in accordance with the scoring criteria. In disclosed embodiments, the methods provide such distribution for multiple scorers on a network located at a single site, for multiple scorers on a network remote from a central site; and scoring by individual scorers remote from the central site. Distribution may be accomplished by transferring data from electronic files using diverse techniques such as magnetic storage discs, CD-ROM, cable or high-speed public communications network, in accordance with the needs of the particular job and setting.

Abstract: A method and system for on-line resolving is described that dynamically provides resolvers with rules for evaluating answers to particular types of data items. Rules are stored which relate to procedures for evaluating the data items. When data items are displayed to a resolver for evaluating, the corresponding rules are displayed for evaluating the displayed data item.

Abstract: A method for electronically displaying a test item in a consistent manner for a plurality of workstation configurations. Test content is stored in a memory system and includes at least a question portion and a response portion. A portion of the test content is captured in a test item image. The test item image is overlaid with a response control. Together, the test item image and response control define a test item. The test item is presented to a respondent at a first workstation. An electronic response may be received from the respondent. The test item may appear in a consistent manner when presented on a plurality of workstation configurations, for example, on display devices with different screen resolutions.

Abstract: A method for increasing the speed at which data items are processed. The method receives electronic representation of data items and organizes the data items into separate groupings. The method next displays a particular grouping of data items to a resolver so that the resolver can evaluate the data items. In displaying the data items, the method uses a matrix of cells and displays one data item in each of the cells. The resolver can thus simultaneously view multiple data items, which facilitates efficient evaluations of the data items.

Abstract: A system and method for electronically archiving electronic images of responses received from a response provider in such a manner that reduces the amount of electronic resources required for said archival.

Abstract: A method and system for processing groupings of data items. Data items are received in electronic form, after which the data items are divided according to predefined categories and organized into separate groupings. The data items in the groupings are reported.

Abstract: A method and system for organizing and tracking the workflow of responses provided by a response provider. The responses are electronically imaged. The electronic images are tracked using individual identifiers based in part on the type of optical imaging device used. The invention further tracks and reports a group or batch of the electronic images.

Abstract: A method and system for on-line resolving is described that dynamically provides resolvers with rules for evaluating answers to particular types of data items. Rules are stored which relate to procedures for evaluating the data items. When data items are displayed to a resolver for evaluating, the corresponding rules are displayed for evaluating the displayed data item.

Abstract: A method and system for managing evaluators of data items to maintain consistent and efficient scoring of data items. Collaborative scoring allows two evaluators to score the same data item and resolves potential discrepancies in the scores provided by the evaluators. Quality items monitor a evaluator's scoring performance against a known standard. A quality-control data item, having a known score, is transmitted to a evaluator. After receiving a score from the evaluator, the known score is compared with the evaluator's score and a quality profile is generated for the evaluator.

Abstract: A system for managing resolvers of test answers to maintain consistent and efficient scoring of test answers. Collaborative scoring monitors and resolves discrepancies in two or more scores for the same test answers. Quality items monitor a test resolver's scoring performance against a known standard. Performance feedback data tracks a resolver's scoring rate to monitor efficiency and optionally recommend break times or provide the resolver with diversionary activities during breaks. Consistent scoring is further promoted by selectively matching resolvers with test answers based on classifications of the resolvers and persons who generated the test answers. An on-line scoring guide dynamically provides test resolvers with rules for scoring answers to particular test questions.

Abstract: A system for managing resolvers of test answers to maintain consistent and efficient scoring of test answers. Collaborative scoring monitors and resolves discrepancies in two or more scores for the same test answers. Quality items monitor a test resolver's scoring performance against a known standard. Performance feedback data tracks a resolver's scoring rate to monitor efficiency and optionally recommend break times or provide the resolver with diversionary activities during breaks. Consistent scoring is further promoted by selectively matching resolvers with test answers based on classifications of the resolvers and persons who generated the test answers. An on-line scoring guide dynamically provides test resolvers with rules for scoring answers to particular test questions.

Abstract: A system and method for determining which of a plurality of pre-data-collection software applications to apply to optically imaged responses received from a response provider.

Abstract: A system for managing resolvers of test answers to maintain consistent and efficient scoring of test answers. Collaborative scoring monitors and resolves discrepancies in two or more scores for the same test answers. Quality items monitor a test resolver's scoring performance against a known standard. Performance feedback data tracks a resolver's scoring rate to monitor efficiency and optionally recommend break times or provide the resolver with diversionary activities during breaks. Consistent scoring is further promoted by selectively matching resolvers with test answers based on classifications of the resolvers and persons who generated the test answers. An on-line scoring guide dynamically provides test resolvers with rules for scoring answers to particular test questions.

Abstract: Methods are disclosed for distributing digitized electronic representations of representative responses to non-objective assessment tasks and established scores for the responses to readers in order to train the readers to determine scores for responses in accordance with a scoring criteria, to qualify the readers to determine scores for responses in accordance with the scoring criteria, and to monitor how well a reader is determining scores for such responses in accordance with the scoring criteria. In disclosed embodiments, the methods provide such distribution for multiple scorers on a network located at a single site, for multiple scorers on a network remote from a central site; and scoring by individual scorers remote from the central site. Distribution may be accomplished by transferring data from electronic files using diverse techniques such as magnetic storage discs, CD-ROM, cable or high-speed public communications network, in accordance with the needs of the particular job and setting.

Abstract: A method and system for increasing the speed at which data items are processed and evaluated. The method and system receives electronic representations of data items and organizes the data items into separate groupings. Next a particular grouping of data items is displayed to an evaluator so that the evaluator can evaluate the data items. In displaying the data items, the method and system may use a matrix of cells and display one data item in each of the cells. The evaluator can thus simultaneously view multiple data items, which facilitates efficient evaluation of the data items.

Abstract: An optical reading head system has a calibrated pixel output, consisting of a discrete value for each of a plurality of pixels. The system has a scanning station with an exposure region that includes an image region and a detector including an array of photosensitive sites that sense reflected light from the exposure region during a detector exposure period to generate a pixel output profile of the image region. There are a plurality of light sources for lighting the exposure region. Each light source emits an individually controllable amount of light to the exposure region during a detector exposure period such that the system's pixel output is acceptably calibrated. A calibration medium is supplied with a calibration region, and a target pixel output profile is defined for the calibration. The individually controllable light sources are adjusted to make the pixel output profile acceptably within range of the target pixel output profile.