AUTOMATED GRADING FOR INTERACTIVE LEARNING APPLICATIONS

Abstract

In various embodiments, grades are assigned to student annotations associated with an educational resource based on grading features derived utilizing human grades of annotations in a training set of annotations.

Description

RELATED APPLICATIONS

This application claims the benefit of and priority to U.S. Provisional Patent Application No. 62/261,387, filed Dec. 1, 2015, U.S. Provisional Patent Application No. 62/261,397, filed Dec. 1, 2015, U.S. Provisional Patent Application No. 62/261,398, filed Dec. 1, 2015, and U.S. Provisional Patent Application No. 62/261,400, filed Dec. 1, 2015, the entire disclosure of each of which is hereby incorporated herein by reference.

TECHNICAL FIELD

In various embodiments, the present invention relates generally to online learning, and in particular to resources for enhancing and personalizing learning experiences involving an online component.

BACKGROUND

As digital textbooks inexorably replace traditional printed media, and online social resources such as discussion boards supplement classroom instruction, teachers and publishers are finding new opportunities for engaging students. Students with access to digital materials may annotate a shared digital version of a class text or videos, ask and answer each other's questions, and interact with the teaching staff while reading. The advantages are substantial: instead of waiting days until office hours to get past a conceptual roadblock, students can ask a question at any time and often get a response within minutes. Student motivation is enhanced through online interactions that enable them to share interest and knowledge.

In increasing number of classrooms, when students are given reading material as homework assignments, it is in digital format and they are allowed to highlight a passage and add a comment or question. Other students (and the teaching staff) can then see this immediately and can answer questions or add their own comments (in an interaction that looks roughly as it does on Facebook). Students stumped about some problem can easily address it, whatever the hour, if other students are reading at the same time or soon after. When students are assigned videos, they may now be able to annotate the timeline, with comments and interactions following.

Research has shown that students who engage in high levels of meaningful online discussion using annotation systems have higher normalized learning gain scores than students who participate just to fulfill basic requirements. Moreover, providing students with incentives to complete the readings thoughtfully and feedback on their annotations helps ensure that students do the assigned readings on time. Overall, when integrated properly into the classroom experience, annotations and their evaluation contribute meaningfully to student learning.

As annotation systems assume a greater role in learning, students may be graded on their performance—how insightful their questions are, how much they help their fellow students, how engaged they are with the readings, what kinds of contributions they make, etc. Indeed, the success of these new approaches may depend on feedback provided in the form of grades. Unfortunately, expanded opportunities for assessment add to instructors' grading burden; ironically, the more successful the teachers are in encouraging out-of-class learning, the more work they will have.

Unlike direct assessments such as quizzes or tests, evaluating an annotation requires considerable expertise that cannot readily be automated. But teachers will hesitate to adopt new educational technology if it adds to an already onerous workload. Thus, there is a need for techniques and systems for the evaluation of annotations supplied by participants in online learning scenarios without significant increases to the workload of the instructor and/or other course managers.

SUMMARY

Embodiments of the invention automate critical aspects of student performance evaluation for online resources in a manner more efficient and accurate than a single human evaluator could achieve on his or her own. In various embodiments, systems and methods in accordance with the invention monitor users of an online resource (e.g., readers of an e-book), and grade the users' annotations of the online resource in an automated fashion based on criteria derived from the inputs of multiple human evaluators. As utilized herein, the term “annotation” refers to any feedback supplied by a student in response to and/or associated with an educational resource. Annotations may include, for example, answers to embedded questions, comments related to specific passages of the resource, or both. Systems and methods in accordance with embodiments of the invention may utilize as inputs (1) a large number of predetermined features (i.e., grading criteria) that are automatically and adaptively selected, modified, and combined to produce novel features not previously utilized for grading, (2) optionally, criteria from instructional staff and teaching assistants, and/or (3) judgments from students completing an annotation-based assignment, which may simply be aimed at allowing the students to learn about a specific class, reading, and/or grading rubric.

In various embodiments of the invention, these inputs are utilized to obtain multiple judgments about each annotation or each set of annotations by a student for an assignment. These multiple judgments are then averaged in order to produce a training set of annotations utilized as the basis for grading of subsequent annotations. The average has much less error and noise than what any one human coder could accomplish, especially since a human coder, such as the instructor, would need to grade thousands of annotations to keep up with a class over many assignments. In real classrooms, it would be infeasible to assign multiple instructors or teaching staff to code each individual annotation; in most classrooms, it may even be infeasible for each of the thousands of individual annotations to be graded by one instructor. Embodiments of the present invention, trained on the much less noisy average for each annotation in the training set, perform much better in practice than any individual human coder is likely to do or even could do, and also better than any automated algorithm based on a training set constructed in a conventional manner with one (noisy) coder per annotation.

As utilized herein, the term “class” refers to a gathering of “users,” “participants,” or “students” led by one or more “instructors.” Participants need not be in the same room as each other or the instructor, so classes encompass distance learning situations. In addition, participants need not be students; they might be employees participating in a corporate training event or workshop participants attending an educational workshop. Accordingly, the terms “participant” and “student” are used interchangeably herein, it being understood that the utility of the invention is not limited to students in classroom environments. In addition, the term “instructor” used herein is not limited to a teacher or a professor in the classroom; the “instructor” may be a facilitator in a corporate event or in any group pursuing a pedagogical or intellectual endeavor.

In an aspect, embodiments of the invention feature an automated grading method for student annotations in an interactive learning application. In a step (a), an interactive educational resource is distributed over a network to a plurality of student devices. In a step (b), an initial set of annotations generated at the student devices in response to the educational resource is received at a server. In a step (c), a plurality of grades for each of the annotations in the initial set of annotations is received at the server, each of the grades being provided by a different human grader. In a step (d), the plurality of grades to produce an average grade for each of the annotations in the initial set of annotations, at least a portion of the initial set of annotations constituting a training set. In a step (e), portions of annotations within the training set are extracted, thereby producing a plurality of seed features. In a step (f), one or more grading features predictive of the average grades associated with the training set are computationally derived from the seed features. In a step (g), a grade is assigned to a new annotation based on the one or more grading features.

Embodiments of the invention may include one or more of the following in any of a variety of combinations. The new annotation may be ungraded by human graders prior to the grade being assigned based on the one or more grading features. Step (g) may include or consist essentially of using a machine-learning model to predict the grade assigned to the new annotation based on the one or more grading features. The model may be predictive in accordance with a prediction algorithm and may be generated by steps including or consisting essentially of (i) dividing the initial set of annotations into the training set and a testing set, each of the training set and testing set including, consisting essentially of, or consisting of a plurality of annotations and average grades associated therewith, and (ii) identifying the one or more grading features based on predictive reliability in accordance with the prediction algorithm. Grades for one or more annotations within the testing set may be computationally predicted based on the one or more grading features. Parameters of the model may be adjusted, e.g., prior to assigning the grade to the new annotation. Parameters of the model may be adjusted based on the predicted grades for one or more annotations within the testing set. Parameters of the model may be adjusted based on grades assigned by an instructor, e.g., grades assigned by an instructor that override grades predicted by the prediction algorithm. The prediction algorithm may include, consist essentially of, or consist of a classification tree. The prediction algorithm may include, consist essentially of, or consist of a random forest. The random forest may include, consist essentially of, or consist of a plurality of regression trees. At least a portion of the initial set of annotations may be distributed over the network to a plurality of human graders before step (c). Access to the educational resource by the student device at which the new annotation was generated may be controlled based at least in part on the grade assigned to the new annotation. The grade assigned to the new annotation may be displayed, e.g., on the student device at which the new annotation was generated, on an instructor device, and/or on a grading device. Step (e) may include, consist essentially of, or consist of applying natural-language processing to annotations within the training set.

In another aspect, embodiments of the invention feature an education system that includes, consists essentially of, or consists of a plurality of student devices for executing an interactive educational resource received over a network and a server in electronic communication with the student devices. The student devices are configured to receive student annotations of the educational resource and transmit the annotations to the server. The server includes, consists essentially of, or consists of a communication module and an analysis module. The communication module is configured to (i) receive annotations from the student devices, and (ii) receive grades associated with annotations from a plurality of human graders. The analysis module is configured to (i) associate composite grades (e.g., an average of a plurality of grades received from different human graders) with each of an initial set of annotations, at least a portion of the initial set of annotations constituting a training set, (ii) computationally derive one or more grading features predictive of the average grades associated with the training set, and (iii) assign grades to ungraded annotations based on the one or more grading features.

Embodiments of the invention may include one or more of the following in any of a variety of combinations. The analysis module may be configured to extract portions of annotations within the training set, thereby producing a plurality of seed features. The one or more grading features may be computationally derived from the seed features. The analysis module may use a machine-learning model to predict the grades assigned to the ungraded annotations based on the one or more grading features. The model may be predictive in accordance with a prediction algorithm and may be generated by steps including or consisting essentially of (i) dividing the initial set of annotations into the training set and a testing set, each of the training set and testing set including, consisting essentially of, or consisting of a plurality of annotations and average grades associated therewith, and (ii) identifying the one or more grading features based on predictive reliability in accordance with the prediction algorithm. The analysis module may be configured to (i) computationally predict, based on the one or more grading features, grades for one or more annotations within the testing set, and (ii) adjust parameters of the model based on the predictions, e.g., prior to assigning grades to ungraded annotations. The prediction algorithm may include, consist essentially of, or consist of a classification tree. The prediction algorithm may include, consist essentially of, or consist of a random forest. The random forest may include, consist essentially of, or consist of a plurality of regression trees. At least one of the student devices may include, consist essentially of, or consist of a computer or a handheld device. The communication module may be configured to transmit grades assigned by the analysis module to one or more student devices, to one or more instructor devices, and/or to one or more grading devices. The system may include a plurality of grading devices configured to display student annotations of the educational resource, receive grades associated with the student annotations from a human grader, and transmit the grades to the server. At least one of the grading devices may include, consist essentially of, or consist of a computer or a handheld device.

These and other objects, along with advantages and features of the present invention herein disclosed, will become more apparent through reference to the following description, the accompanying drawings, and the claims. Furthermore, it is to be understood that the features of the various embodiments described herein are not mutually exclusive and may exist in various combinations and permutations. As used herein, the terms “approximately” and “substantially” mean ±10%, and in some embodiments, ±5%. The term “consists essentially of” means excluding other materials that contribute to function, unless otherwise defined herein.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, like reference characters generally refer to the same parts throughout the different views. Also, the drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments of the present invention are described with reference to the following drawings, in which:

FIG. 1 is a schematic depiction of an educational environment in accordance with various embodiments of the invention;

FIG. 2 is a block diagram of an educational server or system utilized in accordance with various embodiments of the invention; and

FIG. 3 is a flowchart of a technique of automated grading of student annotations in accordance with various embodiments of the invention.

DETAILED DESCRIPTION

FIG. 1 illustrates an exemplary educational environment 100 in accordance with embodiments of the present invention. As shown, within the environment 100, communication is established, via a network 110, among an instructor 120 utilizing an instructor device 130, various students 140 each utilizing a student device 150, one or more graders 160 each utilizing a grading device 170, and an educational system or server 180. The network 110 may include or consist essentially of, for example, the Internet and/or one or more local-area networks (LANs) or wide-area networks (WANs). The terms “student device,” “instructor device,” and “grading device” as used herein broadly connote any electronic device or system facilitating wired and/or wireless bi-directional communications, and may include computers (e.g., laptop computers and/or desktop computers), handheld devices, or other personal communication devices. Handheld devices include, for example, smart phones or tablets capable of executing locally stored applications and supporting wireless communication and data transfer via the Internet or the public telecommunications infrastructure. Smart phones include, for example, IPHONES (available from Apple Inc., Cupertino, Calif.), BLACKBERRIES (available from RIM, Waterloo, Ontario, Canada), or any mobile phones equipped with the ANDROID platform (available from Google Inc., Mountain View, Calif.); tablets, such as the IPAD and KINDLE FIRE; and personal digital assistants (PDAs). The bi-directional communication and data transfer may take place via, for example, one or more of cellular telecommunication, a Wi-Fi LAN, a point-to-point Bluetooth connection, and/or an NFC communication.

FIG. 2 depicts a more detailed schematic of the server 180, which includes or consists essentially of a general-purpose computing device whose operation is directed by a computer processor, i.e., central processing unit (CPU) 200. The server 180 includes a network interface 205 that facilitates communication over the network 110, using hypertext transfer protocol (HTTP) or other suitable protocols. For example, the network interface 205 may include or consist essentially of one or more hardware interfaces enabling data communication via network 110, as well as a communications module for sending, receiving, and routing such communications within server 180 (e.g., via system bus 210). The server 180 further includes a bi-directional system bus 210, over which the system components communicate, a main (typically volatile) system memory 215, and a non-volatile mass storage device (such as one or more hard disks and/or optical storage units) 220, which may contain resources, such as digital textbooks and/or other educational resources, that may be delivered to the student devices 150.

The main memory 215 contains instructions, conceptually illustrated as a group of modules, which control the operation of the CPU 200 and its interaction with the other hardware components. An operating system 225 directs the execution of low-level, basic system functions such as memory allocation, file management and operation of mass storage devices 220. The operating system 225 may be or include a variety of operating systems such as Microsoft WINDOWS operating system, the Unix operating system, the Linux operating system, the Xenix operating system, the IBM AIX operating system, the Hewlett Packard UX operating system, the Novell NETWARE operating system, the Sun Microsystems SOLARIS operating system, the OS/2 operating system, the BeOS operating system, the MACINTOSH operating system, the APACHE operating system, an OPENSTEP operating system or another operating system of platform.

A resource-management module 230 is responsible for, e.g., allowing properly authenticated students 140 to access privileged educational resources via their devices 150, and for monitoring the students' interactions with these resources. The resource-management module 230 may also control and facilitate access to educational resources for the instructor 120 via the instructor device 130 and/or for the graders 160 via grading devices 170. It should be understood that resources provided to the student devices 150 need not reside physically within the server 180; the resource-management module 230 may obtain resources from other servers, or direct other servers (e.g., an educational publisher's server) to provide resources to student devices. It should further be understood that the access-control functions of the resource-management module 230 are well known to those skilled in the art of online educational platforms and, more generally, to access control for resources available online or via a private network.

An analysis engine 235 monitors student interaction with educational resources provided by the server 180 and utilizes human-originated grades of a subset of student-generated content (i.e., annotations) to grade subsequent annotations. As is well known, digital textbooks and similar electronic materials, when opened on a student device 150, may have embedded code that communicates actions (e.g., page turning) or annotations via the device 150 back to the server 180. The annotations are analyzed and grading is performed as described further below. The server 180 performs grading based on a machine-learning or statistical model trained on the average of two or more human grades provided for each annotation in an initial set. The human grades are received from the grading devices 170 and are based on the assessments of the graders 160 of student-generated content (e.g., annotations) supplied to the grading devices 170 after such content is generated by the students 140 via student devices 150. Once developed, the machine-learning model may be utilized by server 180 for the grading of subsequent sets of annotations without the need for further human input (e.g., further grading by human graders 160).

The grades assigned by the grading algorithm may be delivered to students by resource-management module 230 (e.g., via the student's device 150 as a pop-up message or via another channel, such as the student's mobile phone, wireless tablet, or other device). For example, the server 180 may maintain or have access to a student database 240 containing contact information for each student, including email addresses, phone numbers (e.g., to which text messages may be sent). In various embodiments of the invention, the student database 240 maintains rosters of classes, sections, and students within each class section. The server 180 may even utilize the grades assigned by the grading algorithm, for example, to assign supplementary lessons or other materials to low-performing students 140, and/or to control student access to subsequent lessons. For example, a student 140 might not have the ability to access a subsequent lesson via his or her device 150 until a satisfactory grade is achieved on a particular lesson and/or aspects of a lesson are repeated. In another exemplary embodiment, access to different portions of the educational resource, or even to different educational resources may be selectively provided to one or more students 140 depending upon performance, e.g., grades assigned to one or more annotations.

In various embodiments of the invention, the server 180 may also incorporate a discussion hosting server 245 that supports a discussion platform and makes this available to students 140 via their devices 150. The discussion platform may be a server-hosted discussion board that operates autonomously, in the manner of a social-media platform, or may be associated with resources 220. For example, server 245 may perform the functions of resource-management module 230 and facilitate access to resources 220 that have annotation fields into which students 140 may enter comments that server 245 organizes as annotation threads. Server 245 may be part of the main server 180 or may be a separate device.

The server 180 may also include, in various embodiments of the invention, a repository or database 250 that stores various reports related to the interactions of students 140, the instructor 120, and/or graders 160 with the resources 220 (and/or with content related thereto, such as student annotations). For example, the repository 250 may store grade reports generated by graders 160 or reports for the instructor 120 based on and/or highlighting questions, comments, and/or annotations generated by the students 140. For example, such reports may include links to annotations stored on the discussion server 245.

FIG. 3 depicts a method 300 for the automated grading of student annotations in accordance with various embodiments of the present invention. In step 305, an educational resource or a portion thereof (e.g., from storage 220) is electronically distributed to one or more student devices 150 via network 110. During use of the educational resource (e.g., reading of one or more passages in an electronic textbook and/or answering questions related to the resource) by the students 140, the students 140 supply annotations related to the resource via their student devices 150. In step 310, those annotations are transmitted to the server 180 via network 110, and in step 315, the server 180 receives grades from human graders via their grading devices 170 based on the annotations received in step 310. For example, an initial set of annotations generated by a class or other group of students 140 (e.g., annotations taken from the first reading assignment for a class) may be graded at least twice utilizing human graders 160. For example, the initial annotations may be graded by graders 160 that include or consist essentially of (1) staff graders, i.e., teaching assistants hand-grading the set of annotations with a research-based rubric, (2) peer graders who, in the process of learning about the scoring rubric that will be used to evaluate their annotations, score a subset of their peers' annotations through a calibration grading exercise (thus, one or more of the graders 160 may also be a student 140), and/or (3) dedicated human graders not enrolled in the class.

In step 320, composite grades are determined for each of the initial annotations, by the analysis engine 235, based on the grades assigned by each of the different human graders 160, thereby minimizing the noise inherent in the utilization of grades from only a single evaluator. For example, the different grades for the initial annotations may be averaged together to derive a composite grade for each annotation. Given the well-known variability among even expert human graders, averaging generally increases the statistical reliability of the grade.

In step 325, the set of initial annotations with the composite grades is then divided into a training set and a testing set by the analysis engine 235 for the development of the machine-learning model utilized in embodiments of the present invention. In step 330, the training set is analyzed by analysis engine 235 to extract seed features from the graded annotations. Typically, these seed features are simple text attributes heuristically associated with the quality of the annotation—e.g., annotation length and vocabulary sophistication (as measured, for example, by word length or a frequency metric such as term frequency—inverse document frequency, which indicates the importance of a word within an annotation based on its frequency of use among annotations or in written language with subject-matter relevance to the class). The process of identifying seed features may also utilize conventional natural-language processing techniques such as stemming, stop-word removal and/or part-of-speech tagging. In general, the seed features include or consist essentially of, e.g., words, phrases, portions of words or phrases, and/or one or more predetermined answer choices supplied to students in response to questions (for example in a multiple-choice or true/false format) from the training set.

In step 335, one or more grading features are subsequently produced from combinations and/or permutations of the seed features, and those grading features are utilized in step 340 to predict grades for the annotations within the testing set defined in step 325. This is accomplished by training the machine-learning model on the seed features. For example, the initial, human-graded annotations may be used as a training set for a text-analytic regression procedure (e.g., logistic regression, classification tree, random forest classifier, etc.) that constitutes the machine-learning model. More generally, the machine-learning model may be any suitable analytic framework for analyzing text and making predictions based on a training set, including classification and regression trees (CART), neural networks, or other suitable framework. Machine-learning models are well-characterized in the art and may be implemented without undue experimentation.

For example, in various embodiments, the analysis engine 235 utilizes a random forest classifier as the basis for the machine-learning model. In one embodiment, human coders grade annotations to form a training set, and the grades they assign to each annotation are averaged. The grades serve as category labels for the machine-learning model. Seed features are extracted from the text and one or, more preferably, an ensemble of classifiers is used to fit the model to predict the category labels from the seed features. The seed features, along with permutations and combinations thereof, form a set of candidate grading features.

The grading features with sufficient predictive reliability against the training set may be selected for use in the model. The predictive reliability of a feature may be deemed sufficient, for example, based on standard error, t value, p value or another statistical metric, for example, a minimum p value required for a seed feature to qualify as a grading feature set at a standard level of 0.01 or less. (The p value reflects the probability that the feature has no predictive value.) Typically, the training set will have 100 or more entries each reflecting a grade assigned to an annotation.

Following creation of the model and feature selection using the training set, the performance of the model may be evaluated using the testing set. For example, the false positive and false negative predictions obtained against the testing set may be used to detect overfitting, identify and prune features exhibiting multicollinearity, and set a classification threshold that produces a desired level of sensitivity (true positive rate) and specificity (true negative rate). Features may also be assessed against the “inter-coder” reliability, i.e., the fraction of times the different human graders agree on a grade for a particular annotation. For example, a set of grading features that produces predicted grades having values different from the composite grades by less than a predetermined threshold amount (e.g., ±10%, ±5%, ±2%, etc.) may be selected for subsequent automated grading of student annotations. In this manner, the grades in the testing set may be predicted more accurately than via training based on grades from a single human grader.

As known to those of skill in the art, random forest classifiers operate via the construction of several decision trees based on the training set, and output the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees—thereby correcting for potential overfitting of the training set. Other text-analytic techniques, as noted above, may be utilized by the analysis engine 235 to determine grading features that predict grades assigned to the annotations in the testing group. As shown in FIG. 3, steps 335 and 340 may be repeated one or more times to refine the machine-learning model to include an ensemble of grading features (i.e., classifiers extracted from the seed features) that most accurately predicts the composite grades in the testing set, or an ensemble of grading features that at least predicts the composite grades to within a desired level of accuracy. In addition, the machine-learning model (and grading features utilized thereby) may be refined based upon grades assigned to particular annotations by, e.g., the instructor 120 or a grader 160. Such human-assigned grades may override grades previously predicted and/or assigned by the machine-learning model and may be utilized, e.g., within the testing set or training set, to further refine the machine-learning model for one or more students 140.

In step 345, new student annotations are received by the server 180. For example, such annotations may include annotations related to a subsequent exercise or received from a different class of students 140 than those whose annotations were utilized to define the grading features. Using the grading features, the analysis engine 235 assigns grades to such subsequent sets of student annotations in step 350. The grades thus assigned may be displayed, for example, to the instructor 120 (e.g., via instructor device 130) and/or to the individual students 140 (e.g., via their student devices 150) in optional step 355.

The resource-management module 230 and analysis engine 235 (and, e.g., a communications module within or corresponding to network interface 205) may be implemented by computer-executable instructions, such as program modules, that are executed by a conventional computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that performs particular tasks or implement particular abstract data types. Those skilled in the art will appreciate that embodiments of the invention may be practiced with various computer system configurations, including multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. Embodiments of the invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer-storage media including memory storage devices.

Any suitable programming language may be used to implement without undue experimentation the analytical functions described above. Illustratively, the programming language used may include assembly language, Ada, APL, Basic, C, C++, C*, COBOL, dBase, Forth, FORTRAN, Java, Modula-2, Pascal, Prolog, Python, REXX, and/or JavaScript, for example. Regression-based models (e.g., logistic regression, classification trees and random forests) are readily implemented in the R programming language without undue experimentation (using, e.g., the rpart and randomForest libraries), and neural networks may be implemented in Python or MATLAB. Further, it is not necessary that a single type of instruction or programming language be utilized in conjunction with the operation of embodiments of the invention. Rather, any number of different programming languages may be utilized as is necessary or desirable.

The server 180 may also include other removable/nonremovable, volatile/nonvolatile computer storage media. For example, a hard disk drive may read or write to nonremovable, nonvolatile magnetic media. A magnetic disk drive may read from or writes to a removable, nonvolatile magnetic disk, and an optical disk drive may read from or write to a removable, nonvolatile optical disk such as a CD-ROM or other optical media. Other removable/nonremovable, volatile/nonvolatile computer storage media that may be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The storage media are typically connected to the system bus through a removable or non-removable memory interface.

The processing units that execute commands and instructions may be general-purpose processors, but may utilize any of a wide variety of other technologies including special-purpose hardware, a microcomputer, mini-computer, mainframe computer, programmed microprocessor, microcontroller, peripheral integrated circuit element, a CSIC (customer-specific integrated circuit), ASIC (application-specific integrated circuit), a logic circuit, a digital signal processor, a programmable logic device such as an FPGA (field-programmable gate array), PLD (programmable logic device), PLA (programmable logic array), RFID processor, smart chip, or any other device or arrangement of devices that is capable of implementing the steps of the processes of the invention.

Communication may occur over the Internet, as illustrated, and/or over an intranet, extranet, Ethernet, the public telecommunications infrastructure, or any other system that provides communications. Some suitable communications protocols may include TCP/IP, UDP, or OSI for example. For wireless communications, communications protocols may include Bluetooth, Zigbee, IrDa or other suitable protocol. Furthermore, components of the system may communicate through a combination of wired or wireless paths.

The terms and expressions employed herein are used as terms and expressions of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding any equivalents of the features shown and described or portions thereof. In addition, having described certain embodiments of the invention, it will be apparent to those of ordinary skill in the art that other embodiments incorporating the concepts disclosed herein may be used without departing from the spirit and scope of the invention. Accordingly, the described embodiments are to be considered in all respects as only illustrative and not restrictive.

Claims

1. An automated grading method for student annotations in an interactive learning application, the method comprising:

(a) distributing an interactive educational resource over a network to a plurality of student devices;

(b) receiving, at a server, an initial set of annotations generated at the student devices in response to the educational resource;

(c) receiving, at the server, a plurality of grades for each of the annotations in the initial set of annotations, each of the grades being provided by a different human grader;

(d) averaging the plurality of grades to produce an average grade for each of the annotations in the initial set of annotations, at least a portion of the initial set of annotations constituting a training set;

(e) extracting portions of annotations within the training set, thereby producing a plurality of seed features;

(f) computationally deriving, from the seed features, one or more grading features predictive of the average grades associated with the training set; and

(g) assigning a grade to a new annotation based on the one or more grading features.

2. The method of claim 1, wherein step (g) comprises using a machine-learning model to predict the grade assigned to the new annotation based on the one or more grading features, the model being predictive in accordance with a prediction algorithm and generated by steps comprising:

dividing the initial set of annotations into the training set and a testing set, each of the training set and testing set comprising a plurality of annotations and average grades associated therewith; and

identifying the one or more grading features based on predictive reliability in accordance with the prediction algorithm.

3. The method of claim 2, further comprising the steps of:

computationally predicting, based on the one or more grading features, grades for one or more annotations within the testing set; and

adjusting parameters of the model prior to assigning the grade to the new annotation.

4. The method of claim 2, wherein the prediction algorithm is a classification tree.

5. The method of claim 4, wherein the prediction algorithm is a random forest comprising a plurality of regression trees.

6. The method of claim 1, further comprising, before step (c), distributing the initial set of annotations over the network to a plurality of human graders.

7. The method of claim 1, further comprising controlling access to the educational resource by the student device at which the new annotation was generated based at least in part on the grade assigned to the new annotation.

8. The method of claim 1, further comprising displaying the grade assigned to the new annotation on the student device at which the new annotation was generated.

9. The method of claim 1, wherein step (e) comprises applying natural-language processing to annotations within the training set.

10. An educational system comprising:

a plurality of student devices for executing an interactive educational resource received over a network, the student devices being configured to receive student annotations of the educational resource and transmit the annotations to a server; and

a server in electronic communication with the student devices, the server comprising: a communication module configured to (i) receive annotations from the student devices, and (ii) receive grades associated with annotations from a plurality of human graders, and an analysis module configured to (i) associate an average of a plurality of grades received from different human graders with each of an initial set of annotations, at least a portion of the initial set of annotations constituting a training set, (ii) computationally derive one or more grading features predictive of the average grades associated with the training set, and (iii) assign grades to ungraded annotations based on the one or more grading features.

11. The system of claim 10, wherein the analysis module is configured to extract portions of annotations within the training set, thereby producing a plurality of seed features, wherein the one or more grading features are computationally derived from the seed features.

12. The system of claim 10, wherein the analysis module uses a machine-learning model to predict the grades assigned to the ungraded annotations based on the one or more grading features, the model being predictive in accordance with a prediction algorithm and generated by steps comprising:

dividing the initial set of annotations into the training set and a testing set, each of the training set and testing set comprising a plurality of annotations and average grades associated therewith; and

identifying the one or more grading features based on predictive reliability in accordance with the prediction algorithm.

13. The system of claim 12, wherein the analysis module is configured to:

computationally predict, based on the one or more grading features, grades for one or more annotations within the testing set; and

adjust parameters of the model based on the predictions prior to assigning grades to ungraded annotations.

14. The system of claim 12, wherein the prediction algorithm is a classification tree.

15. The system of claim 14, wherein the prediction algorithm is a random forest comprising a plurality of regression trees.

16. The system of claim 10, wherein at least one of the student devices comprises a computer or a handheld device.

17. The system of claim 10, wherein the communication module is configured to transmit grades assigned by the analysis module to the student devices.

18. The system of claim 10, further comprising a plurality of grading devices configured to display student annotations of the educational resource, receive grades associated with the student annotations from a human grader, and transmit the grades to the server.

19. The system of claim 18, wherein at least one of the grading devices comprises a computer or a handheld device.