AUTOMATED PERSONALIZED FEEDBACK FOR INTERACTIVE LEARNING APPLICATIONS

Abstract

In various embodiments, actions taken on student devices by students using an interactive educational resource are analyzed to determine the students' likelihood of educational success and, if the likelihood is less than a threshold, the students are notified of deficiencies in their actions.

Description

RELATED APPLICATIONS

This application claims the benefit of and priority to U.S. Provisional Patent Application No. 62/261,387, filed Dec. 1, 2015, U.S. Provisional Patent Application No. 62/261,397, filed Dec. 1, 2015, U.S. Provisional Patent Application No. 62/261,398, filed Dec. 1, 2015, and U.S. Provisional Patent Application No. 62/261,400, filed Dec. 1, 2015, the entire disclosure of each of which is hereby incorporated herein by reference.

TECHNICAL FIELD

In various embodiments, the present invention relates generally to online learning, and in particular to resources for enhancing and personalizing learning experiences involving an online component.

BACKGROUND

As digital textbooks inexorably replace traditional printed media, and online social resources such as discussion boards supplement classroom instruction, teachers and publishers are finding new opportunities for engaging students. Students with access to digital materials may annotate a shared digital version of a class text or videos, ask and answer each other's questions, and interact with the teaching staff while reading. The advantages are substantial: instead of waiting days until office hours to get past a conceptual roadblock, students can ask a question at any time and often get a response within minutes. Student motivation is enhanced through online interactions that enable them to share interest and knowledge.

In increasing number of classrooms, when students are given reading material as homework assignments, it is in digital format and they are allowed to highlight a passage and add a comment or question. Other students (and the teaching staff) can then see this immediately and can answer questions or add their own comments (in an interaction that looks roughly as it does on Facebook). Students stumped about some problem can easily address it, whatever the hour, if other students are reading at the same time or soon after. When students are assigned videos, they may now be able to annotate the timeline, with comments and interactions following.

Research has shown that students who engage in high levels of meaningful online discussion using annotation systems have higher normalized learning gain scores than students who participate just to fulfill basic requirements. Moreover, providing students with incentives to complete the readings thoughtfully and feedback on their annotations helps ensure that students do the assigned readings on time. Overall, when integrated properly into the classroom experience, annotations and their evaluation contribute meaningfully to student learning.

As annotation systems assume a greater role in learning, students may be graded on their performance—how insightful their questions are, how much they help their fellow students, how engaged they are with the readings, what kinds of contributions they make, etc. Indeed, the success of these new approaches may depend on feedback provided in the form of grades. Unfortunately, expanded opportunities for assessment add to instructors' grading burden; ironically, the more successful the teachers are in encouraging out-of-class learning, the more work they will have.

Unlike direct assessments such as quizzes or tests, evaluating an annotation requires considerable expertise that cannot readily be automated. But teachers will hesitate to adopt new educational technology if it adds to an already onerous workload. Thus, there is a need for techniques and systems for the evaluation of participants in online learning scenarios while encouraging appropriate engagement with online educational resources without significant increases to the workload of the instructor and/or other course managers.

SUMMARY

Embodiments of the invention automate important aspects of student performance evaluation for online resources while retaining personalization and individualized assessment. In various embodiments, systems and techniques in accordance with the invention assist users of an online resource (e.g., readers of an e-book), and ensure for the instructor that the students get out of it what is expected, by tagging passages in the resource using metrics derived based on students' reading and commenting behavior (e.g., to take two simple examples, the amount of time spent reading or the number of comments made on each passage in the readings). Historical data may be used to model how these metrics predict student understanding and then alert students when their reading or commenting behavior falls short of key benchmarks. By nudging students appropriately, embodiments of the invention may encourage them to engage the material well in advance of class. Thus, when suboptimal behavior for a particular reading assignment is detected (e.g., too few annotations by a certain time before the reading deadline, or a particular scrolling pattern that suggests the student is skimming and not reading, to name just two examples), the student may receive an electronic alert (e.g., via email, text message, mobile phone notification, etc.).

Systems in accordance with embodiments of the present invention may also use aggregate reading behavior data and establish context-specific norms for the various passages of the reading. Such norms may reflect instructor preferences or student success in the course, i.e., reading behaviors correlated with successful course outcomes may be assessed at the level of specific passages, and the favored behaviors may be employed as norms in subsequent class sessions.

Students may receive immediate feedback (e.g., private “pop-up” nudges, or nudges in the form of email or other electronic communication) when online reading behavior deviates significantly from that of a student cohort—most simply, the majority of other students in the class, but the cohort may be more narrow, e.g., students from an earlier class session who passed or did well on the particular assignment. For example, a student who spends very little time on a passage (e.g., as measured by page turns or scrolling patterns) that many others students had previously read carefully or commented extensively on may receive a suggestion from the system that, based on their classmates' behavior, s/he should take greater care with that passage of the reading to ensure full understanding.

Students may also receive “personalized” emails or other electronic communications that, at the instructor's choice, appear to come from their instructor but, in fact, are auto-generated by the system based on that student's reading and annotating behavior. These emails provide general feedback on the student's behavior in terms of its likely effect on overall performance in the course, or to encourage the student to look at a specific annotation based on questions that student has asked. Depending on the triggering behavior, the instructor message may provide encouragement and inspiration to continue active participation in the discussion or an early warning to the student that the detected behavior is predictive of a poor final grade in the class; the message may also suggest steps the student might follow to increase the chances of success.

As utilized herein, the term “annotation” refers to any feedback supplied by a student in response to and/or associated with an educational resource. Annotations may include, for example, answers to embedded questions, comments related to specific passages of the resource, or both. As utilized herein, the term “class” refers to a gathering of “users,” “participants,” or “students” led by one or more “instructors.” Participants need not be in the same room as each other or the instructor, so classes encompass distance learning situations. In addition, participants need not be students; they might be employees participating in a corporate training event or workshop participants attending an educational workshop. Accordingly, the terms “participant” and “student” are used interchangeably herein, it being understood that the utility of the invention is not limited to students in classroom environments. In addition, the term “instructor” used herein is not limited to a teacher or a professor in the classroom; the “instructor” may be a facilitator in a corporate event or in any group pursuing a pedagogical or intellectual endeavor.

In an aspect, embodiments of the invention feature a method of providing automated personalized feedback in an interactive learning application. In a step (a), an interactive educational resource is distributed over a network to a plurality of student devices each associated with a student. In a step (b), signals indicative of actions taken on the student devices with respect to the educational resource are received at a server from the student devices as the resource is used on the devices. The actions are predictive of success with the resource. In a step (c), a likelihood of educational success with the resource for the student associated with the student device is computationally determined, at the server for each of the student devices, by computationally processing the received signals corresponding to each student device with a machine-learning model relating actions to educational success with material contained in the resource. In a step (d), if the likelihood is less than a threshold, (i) at least one deficiency in an action associated with failure of the likelihood to meet the threshold is identified and (ii) a first type of notification of the deficiency is electronically communicated to the student.

Embodiments of the invention may include one or more of the following in any of a variety of combinations. The notification may be a message appearing to originate with an instructor. For each of the student devices, steps (c) and (d) may occur during use of the resource on the student device. The actions may include, consist essentially of, or consist of (i) a number of annotations made, (ii) a scrolling pattern, and/or (iii) an amount of time spent reading. For at least one of the student devices, an effect of the first type of notification on rectifying the deficiency may be monitored, and, if the deficiency is not subsequently rectified, a second type of notification different from the first type may be subsequently sent. The at least one deficiency may include, consist essentially of, or consist of a plurality of deficiencies. For at least one of the devices, different types of notifications may be sent for different deficiencies. An effect of each of the types of notification on rectifying the associated deficiencies may be monitored, and notifications of the type having the best effect may be subsequently sent. The machine-learning model may include, consist essentially of, or consist of a regression model, e.g., a logistic-regression model and/or a linear-regression model.

In another aspect, embodiments of the invention feature an educational system that includes, consists essentially of, or consists of a plurality of student devices for executing an interactive educational resource received over a network and a server in electronic communication with the student devices. The student devices are configured to send signals indicative of actions taken on the student devices with respect to the educational resource, the actions being predictive of success with the resource. The server includes a communication module and is configured to (i) receive the signals from the student devices, (ii) for each of the student devices, computationally determine a likelihood of educational success with the material for a student associated with the student device by computationally processing the received signals corresponding to each student device with a machine-learning model relating actions to educational success with material contained in the resource, and (iii) if the likelihood is less than a threshold, identify at least one deficiency in an action associated with failure of the likelihood to meet the threshold and electronically communicate, via the communication module, a first type of notification of the deficiency to the student.

Embodiments of the invention may include one or more of the following in any of a variety of combinations. The communication module may be configured to send the notification in the form of a message appearing to originate with an instructor. The actions may include, consist essentially of, or consist of (i) a number of annotations made, (ii) a scrolling pattern, and/or (iii) an amount of time spent reading. The server may be configured to, for at least one of the student devices, monitor an effect of the first type of notification on rectifying the deficiency, and, if the deficiency is not subsequently rectified, subsequently send a second type of notification different from the first type. The server may be configured to, for at least one of the student devices, send different types of notifications for different deficiencies, monitor an effect of each of the types of notification on rectifying the associated deficiencies, and subsequently send notifications of the type having the best effect. The machine-learning model may include, consist essentially of, or consist of a regression model, e.g., a logistic-regression model and/or a linear-regression model.

These and other objects, along with advantages and features of the present invention herein disclosed, will become more apparent through reference to the following description, the accompanying drawings, and the claims. Furthermore, it is to be understood that the features of the various embodiments described herein are not mutually exclusive and may exist in various combinations and permutations. As used herein, the terms “approximately” and “substantially” mean±10%, and in some embodiments, ±5%. The term “consists essentially of” means excluding other materials that contribute to function, unless otherwise defined herein.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, like reference characters generally refer to the same parts throughout the different views. Also, the drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments of the present invention are described with reference to the following drawings, in which:

FIG. 1 is a schematic depiction of an educational environment in accordance with various embodiments of the invention;

FIG. 2 is a block diagram of an educational server or system utilized in accordance with various embodiments of the invention; and

FIG. 3 is a flowchart of a technique of providing automated personalized feedback in accordance with various embodiments of the invention.

DETAILED DESCRIPTION

In accordance with embodiments of the invention, iterative and adaptive feedback to the student is used to change student behavior and encourage students to engage more and better understand online learning resources such as electronic textbooks. Systems in accordance with embodiments of the invention may measure the student's response to each electronic nudge and thereby learn the effectiveness of a particular form of nudge to each student; the system may then use this data to guide subsequent communication with the student, to judge the effectiveness of a particular type of nudge across all students, and/or to design a new type of nudge. Nudging students in an adaptive manner tailored to individual students results in better engagement and more successful learning outcomes.

FIG. 1 illustrates an exemplary educational environment 100 in accordance with embodiments of the present invention. As shown, within the environment 100, communication is established, via a network 110, among an instructor 120 utilizing an instructor device 130, various students 140 each utilizing a student device 150, one or more optional graders 160 each utilizing a grading device 170, and an educational system or server 180. Graders 160 may include or consist essentially of, for example, (1) staff graders, i.e., teaching assistants hand-grading student annotations with a research-based rubric, (2) peer graders who, in the process of learning about scoring rubrics used to evaluate annotations, score a subset of their peers' annotations through a calibration grading exercise (thus, one or more of the graders 160 may also be a student 140), and/or (3) dedicated human graders not enrolled in the class.

The network 110 may include or consist essentially of, for example, the Internet and/or one or more local-area networks (LANs) or wide-area networks (WANs). The terms “student device,” “instructor device,” and “grading device” as used herein broadly connote any electronic device or system facilitating wired and/or wireless bi-directional communications, and may include computers (e.g., laptop computers and/or desktop computers), handheld devices, or other personal communication devices. Handheld devices include, for example, smart phones or tablets capable of executing locally stored applications and supporting wireless communication and data transfer via the Internet or the public telecommunications infrastructure. Smart phones include, for example, IPHONES (available from Apple Inc., Cupertino, Calif.), BLACKBERRIES (available from RIM, Waterloo, Ontario, Canada), or any mobile phones equipped with the ANDROID platform (available from Google Inc., Mountain View, Calif.); tablets, such as the IPAD and KINDLE FIRE; and personal digital assistants (PDAs). The bi-directional communication and data transfer may take place via, for example, one or more of cellular telecommunication, a Wi-Fi LAN, a point-to-point Bluetooth connection, and/or an NFC communication.

FIG. 2 depicts a more detailed schematic of the server 180, which includes or consists essentially of a general-purpose computing device whose operation is directed by a computer processor, i.e., central processing unit (CPU) 200. The server 180 includes a network interface 205 that facilitates communication over the network 110, using hypertext transfer protocol (HTTP) or other suitable protocols. For example, the network interface 205 may include or consist essentially of one or more hardware interfaces enabling data communication via network 110, as well as a communications module for sending, receiving, and routing such communications within server 180 (e.g., via system bus 210). The server 180 further includes a bi-directional system bus 210, over which the system components communicate, a main (typically volatile) system memory 215, and a non-volatile mass storage device (such as one or more hard disks and/or optical storage units) 220, which may contain resources, such as digital textbooks and/or other educational resources, that may be delivered to the student devices 150.

The main memory 215 contains instructions, conceptually illustrated as a group of modules, which control the operation of the CPU 200 and its interaction with the other hardware components. An operating system 225 directs the execution of low-level, basic system functions such as memory allocation, file management and operation of mass storage devices 220. The operating system 225 may be or include a variety of operating systems such as Microsoft WINDOWS operating system, the Unix operating system, the Linux operating system, the Xenix operating system, the IBM AIX operating system, the Hewlett Packard UX operating system, the Novell NETWARE operating system, the Sun Microsystems SOLARIS operating system, the OS/2 operating system, the BeOS operating system, the MACINTOSH operating system, the APACHE operating system, an OPENSTEP operating system or another operating system of platform.

A resource-management module 230 is responsible for, e.g., allowing properly authenticated students 140 to access privileged educational resources via their devices 150, and for monitoring the students' interactions with these resources. The resource-management module 230 may also control and facilitate access to educational resources for the instructor 120 via the instructor device 130 and/or for the graders 160 via grading devices 170. It should be understood that resources provided to the student devices 150 need not reside physically within the server 180; the resource-management module 230 may obtain resources from other servers, or direct other servers (e.g., an educational publisher's server) to provide resources to student devices. It should further be understood that the access-control functions of the resource-management module 230 are well known to those skilled in the art of online educational platforms and, more generally, to access control for resources available online or via a private network.

An analysis engine 235 monitors student interaction with educational resources provided by the server 180 and utilizes human-originated grades of a subset of student-generated content (i.e., annotations) as a training set for modeling the relationship between annotations and successful outcomes. As is well known, digital textbooks and similar electronic materials, when opened on a student device 150, may have embedded code that communicates actions (e.g., page turning) or annotations via the device 150 back to the server 180; in other words, the actions are executed on the student device 150 (e.g., the page turns on the device display) and recorded on the server 180 as having occurred for monitoring purposes. These actions and annotations may be analyzed as described further below. Depending on observed student behavior and progress, the analysis engine 235 may signal the resource-management module 230 to prepare a message, alert, or other form of “nudge” and cause this to be delivered to a student (e.g., via the student's device 150 as a pop-up or via another channel, such as the student's mobile phone, wireless tablet, or other device). For example, the server 180 may maintain or have access to a student database 240 containing contact information for each student, including email addresses, phone numbers (e.g., to which text messages may be sent). The student database 240 may also maintain rosters of classes, sections, and students within each class section. As explained below, the analysis engine 235 may determine the best form of communication for individual students, and may both formulate the nudge and direct the resource-management module 230 to send it via the proper medium. In relaying the nudge in the manner dictated by the analysis engine 235, the resource-management module 230 is once again performing conventional communication functions common to online educational platforms.

In various embodiments of the invention, the server 180 may also incorporate a discussion hosting server 245 that supports a discussion platform and makes this available to students 140 via their devices 150. The discussion platform may be a server-hosted discussion board that operates autonomously, in the manner of a social-media platform, or may be associated with resources 220 in the manner of discussion boards maintained by online educational platforms such as edX or COURSERA. For example, server 245 may perform the functions of resource-management module 230 and facilitate access to resources 220 that have annotation fields into which students 140 may enter comments that server 245 organizes as annotation threads. Server 245 may be part of the main server 180 or may be a separate device.

The server 180 may also include, in various embodiments of the invention, a repository or database 250 that stores various reports related to the interactions of students 140, the instructor 120, and/or graders 160 with the resources 220 (and/or with content related thereto, such as student annotations). For example, the repository 250 may store grade reports generated by graders 160 or reports for the instructor 120 based on and/or highlighting questions, comments, and/or annotations generated by the students 140. For example, such reports may include links to annotations stored on the discussion server 245.

FIG. 3 depicts a method 300 for the automated analysis and encouragement of student behavior in accordance with various embodiments of the present invention. In step 305, an educational resource or a portion thereof (e.g., from storage 220) is electronically distributed to one or more student devices 150 via network 110. During use of the educational resource (e.g., reading of one or more passages in an electronic textbook and/or answering questions related to the resource) by the students 140, the students 140 may supply annotations related to the resource via their student devices 150. During use of the resource, one or more metrics meaningful to its effective use are tracked for each student 140, for example, annotation and reading behavior that are predictive of reading comprehension and overall course performance. Useful predictive metrics include the amount of time spent by students reading and annotating before each reading deadline, the amount of time spent reading, the average amount of time students spend on each page of the resource (e.g., textbook), the number of annotations, the ratio of explanations to total annotations (i.e., the proportion of annotations that are explanatory), and the average annotation quality. These predictive metrics may be computed from or, more typically, directly represented by more primitive data such as scrolling and mouse movements. For example, by recording scrolling and mouse movement data, the analysis engine 235 may capture the average amount of time students spend reading each passage of a text. The number and/or length of annotations made by students on the student devices may also be monitored and counted by monitoring actions taken via keyboards (physical or virtual “touchpads”) and/or other input devices. The quality of student annotations may be defined and evaluated as described in accordance with U.S. Provisional Application No. 62/261,398, filed on Dec. 1, 2015, and in U.S. patent application Ser. No. ______, entitled “AUTOMATED GRADING FOR INTERACTIVE LEARNING APPLICATIONS,” filed concurrently herewith, the entire disclosure of each of which is incorporated by reference herein.

Thus, in accordance with embodiments of the present invention, metrics predictive of course performance may be extracted from annotations and/or reading behavior of individual students. In addition, from aggregate, context-specific reading behavior data, the analysis engine 235 may establish norms for various parts of the resource (e.g., passages of a reading). Such norms may reflect instructor preferences or student success in the course, i.e., reading behaviors correlated with successful course outcomes may be assessed at the level of specific passages, and the favored behaviors may be employed as predictive metrics in accordance with embodiments of the invention.

In accordance with embodiments of the invention, a machine-learning algorithm or regression technique (e.g., linear or logistic regression, classification tree, random forest classifier, etc.) is applied to the predictive metrics or, more typically, the underlying tracked parameter values accumulated over one or more classes in which the resource was used, and this results in a model that predicts a student's overall success with the educational resource and/or in the course (e.g., his or her final grade). For example, the analysis engine 235 may apply the machine-learning algorithm or regression technique to a stored collection of tracked student behaviors or actions—represented as high-level metrics (e.g., time spent reading a passage) or low-level indicators (e.g., mouse dwells or movements, number and/or length of annotations, etc.) or a combination—for a collection of former students (e.g., of the same class or utilizing the same or similar resource) that is cross-referenced to the final course grades for those students who took those actions. In this manner, embodiments of the present invention may correlate the student actions with educational success with the material contained within the particular resource in accordance with the selected machine-learning technique. Moreover, since the collection of student actions will also typically include actions that led to sub-optimal outcomes (e.g., poor final grades), the model may also be utilized to identify particular actions or deficiencies in actions (e.g., insufficient numbers of annotations, insufficient time spent on readings overall and/or on particular reading passages, etc.) that are associated with poor performance. As the input reading and annotating actions are collected in real-time for each student 140 from each student device 150, in step 315 the analysis engine 235 applies the model to compute the student's predicted course performance output, i.e., to determine the likelihood of the student's educational success with the resource. For example, the model may be a regression technique that treats the final grade as a categorical outcome variable and, based on the observed data, assigns weights to each metric or indicator based on its predictive value. The observations with known outcomes constitute the training set based on which the model weights are derived. In step 315, the model is applied to one or more new observations (a “testing” set) and makes likelihood-of-success predictions based thereon. These predictions may be associated with a confidence level based on the accuracy of the model based on a metric such as R² error.

In step 320, the computed likelihood of success for each student is compared to a threshold (e.g., set by the instructor) and, if it exceeds the threshold, the system may therefore conclude that the particular student needs no further nudging or encouragement, at least based the actions taken by the student thus far (step 325). For example, if the predicted outcome variable is a grade, the threshold may be set at a B-level. However, when the computed likelihood of success for the student dips below the defined threshold, e.g., with a particular level of confidence, the analysis engine 235 may identify one or more actions and/or deficiencies in actions for the student associated with failure of the likelihood to meet the threshold (step 330). In addition, in step 335 the analysis engine 235 may take an action, e.g., delivering a nudge to the student and/or alerting the instructor. For example, the analysis engine 235 may cause the resource-management module 230 to issue a nudge when a student demonstrates suboptimal behavior based on the machine-learning model, when the student's behavior deviates substantially from that of the majority of other readers, or when analysis of the historical data suggests that the student will respond positively to the nudge and engage in the reading more. Thus, beyond modeling predictive metrics, the analysis engine 235 may be further configured to track the response of each student to different types of nudges delivered in a sequence or based on a previously established correlation between a particular type of nudge and success in motivating positive behavior across students for a given educational resource.

Thus, based on aggregate student behavior over the long-term as captured and stored in the student database 240, the analysis engine 235 may generate personalized emails or other messages containing feedback and that, at the instructor's discretion, may appear to come from the instructor or which the instructor may review, edit, and send to the student. The analysis engine 235 may also record each student's response to each nudge in the database 240. In various embodiments, one or more different types of notifications may be sent to a student if the first notification—selected at random or, as noted above, based on long-term analysis of the correlation between the particular resource and nudge types that have motivated improved behavior—does not correct or improve the student's behavior or performance. The analysis engine 235 records the effect of each successive nudge given to each underperforming student to (i) determine which forms of contact work best for each student using any suitable learning algorithm or simple rote outcome comparison and (ii) improve the accuracy of the model correlating nudge types to particular resources. For example, the system may send different types of notifications to different students and/or in response to deficiencies and, based on the resulting student behavior, optimize the type of feedback for the particular student and/or associated with the particular type of deficiency.

The resource-management module 230 and analysis engine 235 (and, e.g., a communications module within or corresponding to network interface 205) may be implemented by computer-executable instructions, such as program modules, that are executed by a conventional computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that performs particular tasks or implement particular abstract data types. Those skilled in the art will appreciate that embodiments of the invention may be practiced with various computer system configurations, including multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. Embodiments of the invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer-storage media including memory storage devices.

Any suitable programming language may be used to implement without undue experimentation the analytical functions described above. Illustratively, the programming language used may include assembly language, Ada, APL, Basic, C, C++, C*, COBOL, dBase, Forth, FORTRAN, Java, Modula-2, Pascal, Prolog, Python, REXX, and/or JavaScript for example. Regression-based models (e.g., logistic regression, classification trees and random forests) are readily implemented in the R programming language without undue experimentation (using, e.g., the rpart and randomForest libraries), and neural networks may be implemented in Python or MATLAB. Further, it is not necessary that a single type of instruction or programming language be utilized in conjunction with the operation of embodiments of the invention. Rather, any number of different programming languages may be utilized as is necessary or desirable.

The server 180 may also include other removable/nonremovable, volatile/nonvolatile computer storage media. For example, a hard disk drive may read or write to nonremovable, nonvolatile magnetic media. A magnetic disk drive may read from or writes to a removable, nonvolatile magnetic disk, and an optical disk drive may read from or write to a removable, nonvolatile optical disk such as a CD-ROM or other optical media. Other removable/nonremovable, volatile/nonvolatile computer storage media that may be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The storage media are typically connected to the system bus through a removable or non-removable memory interface.

The processing units that execute commands and instructions may be general-purpose processors, but may utilize any of a wide variety of other technologies including special-purpose hardware, a microcomputer, mini-computer, mainframe computer, programmed microprocessor, microcontroller, peripheral integrated circuit element, a CSIC (customer-specific integrated circuit), ASIC (application-specific integrated circuit), a logic circuit, a digital signal processor, a programmable logic device such as an FPGA (field-programmable gate array), PLD (programmable logic device), PLA (programmable logic array), RFID processor, smart chip, or any other device or arrangement of devices that is capable of implementing the steps of the processes of the invention.

Communication may occur over the Internet, as illustrated, and/or over an intranet, extranet, Ethernet, the public telecommunications infrastructure, or any other system that provides communications. Some suitable communications protocols may include TCP/IP, UDP, or OSI for example. For wireless communications, communications protocols may include Bluetooth, Zigbee, IrDa or other suitable protocol. Furthermore, components of the system may communicate through a combination of wired or wireless paths.

The terms and expressions employed herein are used as terms and expressions of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding any equivalents of the features shown and described or portions thereof. In addition, having described certain embodiments of the invention, it will be apparent to those of ordinary skill in the art that other embodiments incorporating the concepts disclosed herein may be used without departing from the spirit and scope of the invention. Accordingly, the described embodiments are to be considered in all respects as only illustrative and not restrictive.

Claims

1. A method of providing automated personalized feedback in an interactive learning application, the method comprising:

(a) distributing an interactive educational resource over a network to a plurality of student devices each associated with a student;

(b) receiving, at a server from the student devices as the resource is used on the devices, signals indicative of actions taken on the student devices with respect to the educational resource, the actions being predictive of success with the resource;

(c) for each of the student devices, computationally determining, at the server, a likelihood of educational success with the resource for the student associated with the student device by computationally processing the received signals corresponding to each student device with a machine-learning model relating actions to educational success with material contained in the resource; and

(d) if the likelihood is less than a threshold, (i) identifying at least one deficiency in an action associated with failure of the likelihood to meet the threshold and (ii) electronically communicating a first type of notification of the deficiency to the student.

2. The method of claim 1, wherein the notification is a message appearing to originate with an instructor.

3. The method of claim 1, wherein, for each of the student devices, steps (c) and (d) occur during use of the resource on the student device.

4. The method of claim 1, wherein the actions comprise at least one of (i) a number of annotations made, (ii) a scrolling pattern, or (iii) an amount of time spent reading.

5. The method of claim 1, further comprising, for each of the student devices:

monitoring an effect of the first type of notification on rectifying the deficiency; and

if the deficiency is not subsequently rectified, subsequently sending a second type of notification different from the first type.

6. The method of claim 1, wherein the at least one deficiency comprises a plurality of deficiencies, and further comprising, for each of the student devices:

sending different types of notifications for different deficiencies;

monitoring an effect of each of the types of notification on rectifying the associated deficiencies; and

subsequently sending notifications of the type having the best effect.

7. The method of claim 1, wherein the machine-learning model comprises a regression model.

8. The method of claim 7, wherein the regression model comprises a logistic regression model.

9. An educational system comprising:

a plurality of student devices for executing an interactive educational resource received over a network, the student devices being configured to send signals indicative of actions taken on the student devices with respect to the educational resource, the actions being predictive of success with the resource;

a server in electronic communication with the student devices, the server comprising a communication module and being configured to (i) receive the signals from the student devices, (ii) for each of the student devices, computationally determine a likelihood of educational success with the material for a student associated with the student device by computationally processing the received signals corresponding to each student device with a machine-learning model relating actions to educational success with material contained in the resource, and (iii) if the likelihood is less than a threshold, identify at least one deficiency in an action associated with failure of the likelihood to meet the threshold and electronically communicate, via the communication module, a first type of notification of the deficiency to the student.

10. The system of claim 9, wherein the communication module is configured to send the notification in the form of a message appearing to originate with an instructor.

11. The system of claim 9, wherein the actions comprise at least one of (i) a number of annotations made, (ii) a scrolling pattern, or (iii) an amount of time spent reading.

12. The system of claim 9, wherein the server is further configured to, for each of the student devices:

monitor an effect of the first type of notification on rectifying the deficiency; and

if the deficiency is not subsequently rectified, subsequently send a second type of notification different from the first type.

13. The system of claim 9, wherein the server is further configured to, for each of the student devices:

send different types of notifications for different deficiencies;

monitor an effect of each of the types of notification on rectifying the associated deficiencies; and

subsequently send notifications of the type having the best effect.

14. The system of claim 9, wherein the machine-learning model comprises a regression model.

15. The system of claim 14, wherein the regression model comprises a logistic regression model.