Abstract: Systems and methods for automated custom training of a scoring model are disclosed herein. The method include: receiving a plurality of responses received from a plurality of students in response to providing of a prompt; identifying an evaluation model relevant to the provided prompt, which evaluation model can be a machine learning model trained to output a score relevant to at least portions of a response; generating a training indicator that provides a graphical depiction of the degree to which the identified evaluation model is trained; determining a training status of the model; receiving at least one evaluation input when the model is identified as insufficiently trained; updating training of the evaluation model based on the at least one received evaluation input; and controlling the training indicator to reflect the degree to which the evaluation model is trained subsequent to the updating of the training of the evaluation model.

Abstract: Methods, systems, and devices for dynamic response entry are disclosed herein. In some embodiments, a dynamic response entry system can include a user device that can be a proctor device or a testee device. The testee device can display a list to a testee for a predetermined time period. After the passing of the predetermined time period, the displaying of the list to the testee can be terminated. The testee can provide response to one or several questions, which responses can be input into the proctor device. The input responses can be evaluated and categorized and displayed according to the evaluation and categorization.

Abstract: Methods and systems are disclosed for interactively developing an educational course and the materials for it using a backward design, or top-down, approach based on course objectives and course outcomes in one embodiment. Educators or other course creators are able to implement this design approach through an Integrated Design and Development Interface (IDDI). The IDDI guides course development by organizing course content in a relational database. The IDDI maps a plurality of course objectives to a plurality of course outcomes to create a course model. The IDDI uses the course model to generate and/or automatically user interfaces used by a course creator to input lower-level course information.

Abstract: Systems and methods are provided by which a machine learning model may be executed to determine the probability that a given user will respond correctly to a given assessment item of a digital assessment on their first attempt. The machine learning model may process feature data corresponding to the user and the assessment item in order to determine the probability. The feature data may be calculated periodically and/or in real time or near-real time according to a machine learning model definition based on assessment data corresponding to the user's activity and/or based on responses submitted by all users to the assessment item and/or to content related to the assessment item.

Abstract: Systems and methods are described which relate to machine learning model validation. A first machine learning model may be trained to dependent variable data for a first population. A second machine learning model may be trained to simulate dependent variable data for the first population. The second machine learning model may then be applied to student activity data of a second population having different characteristics from the first population to produce simulated dependent variable data. The first machine learning model may then generate predictions for the second population, which may be validated via comparison to the simulated dependent variable data. A given simulated dependent variable value may be generated by the second machine learning model at a specific time TX, where some features input to the machine learning model may be derived from datapoints occurring before TX and others being derived from datapoints occurring after TX.

Abstract: Systems and methods of the present invention provide for: storing a plurality of content plugins; generating a graphical user interface (GUI) including components for: selecting a subset of plugins, defining a relationship between the plugins in the subset, and defining a custom pathway through the subset, including rules or conditions for navigation; receiving, from the content creator client device, selection of the subset, the relationship, and the rule or condition; generating, from the subset, relationship, and rule or condition; and transmitting to client devices for display, a learning course content for a learning application.

Abstract: Systems and methods for content aggregation creation are disclosed herein. The system can include memory having a content database and an aggregation database. The system can include a user device having a first network interface and a first I/O subsystem. The system can include a server that can: provide content to the user device via a first electrical signal; receive a selection of a portion of the provided content from the user device via a second electrical signal; automatically extract sentences from the selected portion of the provided content via a natural language processor; automatically generate a parse tree for one of the automatically extracted sentences; identify noun phrases from the part of speech tags within the parse tree; place content associated with one of the noun phrase in a content aggregation; and output the content aggregation to the user device.

Abstract: Systems and methods of the present invention provide for one or more machine learning models configured to generate one or more student outcome predictions, such as whether a student will correctly respond to items of an assessment or will pass the assessment, based on monitored student activity and associated parameters. Inputs to the machine learning models may include outputs of other predictive models, such as an item response theory model and a short term prediction model. The machine learning models may output a student outcome prediction vector containing student outcome predictions for each uncompleted item of an assessment being delivered to a student in real-time. Remediation activities may be recommended to or automatically initiated for the student based on the student outcome predictions. A delivery order of the assessment may be modified based on the student outcome predictions.

Abstract: Systems and methods for feature-based alert triggering are disclosed herein. The system can include memory including a model database containing a machine-learning algorithm. The system can include a user device that can receive inputs from a user; and at least one server. The at least one server can: receive electrical signals from the user device, the electrical signals corresponding to a plurality of user inputs provided to the user device; automatically generate input-based features from the received electrical signals; input the input-based features into the machine-learning algorithm; automatically and directly generate a risk prediction with the machine-learning algorithm from the input-based features; and generate and display an alert when the risk prediction exceeds a threshold value.

Abstract: Systems and methods for accelerated stabilization of data packet metadata are disclosed herein. The system can include a memory having a content database and a user profile database. The system can include a user device having a first network interface and a first I/O subsystem. The system can include one or more servers. The one or more servers can: retrieve data packet metadata for a data packet; determine that the data packet metadata is unstable; identify a set of potential recipients of the data packet; select one of the set of potential recipients as the recipient of the data packet; provide the data packet to the recipient of the data packet; receive a response from the recipient to the provided data packet; and automatically update the data packet metadata based on the response received from the recipient.

Abstract: Systems and methods of the present invention provide for: receiving a digital image data; modifying the digital image data to reduce a width of a feature within the digital image data; executing a dimension reduction process on the feature; storing a feature vector comprising: at least one feature for each of the received digital image data, and a correct or incorrect label associated with each feature vector; selecting the feature vector from a data store; training a classification software engine to classify each feature vector according to the label; classifying the image data as correct or incorrect according to a classification software engine; and generating an output labeling a second digital image data as correct or incorrect.

Abstract: Systems and methods of the present invention provide for: storing a correlation table including images and associated strings, and a secure password table; generating a GUI, displayed on a client computer and including GUI components for visual authentication; receiving a selection of a component; updating the GUI with a menu of images associated with the selected component; receiving a selection of one of the images; identifying an associated string as an authentication string; and storing the authentication string as a secure password in the password table.

Abstract: The present invention provides a system for determining a language proficiency of a user in an evaluated language. A machine learning engine may be trained using audio file variables from a plurality of audio files and human generated scores for a comprehensibility, accentedness and intelligibility for each audio file. The system may receive an audio file from a user and determine a plurality of audio file variables from the audio file. The system may apply the audio file variables to the machine learning engine to determine a comprehensibility, an accentedness and an intelligibility score for the user. The system may determine one or more projects and/or classes for the user based on the user's comprehensibility score, accentedness score and/or intelligibility score.

Abstract: A system and method are configured for receiving a data transmission encoding a content of a hypertext transfer protocol request and an identification of a first service, determining a first schema definition in a memory that is associated with the first service, the first schema definition including a plurality of schema items, and parsing the content of the hypertext transfer protocol request to identify a plurality of parameters and a plurality of values, each value in the plurality of values being associated with a parameter in the plurality of parameters. The system and method are configured for, for each schema item in the plurality of schema items, identifying a parameter in the content of the hypertext transfer protocol request that matches the schema item, and encoding the value associated with the parameter into a schema information object. The method includes storing the schema information object in the memory.

Abstract: Systems and methods of the present invention provide for generating and displaying a progress monitoring assistant to show the growth scale value score comparing two or more assessments and providing a preliminary interpretation of the comparison.

Abstract: Systems and methods for automatic generation of a content presentation plan are disclosed herein. The method can include receiving content identification information, retrieving objective information for the one or several objectives identified for inclusion in a content presentation plan, identifying at least one prerequisite skill for completion of at least one of the one or several objectives, generating at least one remediation question configured to delineate between users having mastery of the at least one prerequisite skill and users not having mastery of the at least one prerequisite skill, pre-selecting remedial content for providing to users identified as not having mastery of the at least one prerequisite skill, selecting objective content corresponding to the at least one objectives, and creating a content presentation plan containing the at least one remediation question, the remedial content, and the objective content.

Abstract: Systems and methods of the present invention provide for storing textbook data, a glossary, and problems within a database; identifying a problem's guided solution, and a keyword within the solution matching an entry within the glossary, from which a skill tag is associated. The disclosed system then automatically generates an assessment including an assessment problem associated with the skill. If an incorrect response is received for the assessment problem, the database is updated to associate a user that input the response with the assessment problem and a skill. The system then automatically generates a customized exercise assignment associated in the database with the skill.

Abstract: Systems and methods for automated custom training of a scoring model are disclosed herein. The method include: receiving a plurality of responses received from a plurality of students in response to providing of a prompt; identifying an evaluation model relevant to the provided prompt, which evaluation model can be a machine learning model trained to output a score relevant to at least portions of a response; generating a training indicator that provides a graphical depiction of the degree to which the identified evaluation model is trained; determining a training status of the model; receiving at least one evaluation input when the model is identified as insufficiently trained; updating training of the evaluation model based on the at least one received evaluation input; and controlling the training indicator to reflect the degree to which the evaluation model is trained subsequent to the updating of the training of the evaluation model.

Abstract: A website may track online activities, such as assignments and/or assessment, of students taking online digital courses (courses). Courseware-level data and student-level data may be extracted from the tracked online activities and as well as student registration data. Institutional-level data may be generated from data regarding the institutions that teach the courses. Teacher-level data may be generated for the teachers teaching the courses. A teacher or student may request on a website an analysis of a course. Data for the course may be weighted in the courseware-level data. Data for the student(s), institution and/or teacher may also be weighted, depending on the desired analysis. A Bayesian multi-level model may generate a plurality of posterior distributions using the collected data. A prediction of a difficult subject matter may be determined from the plurality of posterior distributions and used to select a targeted remediation that may be performed on a website.