Abstract: A network-accessible toolbox permits collaborative annotation, facilitating interaction and communication among users. In an implementation, content on a Web page, or other page or document accessible through a network or the Internet, can be annotated. A user can view the page and with the toolbox, select content on the page and make an annotation or suggestion. Multiple users can view and make annotations to the same page at the same time. After annotations are made, users can view the page with annotations made by other users. The author or owner of the page can accept or reject the annotations or suggestions made to the page. If the owner accepts a particular suggestion, the selected content is replaced with the suggestion.

Abstract: Systems and methods of the present invention may be used to determine metrics and health scores for content that may correspond to an educational course or textbook, which may be in a digital format. The metrics and health scores may be determined at various hierarchical content levels, and may be used to quantitatively assess how well the corresponding content is performing based on responses submitted to assessment item parts of the content by responders. The metrics may include difficulty and discrimination metrics, which may be determined using maximum likelihood estimation methods based on a modified two parameter item response model. A content analytics interface corresponding to a given content element may be generated and displayed via a user device, and may include content health scores of subcontent within that content element. The subcontent may be ordered according to content health score.

Abstract: Systems and methods for automated custom training of a scoring model are disclosed herein. The method include: receiving a plurality of responses received from a plurality of students in response to providing of a prompt; identifying an evaluation model relevant to the provided prompt, which evaluation model can be a machine learning model trained to output a score relevant to at least portions of a response; generating a training indicator that provides a graphical depiction of the degree to which the identified evaluation model is trained; determining a training status of the model; receiving at least one evaluation input when the model is identified as insufficiently trained; updating training of the evaluation model based on the at least one received evaluation input; and controlling the training indicator to reflect the degree to which the evaluation model is trained subsequent to the updating of the training of the evaluation model.

Abstract: Systems and methods of the present invention provide for: generating a GUI comprising survey questions associated with degree factors and associated rating GUI components indicating application of the factor to a user; receiving the factor rating for each survey question; identifying a high factor rating exceeding a threshold; selecting a degree identifier sharing a common high factor rating between the first response and a response stored in the database; generating a candidate degree list including the degree identifier; generating a second GUI including the candidate degree list; and transmitting the second GUI to a client device.

Abstract: Systems and methods of the present invention provide for estimating latent ability of responders to a digital assessment in the form of ability scores and estimating item parameters an assessment item of the digital assessment including difficulty scores and discrimination scores. Maximum likelihood estimation may be performed based on an item response theory model to estimate the item parameters. Supervisory, extraction, and worker modules of a workflow manager module may initiate general purpose graphics processing unit instances and cause these instances to perform the maximum likelihood estimation calculations. The item response theory model may be a two parameter model that is modified to account for changes in difficulty caused by the use of hints.

Abstract: A multi-country data pipeline keeps all of the PII received from a user that is in a first country in the first country. The data pipeline allows the non-personal data received from the user to be transmitted and analyzed in a second country. The method further allows the results of the analysis in the second country to be transmitted back to the first country where the PII is added to the results of the analysis. The data pipeline allows the results of the analysis in the second country to be used to take a desired action for the user in the first country, all while the PII of the user never leaves the first country.

Abstract: Systems and methods for automated custom training of a scoring model are disclosed herein. The method include: receiving a plurality of responses received from a plurality of students in response to providing of a prompt; identifying an evaluation model relevant to the provided prompt, which evaluation model can be a machine learning model trained to output a score relevant to at least portions of a response; generating a training indicator that provides a graphical depiction of the degree to which the identified evaluation model is trained; determining a training status of the model; receiving at least one evaluation input when the model is identified as insufficiently trained; updating training of the evaluation model based on the at least one received evaluation input; and controlling the training indicator to reflect the degree to which the evaluation model is trained subsequent to the updating of the training of the evaluation model.

Abstract: Systems and methods for automatic content remediation notification are disclosed herein. The system can include memory that can contain a content library database. The system can include a first user device and one or more servers. The one or more servers can: receive a content aggregation creation request from the first user device; identify content information associated with a set of the plurality of data packets; apply a filter request to the set of the plurality of data packets; automatically provide information relating to data packets in the restricted set of data packets to the first user device; receive content aggregate information identifying a content aggregate from the first user device; evaluate the content aggregate according to the metadata associated with the data packets of the content aggregate; and output an indicator of the evaluation result to the first user device.

Abstract: Systems and methods for secure content delivery are disclosed herein. The system can include a content driver communicatingly connected with a user device via a layered protocol model or via a User Datagram Protocol (UDP). The content driver can generate a signal directing the creation of a secured partition on a bootable media device connected to a user device, identify content for delivery, and generate pixel data for the content. The content driver can send the pixel data to the user device, wherein the user device is configured to store the pixel data in the secured partition of the bootable media device. The content driver can receive a plurality of response inputs from the user device, wherein the response inputs are generated by a software application running on the bootable media device, generate a response based on the received response inputs, and provide the generated response to an evaluation module.

Abstract: The present invention provides for one or more server computers configured to receive user input from a user interface (UI) comprising a first and second job description. The server queries a database to identify a first and second competency score associated with the first and second job descriptions respectively. The server then generates a personalized course profile comprising a plurality of objectives stored in the database and each associated with a third competency score between the first and second competency scores. The server then renders a second UI including an ordered list of the objectives and UI controls for accessing assets associated with the objectives. The server then transmits the second UI to a client computer for display.

Abstract: A system including at least one processor configured to: store learning resources and learning outcome frameworks in a database, which are then selected; map sections of learning resources to associated objectives within learning outcome frameworks; generate a GUI, including access to learning resources and frameworks; receive selection of a learning framework and a learning resource; and in response, generate a framework menu, selectable learning objectives, and an associated resource content, respectively.

Abstract: A system and method is configured to determine an initial assessment score for an initial curriculum activity for a learner. A query identifying, within a data logic or at least one database record, a lower threshold value is executed. Responsive to determining that the initial assessment score is less than the lower threshold value, a query is executed identifying a remediation curriculum activity associated with the initial curriculum activity and a first graphical user interface is encoded including an identification of the learner and a first user interface element configured to cause the server hardware computing device to modify a lesson plan of the learner to include the remediation curriculum activity. The first graphical user interface is transmitted to a client hardware computing device for display.

Abstract: Systems and methods for automated evaluation system routing are described herein. The system can include a memory, which can include a model database and a correlation database. The system can include a first user device and a second user device. The system can include at least one server. The at least one server can: receive a response communication from the user device; generate an initial evaluation value according to an AI model; determine a correlation between the initial evaluation value and evaluation range data; accept the initial evaluation value when the correlation exceeds a threshold value; and route the response communication to the second user device for generation of an elevated evaluation value when the correlation does not exceed the threshold value.

Abstract: Systems and methods for automated custom training of a scoring model are disclosed herein. The method include: receiving a plurality of responses received from a plurality of students in response to providing of a prompt; identifying an evaluation model relevant to the provided prompt, which evaluation model can be a machine learning model trained to output a score relevant to at least portions of a response; generating a training indicator that provides a graphical depiction of the degree to which the identified evaluation model is trained; determining a training status of the model; receiving at least one evaluation input when the model is identified as insufficiently trained; updating training of the evaluation model based on the at least one received evaluation input; and controlling the training indicator to reflect the degree to which the evaluation model is trained subsequent to the updating of the training of the evaluation model.

Abstract: Methods, systems, and devices for dynamic response entry are disclosed herein. In some embodiments, a dynamic response entry system can include a user device that can be a proctor device or a testee device. The testee device can display a list to a testee for a predetermined time period. After the passing of the predetermined time period, the displaying of the list to the testee can be terminated. The testee can provide response to one or several questions, which responses can be input into the proctor device. The input responses can be evaluated and categorized and displayed according to the evaluation and categorization.

Abstract: Methods and systems are disclosed for interactively developing an educational course and the materials for it using a backward design, or top-down, approach based on course objectives and course outcomes in one embodiment. Educators or other course creators are able to implement this design approach through an Integrated Design and Development Interface (IDDI). The IDDI guides course development by organizing course content in a relational database. The IDDI maps a plurality of course objectives to a plurality of course outcomes to create a course model. The IDDI uses the course model to generate and/or automatically user interfaces used by a course creator to input lower-level course information.

Abstract: Systems and methods are provided by which a machine learning model may be executed to determine the probability that a given user will respond correctly to a given assessment item of a digital assessment on their first attempt. The machine learning model may process feature data corresponding to the user and the assessment item in order to determine the probability. The feature data may be calculated periodically and/or in real time or near-real time according to a machine learning model definition based on assessment data corresponding to the user's activity and/or based on responses submitted by all users to the assessment item and/or to content related to the assessment item.

Abstract: Systems and methods are described which relate to machine learning model validation. A first machine learning model may be trained to dependent variable data for a first population. A second machine learning model may be trained to simulate dependent variable data for the first population. The second machine learning model may then be applied to student activity data of a second population having different characteristics from the first population to produce simulated dependent variable data. The first machine learning model may then generate predictions for the second population, which may be validated via comparison to the simulated dependent variable data. A given simulated dependent variable value may be generated by the second machine learning model at a specific time TX, where some features input to the machine learning model may be derived from datapoints occurring before TX and others being derived from datapoints occurring after TX.

Abstract: Systems and methods of the present invention provide for: storing a plurality of content plugins; generating a graphical user interface (GUI) including components for: selecting a subset of plugins, defining a relationship between the plugins in the subset, and defining a custom pathway through the subset, including rules or conditions for navigation; receiving, from the content creator client device, selection of the subset, the relationship, and the rule or condition; generating, from the subset, relationship, and rule or condition; and transmitting to client devices for display, a learning course content for a learning application.

Abstract: Systems and methods for content aggregation creation are disclosed herein. The system can include memory having a content database and an aggregation database. The system can include a user device having a first network interface and a first I/O subsystem. The system can include a server that can: provide content to the user device via a first electrical signal; receive a selection of a portion of the provided content from the user device via a second electrical signal; automatically extract sentences from the selected portion of the provided content via a natural language processor; automatically generate a parse tree for one of the automatically extracted sentences; identify noun phrases from the part of speech tags within the parse tree; place content associated with one of the noun phrase in a content aggregation; and output the content aggregation to the user device.