Abstract: Reading comprehension of a user can be assessed by presenting, in a graphical user interface, sequential reading text comprising a plurality of passages. The graphical user interface can alternate between (i) automatically advancing through passages of the reading text and (ii) manually advancing through passages of the reading text within the graphical user interface which is in response to user-generated input received via the graphical user interface. An audio narration is provided during the automatic advancing of the reading text. An audio file is recorded during the manual advancing of the reading text which is used to automatically determine an estimated level of reading comprehension of the user. Data characterizing the determined level of reading comprehension of the user can then be provided (e.g., displayed, loaded into memory, stored on a hard drive, transmitted to a remote computing system, etc.). Related apparatus, systems, techniques and articles are also described.

Abstract: Systems and methods are provided for the detection of sentiment in writing. A plurality of texts is received from a larger collection of writing samples with a computer system. A set of seed words from the plurality of texts are labeled as being of positive sentiment or of negative sentiment with the computer system. The set of seed words is expanded in size with the computer system to provide an expanded set of seed words. Intensity values are assigned to words of the expanded set of seed words. Each of the words of the expanded set of seed words is assigned three intensity values: a value corresponding to the strength of the word's association with a positive polarity class, a value corresponding to the strength of the word's association with a negative polarity class, and a value corresponding to the strength of the word's association with a neutral polarity class.

Abstract: Methods and systems for scoring an argument critique written essay, including identifying a discourse segment of the argument critique written essay, determining a position of each sentence in the discourse segment, classifying sentences into discourse units, evaluating an argumentative content of each sentence, and assigning an argumentative score to the essay based on the argumentative content of each sentence in the discourse segment of that essay. Methods and systems for training the scoring method are also disclosed. Corresponding apparatuses, systems, and methods are also disclosed.

Abstract: Systems and methods of improving reading fluency by presenting a virtual reading partner that takes turns reading aloud with a reader, are presented. First audio data comprising speech corresponding to the reader reading aloud from a first portion of a written transcript is received. The first audio data is processed to identify spoken words from the speech. The reader's speaking of the words in the first portion of the written transcript are compared to correct speaking of words in the first portion of the written transcript. A reading fluency of the reader is assessed. Second audio data comprising a pre-recorded narration of a second portion of the written transcript or automatically generated text to speech of the second portion is generated. The virtual reading partner is presented to the reader and takes turns reading aloud with the reader until the end of the written transcript has been reached.

Abstract: Computer-implemented systems and methods for generating a personalization score for a constructed response are provided. A constructed response generated by a user is received. The constructed response is processed with a processing system to generate one or more feature values representative of one or more aspects of the constructed response. A personalization score for the constructed response is generated using the processing system by applying a computer scoring model to the one or more feature values. The personalization score provides a measure of a personalization of the constructed response.

Abstract: Systems and methods are provided for determining whether an examination prompt elicits responses of a type for which the examination prompt is designed. Word types for each of a plurality of words in a plurality of responses to the examination prompt are identified. An evaluation distribution of the identified word types is determined. Predetermined distributions stored on a computer-readable medium associated with a plurality of different response types are accessed, and the evaluation distribution is compared to each of the predetermined distributions to determine which predetermined distribution the evaluation distribution is most similar. A determination is made as to whether the most similar distribution is associated with a same response type as the type for which the examination prompt is designed. An indication is made as to whether the examination prompt is deemed to elicit responses of the type for which the examination prompt is designed.

Abstract: Methods and systems for scoring written text based on use of idiomatic expressions, including reading pre-selected idiomatic expressions in a canonical form into memory, expanding idiomatic expressions from the canonical form, reading a written response into the memory, pre-processing the written response, searching the pre-processed written response for idiomatic expressions, and assigning a score to the written response. The score may be based at least in part on the number of idiomatic expressions in the written response. Corresponding apparatuses, systems, and methods are also disclosed.

Abstract: Systems and methods are provided for a computer-implemented method for identifying pairs of cohesive words within a text. A supervised model is trained to detect cohesive words within a text to be scored. Training the supervised model includes identifying a plurality of pairs of candidate cohesive words in a training essay and an order associated with the pairs of candidate cohesive words based on an order of words in the training essay. The pairs of candidate cohesive words are filtered to form a set of evaluation pairs. The evaluation pairs are provided via a graphical user interface based on the order associated with the pairs of candidate cohesive words. An indication of cohesion or no cohesion is received for the evaluation pairs via the graphical user interface. The supervised model is trained based on the evaluation pairs and the received indications.

Abstract: Computer-implemented systems and methods are provided for determining a document's complexity. For example, a computer performing the complexity analysis can receive a document. The computer can determine the content words within the document and determine an association measure for each group of content words. An association profile can be created for the document using the association measures. The computer can use the association profile to determine the complexity of the document. The complexity of the document may correspond to the document's suitable reading level or, if the document is an essay, an essay score.

Abstract: Systems and methods described herein utilize supervised machine learning to generate a figure-of-speech prediction model for classify content words in a running text as either being figurative (e.g., as a metaphor, simile, etc.) or non-figurative (i.e., literal). The prediction model may extract and analyze any number of features in making its prediction, including a topic model feature, unigram feature, part-of-speech feature, concreteness feature, concreteness difference feature, literal context feature, non-literal context feature, and off-topic feature, each of which are described in detail herein. Since uses of figure of speech in writings may signal content sophistication, the figure-of-speech prediction model allows scoring engines to further take into consideration a text's use of figure of speech when generating a score.

Abstract: Systems and methods are provided for identifying one or more target words of a corpus that have a lexical relationship to a plurality of provided cue words. The cue words and statistical lexical information derived from a corpus of documents are analyzed to determine candidate words that have a lexical association with the cue words. The statistical information includes numerical values indicative of probabilities of word pairs appearing together as adjacent words in a well-formed text or appearing together within a paragraph of a well-formed text. For each candidate word, a statistical association score between the candidate word and each of the cue words is determined. An aggregate score for each of the candidate words is determined based on the statistical association scores. One or more of the candidate words are selected to be the one or more target words based on the aggregate scores.

Abstract: Systems and methods are provided for a computer-implemented method for identifying pairs of cohesive words within a text. A supervised model is trained to detect cohesive words within a text to be scored. Training the supervised model includes identifying a plurality of pairs of candidate cohesive words in a training essay and an order associated with the pairs of candidate cohesive words based on an order of words in the training essay. The pairs of candidate cohesive words are filtered to form a set of evaluation pairs. The evaluation pairs are provided via a graphical user interface based on the order associated with the pairs of candidate cohesive words. An indication of cohesion or no cohesion is received for the evaluation pairs via the graphical user interface. The supervised model is trained based on the evaluation pairs and the received indications.

Abstract: Systems and methods are provided for identifying one or more target words of a corpus that have a lexical relationship to a plurality of provided cue words. The cue words and statistical lexical information derived from a corpus of documents are analyzed to determine candidate words that have a lexical association with the cue words. The statistical information includes numerical values indicative of probabilities of word pairs appearing together as adjacent words in a well-formed text or appearing together within a paragraph of a well-formed text. For each candidate word, a statistical association score between the candidate word and each of the cue words is determined. An aggregate score for each of the candidate words is determined based on the statistical association scores. One or more of the candidate words are selected to be the one or more target words based on the aggregate scores.

Abstract: Systems and methods are provided for determining whether an examination prompt elicits responses of a type for which the examination prompt is designed. Word types for each of a plurality of words in a plurality of responses to the examination prompt are identified. An evaluation distribution of the identified word types is determined. Predetermined distributions stored on a computer-readable medium associated with a plurality of different response types are accessed, and the evaluation distribution is compared to each of the predetermined distributions to determine which predetermined distribution the evaluation distribution is most similar. A determination is made as to whether the most similar distribution is associated with a same response type as the type for which the examination prompt is designed. An indication is made as to whether the examination prompt is deemed to elicit responses of the type for which the examination prompt is designed.

Abstract: Systems and methods described herein utilize supervised machine learning to generate a figure-of-speech prediction model for classify content words in a running text as either being figurative (e.g., as a metaphor, simile, etc.) or non-figurative (i.e., literal). The prediction model may extract and analyze any number of features in making its prediction, including a topic model feature, unigram feature, part-of-speech feature, concreteness feature, concreteness difference feature, literal context feature, non-literal context feature, and off-topic feature, each of which are described in detail herein. Since uses of figure of speech in writings may signal content sophistication, the figure-of-speech prediction model allows scoring engines to further take into consideration a text's use of figure of speech when generating a score.

Abstract: Systems and methods are provided for a computer-implemented method of providing a score that measures an essay's usage of source material provided in at least one written text and an audio recording. Using one or more data processors, a determination is made of a list of n-grams present in a received essay. For each of a plurality of present n-grams, an n-gram weight is determined, where the n-gram weight is based on a number of appearances of that n-gram in the at least one written text and a number of appearances of that n-gram in the audio recording, and an n-gram sub-metric is determined based on the presence of the n-gram in the essay and the n-gram weight. A source usage metric is determined based on the n-gram sub-metrics for the plurality of present n-grams, and a scoring model is used to generate a score for the essay based on the source usage metric.

Abstract: Computer-implemented systems and methods are provided for determining a document's complexity. For example, a computer performing the complexity analysis can receive a document. The computer can determine the content words within the document and determine an association measure for each group of content words. An association profile can be created for the document using the association measures. The computer can use the association profile to determine the complexity of the document. The complexity of the document may correspond to the document's suitable reading level or, if the document is an essay, an essay score.

Abstract: Systems and methods are provided for identifying factual information in a written document. Named entities and corresponding noun phrases are identified in the written document. A query is built by combining one of the named entities with a respective one of the noun phrases. The query represents an assertion of a potential fact. The query is submitted for comparison with a fact repository which assesses whether the query presents a factual assertion. If the query presents a factual assertion (e.g., it matches a fact within the fact repository), a match is returned. Various modifications may be made to the queries to return additional matches and various combinations of filters may be applied to the matches to filter out less relevant matches.