Abstract: One embodiment provides a method, including: receiving a plurality of transcripts, each transcript corresponding to an interaction between a student and a tutoring agent; identifying conversational patterns within the plurality of transcripts, the identified conversational patterns indicating a level of understanding regarding the interaction between the tutoring agent and the student, wherein the identifying comprises (i) identifying an exchange between the tutoring agent and the student, the exchange comprising a conversational turn by each of the tutoring agent and the student and (ii) classifying each of the conversational turns into a response type; determining, based upon the identified conversational patterns, a possible improvement to the tutoring agent, wherein the possible improvement is determined based upon the classified response types of each of (i) the tutoring agent and (ii) the student; and notifying a developer of the tutoring agent of the identified possible improvement.

Abstract: One embodiment provides a method, including: receiving a plurality of transcripts, each transcript corresponding to an interaction between a student and a tutoring agent; identifying conversational patterns within the plurality of transcripts, the identified conversational patterns indicating a level of understanding regarding the interaction between the tutoring agent and the student, wherein the identifying comprises (i) identifying an exchange between the tutoring agent and the student, the exchange comprising a conversational turn by each of the tutoring agent and the student and (ii) classifying each of the conversational turns into a response type; determining, based upon the identified conversational patterns, a possible improvement to the tutoring agent, wherein the possible improvement is determined based upon the classified response types of each of (i) the tutoring agent and (ii) the student; and notifying a developer of the tutoring agent of the identified possible improvement.

Abstract: A method and system for constructive response grading for spoken language is disclosed. The method and system are computer implemented and involve a crowdsourcing step to derive evaluation features. The method includes steps for posting a speech test through an automated speech assessment tool, receiving candidate responses from candidates for the speech test; delivering the candidate responses to crowdsource volunteers; receiving crowdsourced responses from crowdsource volunteers, where the crowdsourced responses comprise a transcription of the speech test; deriving features from the transcription; and deriving a individual scores based on the features, where the individual scores are representative of pronunciation score, fluency score, content organization score and grammar score of the spoken language for each candidate.

Abstract: A method and system for constructive response grading for spoken language is disclosed. The method and system are computer implemented and involve a crowdsourcing step to derive evaluation features. The method includes steps for posting a speech test through an automated speech assessment tool, receiving candidate responses from candidates for the speech test; delivering the candidate responses to crowdsource volunteers; receiving crowdsourced responses from crowdsource volunteers, where the crowdsourced responses comprise a transcription of the speech test; deriving features from the transcription; and deriving a individual scores based on the features, where the individual scores are representative of pronunciation score, fluency score, content organization score and grammar score of the spoken language for each candidate.