Abstract: Systems, apparatuses, methods, and computer program products are disclosed for automatically determining accuracy of entity recognition of text. An example method includes segmenting a service entity recognition analysis of the text and a gold entity recognition analysis of the text into common superstrings that define entity spans. The example method further includes classifying each of the entity spans based on an accuracy of entity recognition in the service analysis of the text corresponding to the entity spans using a classification system that differentiates accurately identified but improperly bounded entities into at least three subcategories to obtain an entity accuracy classification. The example method also includes obtaining a score report based on the entity accuracy classification. The example method additionally includes performing an action set based on the entity accuracy classification.

Abstract: In general, this disclosure describes techniques for generating and evaluating automatic transcripts of audio recordings containing human speech. In some examples, a computing system is configured to: generate transcripts of a plurality of audio recordings; determine an error rate for each transcript by comparing the transcript to a reference transcript of the audio recording; receive, for each transcript, a subjective ranking selected from a plurality of subjective rank categories; determine, based on the error rates and subjective rankings, objective rank categories defined by error-rate ranges; and assign an objective ranking to a new machine-generated transcript of a new audio recording, based on the objective rank categories and an error rate of the new machine-generated transcript.

Abstract: Systems, apparatuses, methods, and computer program products are disclosed for automatically determining accuracy of entity recognition of text. An example method includes segmenting a service entity recognition analysis of the text and a gold entity recognition analysis of the text into common superstrings that define entity spans. The example method further includes classifying each of the entity spans based on an accuracy of entity recognition in the service analysis of the text corresponding to the entity spans using a classification system that differentiates accurately identified but improperly bounded entities into at least three subcategories to obtain an entity accuracy classification. The example method also includes obtaining a score report based on the entity accuracy classification. The example method additionally includes performing an action set based on the entity accuracy classification.