Abstract: Managing annotation jobs is disclosed, including: receiving an annotation job, including: obtaining an input text to be annotated; and obtaining an ontology specifying a plurality of labels to be used to annotate the input text; distributing the annotation job to a plurality of annotator devices via an annotation platform; receiving a plurality of annotation results with respect to the input text from the plurality of annotator devices; and providing an aggregate annotation report corresponding to the annotation job, wherein the aggregate annotation report combines the plurality of annotation results.

Abstract: Image annotation includes: accessing initial object prediction information associated with an image, wherein the initial object prediction information includes a plurality of initial predictions associated with a plurality of objects in the image, including bounding box information associated with the plurality of objects; presenting the image and at least a portion of the initial object prediction information to be displayed; receiving adjusted object prediction information pertaining to at least some of the plurality of objects, wherein the adjusted object prediction information is obtained from a user input made via a user interface configured for a user to make annotation adjustments to at least some of the initial object prediction information; and outputting updated object prediction information, wherein the updated object prediction information is based at least in part on the adjusted object prediction information.

Abstract: A technique is disclosed for automating tracking of annotated objects and improves the throughput and efficiency of existing methods while maintaining a degree of accuracy comparable to a human annotator. In particular, the disclosed technique provides an automated annotated object tracking tool that allows machine-learning teams to annotate an object within a frame and have that annotation persist across frames as the annotated object is tracked within a series of frames, still ensuring that every frame is accurately reviewed by a human where high quality annotation is required. This technique incorporates human feedback via a user adjustment that allows the tool to adapt and improve its accuracy in tracking an annotated object across a sequence of frames.

Abstract: A set of utterances collected from a plurality of contributors is received. Semantically irrelevant utterances are removed from the set of utterances to obtain a processed set of utterances, including by applying a machine learning model to the set of utterances. An annotation user interface is provided to a plurality of human annotators to perform annotation on the processed set of utterances to obtain an annotated set of utterances. A curation user interface is provided to one or more domain experts to perform curation of the annotated set of utterances to obtain a curated set of utterances. The curated set of utterances is outputted as a training set for an automated dialogue agent.

Abstract: Image annotation includes: accessing an image and a plurality of annotation data sets for the image, wherein the plurality of annotation data sets are made by a plurality of contributors, and the image has a plurality of original image channels; aggregating the plurality of annotation data sets to obtain an aggregated annotation data set for the image; and outputting the aggregated annotation data set. Aggregating the plurality of annotation data sets to obtain an aggregated annotation data set for the image includes: generating an additional image channel based at least in part on weight averages of confidence measures of the plurality of contributors; and applying an object detection model to at least a part of the plurality of original image channels and at least a part of the additional image channel to generate the aggregated annotation data set.

Abstract: Managing annotation jobs is disclosed, including: receiving an annotation job, including: obtaining an input text to be annotated; and obtaining an ontology specifying a plurality of labels to be used to annotate the input text; distributing the annotation job to a plurality of annotator devices via an annotation platform; receiving a plurality of annotation results with respect to the input text from the plurality of annotator devices; and providing an aggregate annotation report corresponding to the annotation job, wherein the aggregate annotation report combines the plurality of annotation results.

Abstract: A technique is disclosed for automating tracking of annotated objects and improves the throughput and efficiency of existing methods while maintaining a degree of accuracy comparable to a human annotator. In particular, the disclosed technique provides an automated annotated object tracking tool that allows machine-learning teams to annotate an object within a frame and have that annotation persist across frames as the annotated object is tracked within a series of frames, still ensuring that every frame is accurately reviewed by a human where high quality annotation is required. This technique incorporates human feedback via a user adjustment that allows the tool to adapt and improve its accuracy in tracking an annotated object across a sequence of frames.

Abstract: Image annotation includes: accessing initial object prediction information associated with an image, wherein the initial object prediction information includes a plurality of initial predictions associated with a plurality of objects in the image, including bounding box information associated with the plurality of objects; presenting the image and at least a portion of the initial object prediction information to be displayed; receiving adjusted object prediction information pertaining to at least some of the plurality of objects, wherein the adjusted object prediction information is obtained from a user input made via a user interface configured for a user to make annotation adjustments to at least some of the initial object prediction information; and outputting updated object prediction information, wherein the updated object prediction information is based at least in part on the adjusted object prediction information.

Abstract: Image annotation includes: accessing an image and a plurality of annotation data sets for the image, wherein the plurality of annotation data sets are made by a plurality of contributors, and the image has a plurality of original image channels; aggregating the plurality of annotation data sets to obtain an aggregated annotation data set for the image; and outputting the aggregated annotation data set. Aggregating the plurality of annotation data sets to obtain an aggregated annotation data set for the image includes: generating an additional image channel based at least in part on weight averages of confidence measures of the plurality of contributors; and applying an object detection model to at least a part of the plurality of original image channels and at least a part of the additional image channel to generate the aggregated annotation data set.

Abstract: A technique is disclosed for automating tracking of annotated objects and improves the throughput and efficiency of existing methods while maintaining a degree of accuracy comparable to a human annotator. In particular, the disclosed technique provides an automated annotated object tracking tool that allows machine-learning teams to annotate an object within a frame and have that annotation persist across frames as the annotated object is tracked within a series of frames, still ensuring that every frame is accurately reviewed by a human where high quality annotation is required. This technique incorporates human feedback via a user adjustment that allows the tool to adapt and improve its accuracy in tracking an annotated object across a sequence of frames.

Abstract: A technique is disclosed for automating tracking of annotated objects and improves the throughput and efficiency of existing methods while maintaining a degree of accuracy comparable to a human annotator. In particular, the disclosed technique provides an automated annotated object tracking tool that allows machine-learning teams to annotate an object within a frame and have that annotation persist across frames as the annotated object is tracked within a series of frames, still ensuring that every frame is accurately reviewed by a human where high quality annotation is required. This technique incorporates human feedback via a user adjustment that allows the tool to adapt and improve its accuracy in tracking an annotated object across a sequence of frames.