Abstract: A method and system pre-trains a convolutional neural network for image recognition based upon masked language modeling by inputting, to the convolutional neural network, an image; outputting, from the convolutional neural network, a visual embedding tensor of visual embedding vectors; tokenizing a caption to create a list of tokens, at least one token having visual correspondence to the image received by the convolutional neural network; randomly selecting one of the tokens in the list of tokens to be masked, the selected token being taken as ground truth; computing, using a language model neural network, hidden representations of the tokens; using the hidden representation of the masked token, as a query vector, to pool the visual embedding vectors in the visual embedding tensor, attentively; predicting the masked token by mapping the pooled visual embedding vectors to the tokens; determining a prediction loss associated with the masked token; and back-propagating the prediction loss to the convolutional ne

Abstract: A method and system pre-trains a convolutional neural network for image recognition based upon masked language modeling by inputting, to the convolutional neural network, an image; outputting, from the convolutional neural network, a visual embedding tensor of visual embedding vectors; tokenizing a caption to create a list of tokens, at least one token having visual correspondence to the image received by the convolutional neural network; randomly selecting one of the tokens in the list of tokens to be masked, the selected token being taken as ground truth; computing, using a language model neural network, hidden representations of the tokens; using the hidden representation of the masked token, as a query vector, to pool the visual embedding vectors in the visual embedding tensor, attentively; predicting the masked token by mapping the pooled visual embedding vectors to the tokens; determining a prediction loss associated with the masked token; and back-propagating the prediction loss to the convolutional ne

Abstract: A method for re-identification of a subject in an image by pre-training a convolutional neural network to recognize individuals within a closed set of possible identifications and further pre-training the convolutional neural network using classification loss; training the pre-trained convolutional neural network by sequentially processing a plurality of triplet of images, each triplet containing a query image degraded by adding random noise to a region of the query image, a positive image corresponding to an image of a same subject as in the query image, and a negative image corresponding to an image of a different subject as in the query image by (a) ranking the triplets by the triplet loss computed, (b) selecting a subset of triplets among the plurality of triplets, and (c) retraining the pre-trained convolutional neural network on each of the triplets of the subset of triplets.

Abstract: A method for re-identification of a subject in an image from a set of images, by pre-training, by a data processor in a first server, a convolutional neural network with ImageNet to recognize individuals within a closed set of possible identifications and further pre-training the convolutional neural network using classification loss to realize person identification; training, by the data processor in the first server, the pre-trained convolutional neural network by sequentially processing a plurality of triplet of images and allowing a different size input for each image, each triplet containing a query image degraded by adding random noise to a region of the query image, a positive image corresponding to an image of a same subject as in the query image, and a negative image corresponding to an image of a different subject as in the query image by (a) ranking the triplets by the triplet loss computed (b) selecting a subset of triplets among the plurality of triplets, the subset of triplets having the largest

Abstract: In a method for detecting an object in an input image, an input image vector representing the input image is generated by performing a regional maximum activations of convolutions (R-MAC) using a convolutional neural network (CNN) applied to the input image and using regions for the R-MAC defined by applying a region proposal network (RPN) to the output of the CNN applied to the input image. Likewise, a reference image vector representing a reference image depicting the object is generated by performing the R-MAC using the CNN applied to the reference image and using regions for the R MAC defined by applying the RPN to the output of the CNN applied to the reference image. A similarity metric between the input image vector and the reference image vector is computed, and the object is detected as present in the input image if the similarity metric satisfies a detection criterion.

Abstract: Similar images are identified by semantically matching human-supplied text captions accompanying training images. An image representation function is trained to produce similar vectors for similar images according to this similarity. The trained function is applied to non-training second images in a different database to produce second vectors. This trained function does not require the second images to contain captions. A query image is matched to the second images by applying the trained function to the query image to produce a query vector, and the second images are ranked based on how closely the second vectors match the query vector, and the top ranking ones of the second images are output as a response to the query image.

Abstract: In a method for detecting an object in an input image, an input image vector representing the input image is generated by performing a regional maximum activations of convolutions (R-MAC) using a convolutional neural network (CNN) applied to the input image and using regions for the R-MAC defined by applying a region proposal network (RPN) to the output of the CNN applied to the input image. Likewise, a reference image vector representing a reference image depicting the object is generated by performing the R-MAC using the CNN applied to the reference image and using regions for the R MAC defined by applying the RPN to the output of the CNN applied to the reference image. A similarity metric between the input image vector and the reference image vector is computed, and the object is detected as present in the input image if the similarity metric satisfies a detection criterion.

Abstract: A method for generating object and part detectors includes accessing a collection of training images. The collection of training images includes images annotated with an object label and images annotated with a respective part label for each of a plurality of parts of the object. Joint appearance-geometric embeddings for regions of a set of the training images are generated. At least one detector for the object and its parts is learnt using annotations of the training images and respective joint appearance-geometric embeddings, e.g., using multi-instance learning for generating parameters of scoring functions which are used to identify high scoring regions for learning the object and its parts. The detectors may be output or used to label regions of a new image with object and part labels.

Abstract: An object detection method includes for each of a set of patches of an image, encoding features of the patch with a non-linear mapping function, and computing per-patch statistics based on the encoded features for approximating a window-level non-linear operation by a patch-level operation. Then, windows are extracted from the image, each window comprising a sub-set of the set of patches. Each of the windows is scored based on the computed patch statistics of the respective sub-set of patches. Objects, if any, can then be detected in the image, based on the window scores. The method and system allow the non-linear operations to be performed only at the patch level, reducing the computation time of the method, since there are generally many more windows than patches, while not impacting performance unduly, as compared to a system which performs non-linear operations at the window level.

Abstract: A system and method for evaluating iconicity of an image are provided. In the method, at least one test image is received, each test image including an object in a selected class. Properties related to iconicity are computed for each test image. The properties may include one or more of: a) a direct measure of iconicity, which is computed with a direct iconicity prediction model which has been learned on a set of training images, each training image labeled with an iconicity score; b) one or more class-independent properties; and c) one or more class-dependent properties. A measure of iconicity of each of the test images is computed, based on the computed properties. By combining a set of complementary properties, an iconicity measure which shows good agreement with human evaluations of iconicity can be obtained.

Abstract: A system and method predict the translation quality of a translated input document. The method includes receiving an input document pair composed of a plurality of sentence pairs, each sentence pair including a source sentence in a source language and a machine translation of the source language sentence to a target language sentence. For each of the sentence pairs, a representation of the sentence pair is generated, based on a set of features extracted for the sentence pair. Using a generative model, a representation of the input document pair is generated, based on the sentence pair representations. A translation quality of the translated input document is computed, based on the representation of the input document pair.

Abstract: In an image classification method, a feature vector representing an input image is generated by unsupervised operations including extracting local descriptors from patches distributed over the input image, and a classification value for the input image is generated by applying a neural network (NN) to the feature vector. Extracting the feature vector may include encoding the local descriptors extracted from each patch using a generative model, such as Fisher vector encoding, aggregating the encoded local descriptors to form a vector, projecting the vector into a space of lower dimensionality, for example using Principal Component Analysis (PCA), and normalizing the feature vector of lower dimensionality to produce the feature vector representing the input image. A set of mid-level features representing the input image may be generated as the output of an intermediate layer of the NN.

Abstract: In an image classification method, a feature vector representing an input image is generated by unsupervised operations including extracting local descriptors from patches distributed over the input image, and a classification value for the input image is generated by applying a neural network (NN) to the feature vector. Extracting the feature vector may include encoding the local descriptors extracted from each patch using a generative model, such as Fisher vector encoding, aggregating the encoded local descriptors to form a vector, projecting the vector into a space of lower dimensionality, for example using Principal Component Analysis (PCA), and normalizing the feature vector of lower dimensionality to produce the feature vector representing the input image. A set of mid-level features representing the input image may be generated as the output of an intermediate layer of the NN.

Abstract: A system and method for object instance localization in an image are disclosed. In the method, keypoints are detected in a target image and candidate regions are detected by matching the detected keypoints to keypoints detected in a set of reference images. Similarity measures between global descriptors computed for the located candidate regions and global descriptors for the reference images are computed and labels are assigned to at least some of the candidate regions based on the computed similarity measures. Performing the region detection based on keypoint matching while performing the labeling based on global descriptors improves object instance detection.

Abstract: A system and method for object instance localization in an image are disclosed. In the method, keypoints are detected in a target image and candidate regions are detected by matching the detected keypoints to keypoints detected in a set of reference images. Similarity measures between global descriptors computed for the located candidate regions and global descriptors for the reference images are computed and labels are assigned to at least some of the candidate regions based on the computed similarity measures. Performing the region detection based on keypoint matching while performing the labeling based on global descriptors improves object instance detection.

Abstract: A system and method predict the translation quality of a translated input document. The method includes receiving an input document pair composed of a plurality of sentence pairs, each sentence pair including a source sentence in a source language and a machine translation of the source language sentence to a target language sentence. For each of the sentence pairs, a representation of the sentence pair is generated, based on a set of features extracted for the sentence pair. Using a generative model, a representation of the input document pair is generated, based on the sentence pair representations. A translation quality of the translated input document is computed, based on the representation of the input document pair.

Abstract: An object detection method includes for each of a set of patches of an image, encoding features of the patch with a non-linear mapping function, and computing per-patch statistics based on the encoded features for approximating a window-level non-linear operation by a patch-level operation. Then, windows are extracted from the image, each window comprising a sub-set of the set of patches. Each of the windows is scored based on the computed patch statistics of the respective sub-set of patches. Objects, if any, can then be detected in the image, based on the window scores. The method and system allow the non-linear operations to be performed only at the patch level, reducing the computation time of the method, since there are generally many more windows than patches, while not impacting performance unduly, as compared to a system which performs non-linear operations at the window level.

Abstract: A system and method for evaluating iconicity of an image are provided. In the method, at least one test image is received, each test image including an object in a selected class. Properties related to iconicity are computed for each test image. The properties may include one or more of: a) a direct measure of iconicity, which is computed with a direct iconicity prediction model which has been learned on a set of training images, each training image labeled with an iconicity score; b) one or more class-independent properties; and c) one or more class-dependent properties. A measure of iconicity of each of the test images is computed, based on the computed properties. By combining a set of complementary properties, an iconicity measure which shows good agreement with human evaluations of iconicity can be obtained.

Abstract: A computer implemented method for localization of an object, such as a license plate, in an input image includes generating a task-dependent representation of the input image based on relevance scores for the object to be localized. The relevance scores are output by a classifier for a plurality of locations in the input image, such as patches. The classifier is trained on patches extracted from training images and their respective relevance labels. One or more similar images are identified from a set of images, based on a comparison of the task-dependent representation of the input image and task-dependent representations of images in the set of images. A location of the object in the input image is identified based on object location annotations for the similar images.

Abstract: A system and method for generating gold questions for labeling tasks are disclosed. The method includes sampling a positive class from a predefined set of classes to be used in labeling documents, based on a computed measure of class popularity. A set of negative classes is identified from the set of classes based on a distance measure between the positive class and other classes in the set of classes. A gold question is generated which includes a document representative of the positive class and a set of candidate answers. The candidate answers include a label for the positive class and a label for each of the negative classes in the identified set of negative classes. A task may be generated which includes the gold question and a plurality of standard questions which each include a document to be labeled. A computer processor may implement all or part of the method.