Abstract: A computer-implemented method for training a deep learning network is presented. The method includes receiving a first image and a second image, mining exemplar thin-plate spline (TPS) to determine transformations for generating point correspondences between the first and second images, using artificial point correspondences to train the deep neural network, learning and using the TPS transformation output through a spatial transformer, and applying heuristics for selecting an acceptable set of images to match for accurate reconstruction. The deep learning network learns to warp points in the first image to points in the second image.

Abstract: A computer-implemented method for training a convolutional neural network (CNN) is presented. The method includes extracting coordinates of corresponding points in the first and second locations, identifying positive points in the first and second locations, identifying negative points in the first and second locations, training features that correspond to positive points of the first and second locations to move closer to each other, and training features that correspond to negative points in the first and second locations to move away from each other.

Abstract: A system is provided for unsupervised cross-domain image generation relative to a first and second image domain that each include real images. A first generator generates synthetic images similar to real images in the second domain while including a semantic content of real images in the first domain. A second generator generates synthetic images similar to real images in the first domain while including a semantic content of real images in the second domain. A first discriminator discriminates real images in the first domain against synthetic images generated by the second generator. A second discriminator discriminates real images in the second domain against synthetic images generated by the first generator. The discriminators and generators are deep neural networks and respectively form a generative network and a discriminative network in a cyclic GAN framework configured to increase an error rate of the discriminative network to improve synthetic image quality.

Abstract: A computer-implemented method executed by at least one processor for training fast models for real-time object detection with knowledge transfer is presented. The method includes employing a Faster Region-based Convolutional Neural Network (R-CNN) as an objection detection framework for performing the real-time object detection, inputting a plurality of images into the Faster R-CNN, and training the Faster R-CNN by learning a student model from a teacher model by employing a weighted cross-entropy loss layer for classification accounting for an imbalance between background classes and object classes, employing a boundary loss layer to enable transfer of knowledge of bounding box regression from the teacher model to the student model, and employing a confidence-weighted binary activation loss layer to train intermediate layers of the student model to achieve similar distribution of neurons as achieved by the teacher model.

Abstract: A face recognition system is provided. The system includes a device configured to capture an input image of a subject. The system further includes a processor. The processor estimates, using a 3D Morphable Model (3DMM) conditioned Generative Adversarial Network, 3DMM coefficients for the subject of the input image. The subject varies from an ideal front pose. The processor produces, using an image generator, a synthetic frontal face image of the subject of the input image based on the input image and the 3DMM coefficients. An area spanning the frontal face of the subject is made larger in the synthetic image than in the input image. The processor provides, using a discriminator, a decision indicative of whether the subject of the synthetic image is an actual person. The processor provides, using a face recognition engine, an identity of the subject in the input image based on the synthetic and input images.

Abstract: An object recognition system is provided that includes a device configured to capture a video sequence formed from unlabeled testing video frames. The system includes a processor configured to pre-train a recognition engine formed from a reference set of CNNs on a still image domain that includes labeled training still image frames. The processor adapts the recognition engine to a video domain to form an adapted recognition engine, by applying a non-reference set of CNNs to a set of domains that include the still image and video domains and a degraded image domain. The degraded image domain includes labeled synthetically degraded versions of the labeled training still image frames included in the still image domain. The video domain includes random unlabeled training video frames. The processor recognizes, using the adapted engine, a set of objects in the video sequence. A display device displays the set of recognized objects.

Abstract: An action recognition system is provided that includes a device configured to capture a video sequence formed from a set of unlabeled testing video frames. The system further includes a processor configured to pre-train a recognition engine formed from a reference set of CNNs on a still image domain that includes labeled training still image frames. The processor adapts the recognition engine to a video domain to form an adapted engine, by applying non-reference CNNs to domains that include the still image and video domains and a degraded image domain that includes labeled synthetically degraded versions of the frames in the still image domain. The video domain includes random unlabeled training video frames. The processor recognizes, using the adapted engine, an action performed by at least one object in the sequence, and controls a device to perform a response action in response to an action type of the action.

Abstract: A video retrieval system is provided that includes a server for retrieving video sequences from a remote database responsive to a text specifying a face recognition result as an identity of a subject of an input image. The face recognition result is determined by a processor of the server, which estimates, using a 3DMM conditioned Generative Adversarial Network, 3DMM coefficients for the subject of the input image. The subject varies from an ideal front pose. The processor produces a synthetic frontal face image of the subject of the input image based on the input image and coefficients. An area spanning the frontal face of the subject is made larger in the synthetic than in the input image. The processor provides a decision of whether the synthetic image subject is an actual person and provides the identity of the subject in the input image based on the synthetic and input images.

Abstract: A surveillance system is provided that includes a device configured to capture a video sequence, formed from a set of unlabeled testing video frames, of a target area. The surveillance system further includes a processor configured to pre-train a recognition engine formed from a reference set of CNNs on a still image domain that includes labeled training still image frames. The processor adapts the recognition engine to a video domain to form an adapted recognition engine, by applying a non-reference set of CNNs to domains including the still image and video domains and a degraded image domain. The degraded image domain includes labeled synthetically degraded versions of the frames included in the still image domain. The video domain includes random unlabeled training video frames. The processor recognizes, using the adapted engine, at least one object in the target area. A display device displays the recognized objects.

Abstract: A video surveillance system is provided. The system includes a device configured to capture an input image of a subject located in an area. The system further includes a processor. The processor estimates, using a three-dimensional Morphable Model (3DMM) conditioned Generative Adversarial Network, 3DMM coefficients for the subject of the input image. The subject varies from an ideal front pose. The processor produces, using an image generator, a synthetic frontal face image of the subject of the input image based on the input image and coefficients. An area spanning the frontal face of the subject is made larger in the synthetic than in the input image. The processor provides, using a discriminator, a decision of whether the subject of the synthetic image is an actual person. The processor provides, using a face recognition engine, an identity of the subject in the input image based on the synthetic and input images.

Abstract: A face recognition system is provided that includes a device configured to capture a video sequence formed from a set of unlabeled testing video frames. The system includes a processor configured to pre-train a face recognition engine formed from reference CNNs on a still image domain that includes labeled training still image frames of faces. The processor adapts the face recognition engine to a video domain to form an adapted engine, by applying non-reference CNNs to domains including the still image and video domains and a degraded image domain. The degraded image domain includes labeled synthetically degraded versions of the frames included in the still image domain. The video domain includes random unlabeled training video frames. The processor recognizes, using the adapted engine, identities of persons corresponding to at least one face in the video sequence to obtain a set of identities. A display device displays the set of identities.

Abstract: Systems and methods are described for multithreaded navigation assistance by acquired with a single camera on-board a vehicle, using 2D-3D correspondences for continuous pose estimation, and combining the pose estimation with 2D-2D epipolar search to replenish 3D points.

Abstract: A system and method are provided. The system includes an image capture device configured to capture a video sequence formed from a set of input image frames and including a set of objects. The system further includes a processor configured to detect the objects to form object detections and track the object detections over the input frames to form tracked detections. The processor is also configured to generate for a current input frame, responsive to conditions, a set of sparse object proposals for a current location of an object based on: (i) the tracked detections of the object from an immediately previous input frame; and (ii) detection proposals for the object derived from the current frame. The processor is additionally configured to provide a user perceptible indication of the current location of the object, based on the set of sparse object proposals.

Abstract: A system and method are provided for driving assistance. The system includes an image capture device configured to capture a video sequence, relative to an outward view from a vehicle, which includes a set of objects and is formed from a set of image frames. The system includes a processor configured to detect the objects to form a set of object detections, and track the set of object detections over the frames to form tracked detections. The processor is configured to generate for a current frame, responsive to conditions, a set of sparse object proposals for a current location of an object based on: (i) the tracked detections of the object from an immediately previous frame; and (ii) detection proposals for the object derived from the current frame. The processor is configured to perform an action to mitigate a likelihood of potential harmful due to a current object location.

Abstract: An action recognition system and method are provided. The system includes an image capture device configured to capture a video sequence formed from image frames and depicting a set of objects. The system includes a processor configured to detect the objects to form object detections. The processor is configured to track the object detections over the frames to form tracked detections. The processor is configured to generate for a current frame, responsive to conditions, sparse object proposals for a current location of an object based on: (i) the tracked detections of the object from an immediately previous frame; and (ii) detection proposals for the object derived from the current frame. The processor is configured to control a hardware device to perform a response action in response to an identification of an action type of an action performed by the object, the identification being based on the sparse object proposals.

Abstract: A surveillance system and method are provided. The surveillance system includes an image capture device configured to capture a video sequence of a target area that includes a set of objects and is formed from a set of image frames. The surveillance system also includes a processor. The processor is configured to detect the objects to form object detections, and track the object detections over the frames to form tracked detections. The processor is further configured to generate for a current input frame, responsive to conditions, a set of sparse object proposals for a current location of an object based on: (i) the tracked detections of the object from an immediately previous frame; and (ii) detection proposals for the object derived from the current frame. The processor is additionally configured to provide a user perceptible indication of the current location of the object, based on the sparse object proposals.

Abstract: A surveillance system and method are provided. The surveillance system includes at least one camera configured to capture a set of images of a given target area that includes a set of objects to be tracked. The surveillance system includes a memory storing a learning model configured to perform multi-object tracking by jointly learning arbitrarily parameterized and differentiable cost functions for all variables in a linear program that associates object detections with bounding boxes to form trajectories. The surveillance system includes a processor configured to perform surveillance of the target area to (i) detect the objects and track locations of the objects by applying the learning model to the images in a surveillance task that uses the multi-object tracking, and (ii), provide a listing of the objects and their locations for surveillance task. A bi-level optimization is used to minimize a loss defined on a solution of the linear program.

Abstract: An action recognition system and method are provided. The action recognition system includes an image capture device configured to capture an actual image depicting an object. The action recognition system includes a processor configured to render, based on a set of 3D CAD models, synthetic images with corresponding intermediate shape concept labels. The processor is configured to form a multi-layer CNN which jointly models multiple intermediate shape concepts, based on the rendered synthetic images. The processor is configured to perform an intra-class appearance variation-aware and occlusion-aware 3D object parsing on the actual image by applying the CNN thereto to generate an image pair including a 2D and 3D geometric structure of the object. The processor is configured to control a device to perform a response action in response to an identification of an action performed by the object, wherein the identification of the action is based on the image pair.

Abstract: A system and method are provided for driving assistance. The system includes an image capture device configured to capture an actual image relative to an outward view from a motor vehicle and depicting an object. The system further includes a processor configured to render, based on a set of 3D CAD models, synthetic images with corresponding intermediate shape concept labels. The processor is further configured to form a multi-layer CNN which jointly models multiple intermediate shape concepts, based on the rendered synthetic images. The processor is also configured to perform an intra-class appearance variation-aware and occlusion-aware 3D object parsing on the actual image by applying the CNN to the actual image to output an image pair including a 2D and 3D geometric structure of the object. The processor is additionally configured to perform an action to mitigate a likelihood of harm involving the motor vehicle, based on the image pair.

Abstract: A computer-implemented method, system, and computer program product is provided for pose-invariant facial recognition. The method includes generating, by a processor using a recognition neural network, a rich feature embedding for identity information and non-identity information for each of one or more images. The method also includes generating, by the processor using a Siamese reconstruction network, one or more pose-invariant features by employing the rich feature embedding for identity information and non-identity information. The method additionally includes identifying, by the processor, a user by employing the one or more pose-invariant features. The method further includes controlling an operation of a processor-based machine to change a state of the processor-based machine, responsive to the identified user in the one or more images.