Abstract: Methods and systems for training a model include combining data from multiple datasets, the datasets having different respective label spaces. Relationships between labels in the different label spaces are identified. A unified neural network model is trained, using the combined data and the identified relationships to generate a unified model, with a class relational binary cross-entropy loss.

Abstract: Video methods and systems include extracting features of a first modality and a second modality from a labeled first training dataset in a first domain and an unlabeled second training dataset in a second domain. A video analysis model is trained using contrastive learning on the extracted features, including optimization of a loss function that includes a cross-domain regularization part and a cross-modality regularization part.

Abstract: A computer-implemented method for fusing geometrical and Convolutional Neural Network (CNN) relative camera pose is provided. The method includes receiving two images having different camera poses. The method further includes inputting the two images into a geometric solver branch to return, as a first solution, an estimated camera pose and an associated pose uncertainty value determined from a Jacobian of a reproduction error function. The method also includes inputting the two images into a CNN branch to return, as a second solution, a predicted camera pose and an associated pose uncertainty value. The method additionally includes fusing, by a processor device, the first solution and the second solution in a probabilistic manner using Bayes' rule to obtain a fused pose.

Abstract: A computer-implemented method for implementing a self-supervised visual odometry framework using long-term modeling includes, within a pose network of the self-supervised visual odometry framework including a plurality of pose encoders, a convolution long short-term memory (ConvLSTM) module having a first-layer ConvLSTM and a second-layer ConvLSTM, and a pose prediction layer, performing a first stage of training over a first image sequence using photometric loss, depth smoothness loss and pose cycle consistency loss, and performing a second stage of training to finetune the second-layer ConvLSTM over a second image sequence longer than the first image sequence.

Abstract: An action recognition system is provided that includes a device configured to capture a video sequence formed from a set of unlabeled testing video frames. The system further includes a processor configured to pre-train a recognition engine formed from a reference set of CNNs on a still image domain that includes labeled training still image frames. The processor adapts the recognition engine to a video domain to form an adapted engine, by applying non-reference CNNs to domains that include the still image and video domains and a degraded image domain that includes labeled synthetically degraded versions of the frames in the still image domain. The video domain includes random unlabeled training video frames. The processor recognizes, using the adapted engine, an action performed by at least one object in the sequence, and controls a device to perform a response action in response to an action type of the action.

Abstract: A method for multi-task learning via gradient split for rich human analysis is presented. The method includes extracting images from training data having a plurality of datasets, each dataset associated with one task, feeding the training data into a neural network model including a feature extractor and task-specific heads, wherein the feature extractor has a feature extractor shared component and a feature extractor task-specific component, dividing filters of deeper layers of convolutional layers of the feature extractor into N groups, N being a number of tasks, assigning one task to each group of the N groups, and manipulating gradients so that each task loss updates only one subset of filters.

Abstract: Methods and systems for determining a path include detecting objects within a perspective image that shows a scene. Depth is predicted within the perspective image. Semantic segmentation is performed on the perspective image. An attention map is generated using the detected objects and the predicted depth. A refined top-down view of the scene is generated using the predicted depth and the semantic segmentation. A parametric top-down representation of the scene is determined using a relational graph model. A path through the scene is determined using the parametric top-down representation.

Abstract: A method is provided for unsupervised domain adaptation for video classification. The method learns a transformation for each target video clips taken from a set of target videos, responsive to original features extracted from the target video clips. The transformation corrects differences between a target domain corresponding to target video clips and a source domain corresponding to source video clips taken from a set of source videos. The method adapts the target to the source domain by applying the transformation to the original features extracted to obtain transformed features for the plurality of target video clips. The method converts the original and transformed features of same ones of the target video clips into a single classification feature for each of the target videos. The method classifies a new target video relative to the set of source videos using the single classification feature for each of the target videos.

Abstract: A method for employing a general label space voting-based differentially private federated learning (DPFL) framework is presented. The method includes labeling a first subset of unlabeled data from a first global server, to generate first pseudo-labeled data, by employing a first voting-based DPFL computation where each agent trains a local agent model by using private local data associated with the agent, labeling a second subset of unlabeled data from a second global server, to generate second pseudo-labeled data, by employing a second voting-based DPFL computation where each agent maintains a data-independent feature extractor, and training a global model by using the first and second pseudo-labeled data to provide provable differential privacy (DP) guarantees for both instance-level and agent-level privacy regimes.

Abstract: A method provided for 3D object localization predicts pairs of 2D bounding boxes. Each pair corresponds to a detected object in each of the two consecutive input monocular images. The method generates, for each detected object, a relative motion estimation specifying a relative motion between the two images. The method constructs an object cost volume by aggregating temporal features from the two images using the pairs of 2D bounding boxes and the relative motion estimation to predict a range of object depth candidates and a confidence score for each object depth candidate and an object depth from the object depth candidates. The method updates the relative motion estimation based on the object cost volume and the object depth to provide a refined object motion and a refined object depth. The method reconstructs a 3D bounding box for each detected object based on the refined object motion and refined object depth.

Abstract: A method for acquiring privacy-enhancing encodings in an optical domain before image capture is presented. The method includes feeding a differentiable sensing model with a plurality of images to obtain encoded images, the differentiable sensing model including parameters for sensor optics, integrating the differentiable sensing model into an adversarial learning framework where parameters of attack networks, parameters of utility networks, and the parameters of the sensor optics are concurrently updated, and, once adversarial training is complete, validating efficacy of a learned sensor design by fixing the parameters of the sensor optics and training the attack networks and the utility networks to learn to estimate private and public attributes, respectively, from a set of the encoded images.

Abstract: A computer-implemented method for implementing face spoofing detection using a physical-cue-guided multi-source multi-channel framework includes receiving a set of data including face recognition data, liveness data and material data associated with at least one face image, obtaining a shared feature from the set of data using a backbone neural network structure, performing, based on the shared feature, a pretext task corresponding to face recognition, a first proxy task corresponding to depth estimation, a liveness detection task, and a second proxy task corresponding to material prediction, and aggregating outputs of the pretext task, the first proxy task, the liveness detection task and the second proxy task using an attention mechanism to boost face spoofing detection performance.

Abstract: A method is provided for drone-video-based action recognition. The method learns a transformation for each of target video clips taken from a set of target videos, responsive to original features extracted from the target video clips. The transformation corrects differences between a target drone domain corresponding to the target video clips and a source non-drone domain corresponding to source video clips taken from a set of source videos. The method adapts the target to the source domain by applying the transformation to the original features to obtain transformed features for the target video clips. The method converts the original and transformed features of same ones of the target video clips into a single classification feature for each of the target videos. The method classifies a human action in a new target video relative to the set of source videos using the single classification feature for each of the target videos.

Abstract: A method for correcting blur effects is presented. The method includes generating a plurality of images from a camera, synthesizing blurred images from sharp image counterparts to generate training data to train a structure-and-motion-aware convolutional neural network (CNN), and predicting a camera motion and a depth map from a single blurred image by employing the structure-and-motion-aware CNN to remove blurring from the single blurred image.

Abstract: Methods and systems for object detection include training dataset-specific object detectors using respective annotated datasets, each of the annotated datasets including annotations for a respective set of one or more object classes. The annotated datasets are cross-annotated using the dataset-specific object detectors. A unified object detector is trained, using the cross-annotated datasets, to detect all of the object classes of the annotated datasets. Objects are detected in an input image using the unified object detector.

Abstract: A computer-implemented method is provided for domain adaptation between a source domain and a target domain. The method includes applying, by a hardware processor, an attention network to features extracted from images included in the source and target domains to provide attended features relating to a given task to be domain adapted between the source and target domains. The method further includes applying, by the hardware processor, a deformation network to at least some of the attended features to align the attended features between the source and target domains using warping to provide attended and warped features. The method also includes training, by the hardware processor, a target domain classifier using the images from the source domain. The method additionally includes classifying, by the hardware processor using the trained target domain classifier, at least one image from the target domain.

Abstract: Methods and systems for training a neural network include generate an image of a mask. A copy of an image is generated from an original set of training data. The copy is altered to add the image of a mask to a face detected within the copy. An augmented set of training data is generated that includes the original set of training data and the altered copy. A neural network model is trained to recognize masked faces using the augmented set of training data.

Abstract: A method for correcting rolling shutter (RS) effects is presented. The method includes generating a plurality of images from a camera, synthesizing RS images from global shutter (GS) counterparts to generate training data to train the structure-and-motion-aware convolutional neural network (CNN), and predicting an RS camera motion and an RS depth map from a single RS image by employing a structure-and-motion-aware CNN to remove RS distortions from the single RS image.

Abstract: Methods and systems for training a trajectory prediction model and performing a vehicle maneuver include encoding a set of training data to generate encoded training vectors, where the training data includes trajectory information for agents over time. Trajectory scenarios are simulated based on the encoded training vectors, with each simulated trajectory scenario representing one or more agents with respective agent trajectories, to generate simulated training data. A predictive neural network model is trained using the simulated training data to generate predicted trajectory scenarios based on a detected scene.

Abstract: Systems and methods for recognizing fine-grained objects are provided. The system divides unlabeled training data from a target domain into two or more target subdomains using an attribute annotation. The system ranks the target subdomains based on a similarity to the source domain. The system applies multiple domain discriminators between each of the target subdomains and a mixture of the source domain and preceding target domains. The system recognizes, using the multiple domain discriminators for the target domain, fine-grained objects.