Abstract: A motion generation system includes: a model configured to generate a rendering of a human performing an action in a space, the model including: an encoder module configured to encode input into encodings; a prediction module configured to generate predicted trajectories of the human performing the action based on the encodings using a latent space; a decoder module configured to generate decodings based on the predicted trajectories; and a rendering module configured to generate the rendering based on the decodings; and a training module configured to: (a) train the model based on input video including humans performing actions; and (b), after (a), train the model based on geometry of a scene, one or more target actions for performance by a human in the scene, and observations of the human during performance of the one or more target actions in the scene.

Abstract: The system and method to improve image retrieval models in case of extreme domain shifts for long term visual localization. An initial dataset of real images of an environment can be obtained from an image database that may comprise of matching pairs and sets of non-matching images. The initial dataset is augmented with synthetic variants of each image to represent the potential changing conditions of the environment. The synthetic variants may be generated by using generative AI models. To validate synthetic images, a geometric consistency score is calculated for each synthetic pair based on a degree of geometric correspondence between images of the synthetic pair. A subset of k synthetic tuples is selected from a set of tuples corresponding to a matching pair, either randomly or based on the geometric consistency score, to compute a contrastive loss function for training of image retrieval model along with other custom/disclosed settings.

Abstract: A computer-implemented method for reconstructing a scene in three dimensions from a plurality of images of one or more viewpoints of the scene acquired using an imaging device includes: receiving the plurality of images without receiving extrinsic or intrinsic properties of the imaging device; and processing the plurality of images using a neural network to produce a plurality of pointmaps of the scene that correspond to the plurality of images and that are aligned in a common coordinate frame, where each pointmap is a one-to-one mapping between pixels of one of the plurality of images and three-dimensional points of the scene.

Abstract: A training system includes: a transformer module having the transformer architecture and configured to perform a vision task; and a training module configured to: receive a training image having a predetermined resolution; determine N windows of tokens of pixels in the training image and mask the tokens of all of the other pixels of the training image that are outside of the N windows, where N is an integer greater than or equal to 2; input the N windows of tokens to the transformer module; train the transformer module based on an output of the transformer module generated based on the N windows of tokens; and test the transformer module using a test image having the predetermined resolution.

Abstract: Methods and systems for training a model for a goal-oriented visual navigation task. A binocular encoder is pretrained on one or more pretext tasks wherein one or more layers of the binocular encoder may be adapted by one or more adaptors, and is combined with a navigation policy module in the navigation model. The navigation model is end-to-end trained on a downstream visual navigation task.

Abstract: A training system includes: an iterative attention module configured to, based on first features in input images, determine ordered sets of second features using iterative attention over T iterations, where T is an integer greater than or equal to two; and a training module configured to: selectively input pairs of matching images to the iterative attention module; selectively input non-matching images to the iterative attention module; and based on the ordered sets generated by the iterative attention module based on the input pairs of matching images and the input non-matching images, train the iterative attention module based on minimizing at least one of: a contrastive loss; and a cosine similarity loss.

Abstract: A method and system for automatically building a three-dimensional pose dataset for use in text to pose retrieval or text to pose generation for a class of poses includes (a) inputting three-dimensional keypoint coordinates of class-centric poses; (b) extracting, from the inputted three-dimensional keypoint coordinates of class-centric poses, posecodes, the posecodes representing a relation between a specific set of joints; (c) selecting extracted posecodes to obtain a discriminative description; (d) aggregating selected posecodes that share semantic information; (e) converting the aggregated posecodes by electronically obtaining individual descriptions by plugging each posecode information into one template sentence, picked at random from a set of possible templates for a given posecode category; (f) concatenating the individual descriptions in random order, using random pre-defined transitions; and (g) mapping the concatenated individual descriptions to class-centric poses to create the three-dimensional po

Abstract: A method and system for text-based pose editing to generate a new pose from an initial pose and user-generated text includes an user input device for inputting the initial pose and the user-generated text; a variational auto-encoder configured to receive the initial pose; a text conditioning pipeline configured to receive the user-generated text; a fusing module configured to produce parameters for a prior Gaussian distribution Np; a pose decoder configured to sample the Gaussian distribution Np and generate, therefrom, the new pose; and an output device to communicate the generated new pose to a user. The variational auto-encoder and the text conditioning pipeline are trained using a PoseFix dataset, wherein the PoseFix dataset includes triplets having a source pose, a target pose, and text modifier.

Abstract: Am image retrieval system includes: a neural network (NN) module configured to generate local features based on an input image; an iterative attention module configured to, via T iterations, generate an ordered set of super features in the input image based on the local features, where T is an integer greater than 1; and a selection module configured to select a second image from a plurality of images in an image database based on the second image having a second ordered set of super features that most closely match the ordered set of super features in the input image, where the super features in the set of super features do not include redundant local features of the input image.

Abstract: A computer-implemented method of recognition of actions performed by individuals includes: by one or more processors, obtaining images including at least a portion of an individual; by the one or more processors, based on the images, generating implicit representations of poses of the individual in the images; and by the one or more processors, determining an action performed by the individual and captured in the images by classifying the implicit representations of the poses of the individual.

Abstract: A computer implemented method and system using an object-agnostic model for predicting a pose of an object in an image receives a query image having a target object therein; receives a set of reference images of the target object from different viewpoints; encodes, using a vision transformer, the received query image and the received set of reference images to generate a set of token features for the received query image and a set of token features for the received set of reference images; extracts, using a transformer decoder, information from the set of token features for the encoded reference images with respect to a set of token features for the received query image; processes, using a prediction head, the combined set of token features to generate a 2D-3D mapping and a confidence map of the query image; and processes the 2D-3D mapping and confidence map to determine the pose of the target object in the query image.

Abstract: A training system includes: an encoder module configured to receive a query image and to generate a first vector representative of one or more features in the query image using an encoder; a mixing module configured to generate a second vector by mixing a third vector, representative of one or more features in a second image that is classified as a negative relative to the query image, with a fourth vector; and an adjustment module configured to train the encoder by selectively adjusting one or more parameters of the encoder based on the first vector and the second vector.

Abstract: A training system includes: a model; and a training module configured to: construct a first pair of images of at least a first portion of a first human captured at different times; construct a second pair of images of at least a second portion of a second human captured at the same time from different points of view; input the first and second pairs of images to the model; the model configured to: generate first and second reconstructed images of the at least the first portion of the first human based on the first and second pairs, respectively, and the training module is configured to selectively adjust one or more parameters of the model based on: the first reconstructed image and the second reconstructed image.

Abstract: A method includes: performing unsupervised pre-training of a model, the model including and a decoder including: obtaining a first image and a second image under different conditions or from different viewpoints; encoding, by the encoder, the first image into a representation of the first image and the second image into a representation of the second image; transforming the representation of the first image into a transformed representation; decoding, by the decoder, the transformed representation into a reconstructed image, where the transforming of the representation of the first image and the decoding of the transformed representation is based on the representation of the first image and the representation of the second image; and adjusting one or more parameters of at least one of the encoder and the decoder based on minimizing a loss; and fine-tuning the model, initialized with a set of task specific encoder parameters, for a geometric vision task.

Abstract: A motion generation system includes: a model configured to generate latent indices for a sequence of images including an entity performing an action based on an action label and a duration of the sequence; and a decoder module configured to: decode the latent indices and to generate the sequence of images including the entity performing the action based on the latent indices; and output the sequence of images including the entity performing the action to a display control module configured to display the sequence of images including the entity performing the action sequentially on a display.

Abstract: A system includes: a hand module to, based on a demonstration of a human hand grasping an object, determine first and second vectors that are normal to and parallel to a palm of the human hand, respectively, and a position of the human hand; a gripper module to determine third and fourth vectors that are normal to and parallel to a palm of a gripper of a robot, respectively, and a present position of the gripper; and an actuation module to: move the gripper when open such that the present position of the gripper is at the position of the human hand, the third and first vectors are aligned, and the fourth and second vectors are aligned; close fingers of the gripper based on minimizing a first loss; and actuate the fingers of the gripper to minimize a second loss determined based on the first loss and a third loss.

Abstract: A computer-implemented method includes: obtaining a pair of images depicting a same scene, the pair of images including a first image with a first pixel grid and a second image with a second pixel grid, the first pixel grid different than the second pixel grid; by a neural network module having a first set of parameters: generating a first feature map based on the first image; and generating a second feature map based on the second image; determining a first correlation volume based on the first and second feature maps; iteratively determining a second correlation volume based on the first correlation volume; determining a loss for the first and second feature maps based on the second correlation volume; generating a second set of the parameters based on minimizing a loss function using the loss; and updating the neural network module to include the second set of parameters.

Abstract: A computer-implemented method of recognition of actions performed by individuals includes: by one or more processors, obtaining images including at least a portion of an individual by the one or more processors, based on the images, generating implicit representations of poses of the individual in the images; and by the one or more processors, determining an action performed by the individual and captured in the images by classifying the implicit representations of the poses of the individual.

Abstract: A system for generating whole body poses includes: a body regression module configured to generate a first pose of a body of an animal in an input image by regressing from a stored body anchor pose; a face regression module configured to generate a second pose of a face of the animal in the input image by regressing from a stored face anchor pose; an extremity regression module configured to generate a third pose of an extremity of the animal in the input image by regressing from a stored extremity anchor pose; and a pose module configured to generate a whole body pose of the animal in the input image based on the first pose, the second pose, and the third pose.

Abstract: A training system includes: a neural network model configured to determine three-dimensional coordinates of joints, respectively, representing poses of animals in images, where the neural network model is trained using a first training dataset including: images including animals; and coordinates of joints of the animals in the images, respectively; and a training module configured to, after the training of the neural network model using the first training dataset, train the neural network model using a second training dataset including motion capture data, where the motion capture data does not include images of animals and includes measured coordinates at points, respectively, on animals.