Abstract: A method and a server for training a generative stylization system to generate motion sequences for digital objects are provided.

Abstract: System, method, and computer readable medium for generating a 3D texture for a 3D object are disclosed. A 3D mesh and a text prompt for a desired texture are obtained. A sequence of texture sampling steps is performed, where each given texture sampling step includes iterating over a plurality of 2D views of the 3D mesh to generate an intermediate texture map. For a given iteration, a given 2D view and the text prompt are processed using a pre-trained 2D image generation diffusion model to fill in a portion of an intermediate texture map based on the given 2D view. A noise estimation generated by the diffusion model is refined, adding the intermediate texture map as guidance, to generate a latent variable to be inputted to a subsequent texture sampling step, enabling generation of a 3D texture, based on a text prompt, with fewer artifacts.

Abstract: The present invention relates to generating at least one 3D texture image, said method comprising acquiring a first plurality of multi-view images of an object; generating a 3D mesh of said object from said first plurality of multi-view images; calculating a UV atlas; generating: one diffuse UV texture using a first multilayer perceptron MLPd, and one specular color image using a second multilayer perceptron MLPs; generating one diffuse color image by rendering said diffuse UV texture combined with said 3D mesh; rendering at least a 3D texture image by combining said specular color image with said diffuse color image; editing one image; propagating said editing onto said diffuse UV texture; generating an edited diffuse color image by rendering said edited diffuse UV texture combined with said 3D mesh; and rendering an edited 3D texture image by combining said specular color image with said edited diffuse color image.

Abstract: Methods and devices for inpainting a three-dimensional scene in response to object removal. A 3D scene is stored in the form of Gaussian splats. When object removal occurs, a never-before-seen (NBS) area may be revealed that requires inpainting as a result of pruning of the Gaussian splats. Object masks are dilated and remapped to the pruned scene to identify pixels bordering the NBS area. A reference image is inpainted and the geography of its NBS area reconstructed using depth prediction and smoothing. The reference image is then warped to other viewpoints and the warped images and their inpainting masks are input to a multi-view restoration model created by modifying a pre-trained diffusion-based inpainting model to use sparse space-time attention layers to ensure consistency among views. Those refined views are then used to refine the pruned 3D Gaussian model.

Abstract: Methods and devices for machine vision-based selection of content are described. One or more hands are detected in a current frame of video data. A respective fingertip location is determined for each of up to two of the detected hands. A content selection gesture is determined corresponding to the up to two detected hands. Selected content is extracted, as indicated by the content selection gesture and based on the up to two fingertip locations. The device may be a smartphone, a tablet, a laptop, a smart light device, a reader device, etc.

Abstract: Systems and methods for temporal action localization of video data are described. A feature representation extracted from video data has a temporal dimension and a spatial dimension. The feature representation is self-aligned in the spatial dimension. Spatial multi-sampling is performed to obtain a plurality of sparse samples of the self-aligned representation along the spatial dimension, and the multi-sampled representation is fused with the self-aligned representation. Attention-based context information aggregation is applied on the fused representation to obtain a spatially refined representation. Local temporal information aggregation is applied on the self-aligned representation to obtain a temporally refined representation. Action localization is performed on a concatenation of the spatially refined representation and the temporally refined representation.

Abstract: System, method, and computer readable medium for generating a 3D texture for a 3D object are disclosed. A 3D mesh and a text prompt for a desired texture are obtained. A sequence of texture sampling steps is performed, where each given texture sampling step includes iterating over a plurality of 2D views of the 3D mesh to generate an intermediate texture map. For a given iteration, a given 2D view and the text prompt are processed using a pre-trained 2D image generation diffusion model to fill in a portion of an intermediate texture map based on the given 2D view. A noise estimation generated by the diffusion model is refined, adding the intermediate texture map as guidance, to generate a latent variable to be inputted to a subsequent texture sampling step, enabling generation of a 3D texture, based on a text prompt, with fewer artifacts.

Abstract: Method and devices for training a keypoint estimation network are described. In each training iteration, synthetic images are generated by a generator, each synthetic image being assigned respective assigned keypoints by the generator. Using a prior-iteration of the keypoint estimation network, a set of predicted keypoints is obtained for each synthetic image. Based on an error score between the predicted keypoints and the assigned keypoints, poor quality synthetic images are discarded. The remaining synthetic images, together with real world images, are used to train an updated keypoint estimation network. The performance of the updated keypoint estimation network is validated, and the training iterations are performed until a convergence criteria is satisfied.

Abstract: Methods and devices are described for computer vision-based gesture detection. From a frame of image data, extracted locations of keypoints of a detected hand are obtained. The extracted locations are normalized to obtain normalized features. The normalized features are processed using a trained decision tree ensemble to generate a probability of a valid gesture for the detected hand. The generated probability is compared with a defined decision threshold to generate a binary classification to classify the detected hand as a valid gesture or invalid gesture.

Abstract: Methods, devices, and processor-readable media for method for performing a 3D reconstruction from a single view image, comprising progressively denoising a randomly initialized set of 3D-gaussian representations with continuous guidance from an input image.

Abstract: A computer-implemented method for vision transforming includes, for each of one or more channels of each of a set of tiles of an image: splitting the channel into at least a first channel portion and a second channel portion; processing the first channel portion using depthwise convolution; processing the second channel portion with multi-head self-attention; and combining the processed first channel portion and the processed second channel portion; and identifying an object in the image at least partially based on the combined processed first channel portion and processed second channel portion for each of the one or more channels of each of the set of tiles of the image.

Abstract: Methods, systems, and media for training deep neural networks for cross-domain few-shot classification are described. The methods comprise an encoder and a decoder of a deep neural network. The training of the autoencoder comprises two training stages. For each iteration in the first training stage, a batch of data samples from the source dataset are sampled and fed to the encoder to generate a plurality of source feature maps, then determining a first training stage loss, which updates the autoencoder's parameters. For each iteration in the second training stage, the novel dataset is split into a support set and a query set. The support set is fed to the encoder to determine a prototype for each class label. The query set is also fed to the encoder to calculate a query set metric classification loss. The query set metric classification loss updates the autoencoder's parameters.

Abstract: Methods and systems for estimation of a 3D hand pose are disclosed. A 2D image containing a detected hand is processed using a U-net network to obtain a global feature vector and a heatmap for the keypoints of the hand. Information from the global feature vector and the heatmap are concatenated to obtain a set of input tokens that are processed using a transformer encoder to obtain a first set of 2D keypoints representing estimated 2D locations of the keypoints in a first view. The first set of 2D keypoints are inputted as a query to a transformer decoder, to obtain a second set of 2D keypoints representing estimated 2D locations of the keypoints in a second view. The first and second sets of 2D keypoints are aggregated to output the set of estimated 3D keypoints.

Abstract: Methods and systems for gesture-based control of a device are described. An input frame is processed to determine a location of a distinguishing anatomical feature in the input frame. A virtual gesture-space is defined based on the location of the distinguishing anatomical feature, the virtual gesture-space being a defined space for detecting a gesture input. The input frame is processed in only the virtual gesture-space, to detect and track a hand. Using information generated from detecting and tracking the at least one hand, a gesture class is determined for the at least one hand. The device may be a smart television, a smart phone, a tablet, etc.

Abstract: Methods and devices for generating a customized video segment from a video are disclosed. The video is partitioned into video segments. For each respective video segment, a respective set of scores is computed, where each score represents a respective content feature in the respective video segment. A respective weighted aggregate score is computed for each respective video segment by applying, to each respective set of scores, a common set of weight values. A selected video segment is outputted as the customized video segment, where the selected video segment is selected from one or more high-ranked video segments having high-ranked weighted aggregate scores.

Abstract: Methods and systems for estimation of a 3D hand pose are disclosed. A 2D image containing a detected hand is processed using a U-net network to obtain a global feature vector and a heatmap for the keypoints of the hand. Information from the global feature vector and the heatmap are concatenated to obtain a set of input tokens that are processed using a transformer encoder to obtain a first set of 2D keypoints representing estimated 2D locations of the keypoints in a first view. The first set of 2D keypoints are inputted as a query to a transformer decoder, to obtain a second set of 2D keypoints representing estimated 2D locations of the keypoints in a second view. The first and second sets of 2D keypoints are aggregated to output the set of estimated 3D keypoints.

Abstract: Methods and systems for gesture-based control of a device are described. A virtual gesture-space is determined in a received input frame. The virtual gesture-space is associated with a primary user from a ranked user list of users. The received input frame is processed in only the virtual gesture-space, to detect and track a hand. Using a hand bounding box generated by detecting and tracking the hand, gesture classification is performed to determine a gesture input associated with the hand. A command input associated with the determined gesture input is processed. The device may be a smart television, a smart phone, a tablet, etc.

Abstract: Methods and systems for gesture-based control of a device are described. A virtual gesture-space is determined in a received input frame. The virtual gesture-space is associated with a primary user from a ranked user list of users. The received input frame is processed in only the virtual gesture-space, to detect and track a hand. Using a hand bounding box generated by detecting and tracking the hand, gesture classification is performed to determine a gesture input associated with the hand. A command input associated with the determined gesture input is processed. The device may be a smart television, a smart phone, a tablet, etc.

Abstract: Methods, systems, and media for image searching are described. Images comprising one query image and a plurality of candidate images are received. For each candidate image, a first model similarity measure from an output of a first model configured for scene classification to perceive scenes in the images is determined. Further, for each candidate image of the plurality of candidate images, a second model similarity measure from the output of a second model configured for attribute classification to perceive attributes in the images is determined. For each candidate image of the plurality of candidate images, a similarity agglomerate index of a weighted aggregate of the first model similarity measure and the second model similarity measure is computed. The plurality of candidate images based on the respective similarity agglomerate index of each candidate image are ranked and a first ranked candidate images corresponding to the searched images are generated.

Abstract: Methods and systems for gesture-based control of a device are described. A virtual gesture-space is determined in a received input frame. The virtual gesture-space is associated with a primary user from a ranked user list of users. The received input frame is processed in only the virtual gesture-space, to detect and track a hand. Using a hand bounding box generated by detecting and tracking the hand, gesture classification is performed to determine a gesture input associated with the hand. A command input associated with the determined gesture input is processed. The device may be a smart television, a smart phone, a tablet, etc.