Abstract: Example embodiments of the present disclosure provide an example computer-implemented method for constructing a three-dimensional semantic segmentation of a scene from two-dimensional inputs. The example method includes obtaining, by a computing system comprising one or more processors, an image set comprising one or more views of a subject scene. The example method includes generating, by the computing system and based at least in part on the image set, a scene representation describing the subject scene in three dimensions. The example method includes generating, by the computing system and using a machine-learned semantic segmentation model framework, a multidimensional field of probability distributions over semantic categories, the multidimensional field defined over the three dimensions of the subject scene. The example method includes outputting, by the computing system, classification data for at least one location in the subject scene.

Abstract: Provided are machine learning models that generate geometry-free neural scene representations through efficient object-centric novel-view synthesis. In particular, one example aspect of the present disclosure provides a novel framework in which an encoder model (e.g., an encoder transformer network) processes one or more RGB images (with or without pose) to produce a fully latent scene representation that can be passed to a decoder model (e.g., a decoder transformer network). Given one or more target poses, the decoder model can synthesize images in a single forward pass. In some example implementations, because transformers are used rather than convolutional or MLP networks, the encoder can learn an attention model that extracts enough 3D information about a scene from a small set of images to render novel views with correct projections, parallax, occlusions, and even semantics, without explicit geometry.

Abstract: A method includes obtaining first feature vectors and second feature vectors representing contents of a first and second image frame, respectively, of an input video. The method may also include generating, based on the first feature vectors, first slot vectors, where each slot vector represents attributes of a corresponding entity as represented in the first image frame, and generating, based on the first slot vectors, predicted slot vectors including a corresponding predicted slot vector that represents a transition of the attributes of the corresponding entity from the first to the second image frame. The method may additionally include generating, based on the predicted slot vectors and the second feature vectors, second slot vectors including a corresponding slot vector that represents the attributes of the corresponding entity as represented in the second image frame, and determining an output based on the predicted slot vectors or the second slot vectors.

Abstract: Disclosed is a computer-implemented method for segmenting input data. In the method a plurality of tags is generated; the input data is masked with the plurality of tags; a plurality of output reconstructions is generated by inputting the plurality of masked input data to one of the following: a denoising neural network, a variational autoencoder; a plurality of values representing distances of each plurality of output reconstructions to the input data are determined; a plurality of updated versions of input data is generated by applying at least one of the determined values representing distances of each plurality of output reconstructions to the input data; and updated output reconstructions are generated by inputting the plurality of updated versions of input data to one of the networks. Also disclosed is a method for training the network and a processing unit.

Abstract: A computer-based method includes receiving an input signal at a neuron in a computer-based neural network that includes a plurality of neuron layers, applying a first non-linear transform to the input signal at the neuron to produce a plain signal, and calculating a weighted sum of a first component of the input signal and the plain signal at the neuron. In a typical implementation, the first non-linear transform is a function of the first component of the input signal and at least a second component of the input signal.

Abstract: Disclosed is a computer-implemented method for segmenting input data. In the method a plurality of tags is generated; the input data is masked with the plurality of tags; a plurality of output reconstructions is generated by inputting the plurality of masked input data to one of the following: a denoising neural network, a variational autoencoder; a plurality of values representing distances of each plurality of output reconstructions to the input data are determined; a plurality of updated versions of input data is generated by applying at least one of the determined values representing distances of each plurality of output reconstructions to the input data; and updated output reconstructions are generated by inputting the plurality of updated versions of input data to one of the networks. Also disclosed is a method for training the network and a processing unit.

Abstract: A computer-based method includes receiving an input signal at a neuron in a computer-based neural network that includes a plurality of neuron layers, applying a first non-linear transform to the input signal at the neuron to produce a plain signal, and calculating a weighted sum of a first component of the input signal and the plain signal at the neuron. In a typical implementation, the first non-linear transform is a function of the first component of the input signal and at least a second component of the input signal.