Abstract: Systems, methods, software, and devices are disclosed herein for training a neural network using multiple images of a scene captured from different viewing directions. The training is expedited by first determining foreground pixels in each of the multiple images, and then selecting a non-uniform sample of pixels from each of the multiple images such that the foreground pixels are overrepresented in the non-uniform sample of pixels relative to background pixels. The neural network may then be trained using radiance values of voxels on rays propagating from each of the non-uniform sample of pixels along a corresponding viewing direction into the scene.

Abstract: Systems, methods, software, and devices are disclosed herein for training a neural network using multiple images of a scene captured from different viewing directions. A method of training the network includes identifying pixels in multiple images of a scene captured from different viewing directions. and determining, for each of the pixels, at least a known color value, a known radiance value, and a spatial value. The training minimizes a loss function having multiple loss terms: a first loss term that is dependent upon at least the known color value and the known radiance value for each of the pixels; and one or more additional loss terms dependent upon the spatial value determined for each pixel, such that the loss function varies for at least some of the pixels.

Abstract: Embodiments for rendering a 2D image of a dynamic 3D scene from different view angles using a neural radiance field (NeRF) are provided. In this regard, an AI image processing system is configured to process coordinates of a point in a dynamic 3D scene with an RNN over a number of time steps indicated by a time instance of interest to produce motion information of the point at the time instance of interest, process the motion information with a fully connected neural network to produce a displacement of the point from the coordinates in the dynamic 3D scene, and process a displaced point from a view angle of interest with the NeRF to render the point on the 2D image of the dynamic 3D scene at the time instance of interest. The displaced point is generated based on an estimate of the displacement of the point by leveraging motion cues.

Abstract: A controller for controlling a robot is provided. The controller comprises a hierarchical multimodal reinforcement learning (RL) neural network including a first level controller and three second level controllers. The second level controllers comprise a first sub level controller to receive input data based on predefined questions, a second sub level controller to receive the input data by generating a validation question based on state of the RL neural network and a third sub level controller to determine the input data based on state of the RL neural network. The controller is configured to select one of the second level controllers using the first level controller to perform a first interaction relating to a task based on the state of the RL neural network; generate a control command using the selected second level controller based on the corresponding input data; and control operation of the robot by executing control command.

Abstract: Systems, methods, and software are disclosed herein that improve computer vision technology in general, and 4D scene reconstruction in particular. An artificial intelligence (AI) image processing system employs multiple dynamic neural radiance fields (NeRFs) to render two-dimensional (2D) images of a four-dimensional (4D) scene from different viewpoints and different instances of time. The AI image processing system collects viewing parameters for rendering a two-dimensional (2D) image of the 4D scene, such as from a desired viewing direction and at a desired instance of time. The system then executes the multiple dynamic NeRFs to obtain, based on the viewing parameters, color and volume density values for voxels associated with pixels in the 2D image. The system then renders the 2D image based on a combination of the color and the volume density values obtained from the multiple dynamic NeRFs.

Abstract: Systems, methods, software, and devices are disclosed herein for training a neural network using multiple images of a scene captured from different viewing directions. The training is expedited by first determining foreground pixels in each of the multiple images, and then selecting a non-uniform sample of pixels from each of the multiple images such that the foreground pixels are overrepresented in the non-uniform sample of pixels relative to background pixels. The neural network may then be trained using radiance values of voxels on rays propagating from each of the non-uniform sample of pixels along a corresponding viewing direction into the scene.

Abstract: Embodiments for rendering a 2D image of a dynamic 3D scene from different view angles using a neural radiance field (NeRF) are provided. In this regard, an AI image processing system is configured to process coordinates of a point in a dynamic 3D scene with an RNN over a number of time steps indicated by a time instance of interest to produce motion information of the point at the time instance of interest, process the motion information with a fully connected neural network to produce a displacement of the point from the coordinates in the dynamic 3D scene, and process a displaced point from a view angle of interest with the NeRF to render the point on the 2D image of the dynamic 3D scene at the time instance of interest. The displaced point is generated based on an estimate of the displacement of the point by leveraging motion cues.

Abstract: A non-uniform video encoder system for generating a multi-depth encoding data for a scene is provided. The non-uniform video encoder system is configured to receive a sequence of video frames of a video of the scene and transform the sequence of video frames into series input data. The series input data is analyzed to identify changes in the evolution of the scene, by partitioning the series input data into a sequence of non-uniform segments. Each segment in the sequence of non-uniform segments is encoded by an encoder of an autoencoder architecture with non-uniform unrolling recursion to produce multi-depth encoding of the series input data. To encode a current segment at a current iteration to produce a current encoding, the non-uniform unrolling recursion combines the current segment with a previous encoding produced at a previous iteration and encodes the combination with the encoder.

Abstract: A controller for controlling a robot is provided. The controller comprises a hierarchical multimodal reinforcement learning (RL) neural network including a first level controller and three second level controllers. The second level controllers comprise a first sub level controller to receive input data based on predefined questions, a second sub level controller to receive the input data by generating a validation question based on state of the RL neural network and a third sub level controller to determine the input data based on state of the RL neural network. The controller is configured to select one of the second level controllers using the first level controller to perform a first interaction relating to a task based on the state of the RL neural network; generate a control command using the selected second level controller based on the corresponding input data; and control operation of the robot by executing control command.

Abstract: Embodiments disclose a method and system for a scene-aware audio-video representation of a scene. The scene-aware audio video representation corresponds to a graph of nodes connected by edges. A node in the graph is indicative of the video features of an object in the scene. An edge in the graph connecting two nodes indicates an interaction of the corresponding two objects in the scene. In the graph, at least one or more edges are associated with audio features of a sound generated by the interaction of the corresponding two objects. The graph of the audio-video representation of the scene may be used to perform a variety of different tasks. Examples of the tasks include one or a combination of an action recognition, an anomaly detection, a sound localization and enhancement, a noisy-background sound removal, and a system control.

Abstract: Embodiments disclose a method and system for a scene-aware audio-video representation of a scene. The scene-aware audio video representation corresponds to a graph of nodes connected by edges. A node in the graph is indicative of the video features of an object in the scene. An edge in the graph connecting two nodes indicates an interaction of the corresponding two objects in the scene. In the graph, at least one or more edges are associated with audio features of a sound generated by the interaction of the corresponding two objects. The graph of the audio-video representation of the scene may be used to perform a variety of different tasks. Examples of the tasks include one or a combination of an action recognition, an anomaly detection, a sound localization and enhancement, a noisy-background sound removal, and a system control.