Abstract: An artificial intelligence-based image processing system comprises a processor that executes instructions stored on a memory to classify an input image with a prototypical part neural network including a backbone subnetwork, a prototype subnetwork, and a readout subnetwork to produce an interpretable classification of the input image including one or a combination of a classification result of the input image and an interpretation of the classification result. The backbone subnetwork is trained with machine learning to process the input image with an incomplete sequence of active convolutional layers producing feature embeddings representing features extracted from pixels of different regions of the input image. The prototype subnetwork is trained to compare the feature embeddings with prototypical feature embeddings to produce results of comparison and the readout subnetwork is configured to analyze the results of comparison to produce the interpretable classification of the input image.

Abstract: A controller for controlling a robot is provided. The controller comprises a hierarchical multimodal reinforcement learning (RL) neural network including a first level controller and three second level controllers. The second level controllers comprise a first sub level controller to receive input data based on predefined questions, a second sub level controller to receive the input data by generating a validation question based on state of the RL neural network and a third sub level controller to determine the input data based on state of the RL neural network. The controller is configured to select one of the second level controllers using the first level controller to perform a first interaction relating to a task based on the state of the RL neural network; generate a control command using the selected second level controller based on the corresponding input data; and control operation of the robot by executing control command.

Abstract: Embodiments disclose a method and system for a scene-aware audio-video representation of a scene. The scene-aware audio video representation corresponds to a graph of nodes connected by edges. A node in the graph is indicative of the video features of an object in the scene. An edge in the graph connecting two nodes indicates an interaction of the corresponding two objects in the scene. In the graph, at least one or more edges are associated with audio features of a sound generated by the interaction of the corresponding two objects. The graph of the audio-video representation of the scene may be used to perform a variety of different tasks. Examples of the tasks include one or a combination of an action recognition, an anomaly detection, a sound localization and enhancement, a noisy-background sound removal, and a system control.

Abstract: An anomaly detector for controlling a system is provided. The system comprises one or multiple tools to perform one or multiple tasks. The anomaly detector collects a feedforward signal indicative of a sequence of control inputs to the plurality of actuators and a feedback signal indicative of a sequence of outputs of the system caused by the plurality of actuators operated based on the sequence of control inputs. The anomaly detector further combines input state variables extracted from the feedforward signal and output state variables extracted from the feedback signal to form a sequence of extended states of the system. The attention model further encodes the sequence of extended states to produce an encoding of each extended state of the sequence of extended states in a latent space. The anomaly detector further detects an anomaly in a current operation of the system based on the encoded sequence of extended states.

Abstract: A controller for controlling an entity is provided. The controller comprises a memory to store a hierarchical multimodal reinforcement learning (RL) neural network, and a processor. The hierarchical multimodal RL neural network includes a first level controller and two second level controllers. Each of the second level controllers comprise a first sub level controller relating to a first modality and a second sub level controller relating to a second modality. The first modality is different from the second modality. The processor is configured to select one of the two second level controllers to perform a first sub-task relating to a task, using the first level controller, based on input data and a state of the hierarchical multimodal RL neural network. The selected second level controller is configured to determine a set of control actions to perform the first sub-task, and control the entity based on the set of control actions.

Abstract: An artificial intelligence (AI) low-latency processing system is provided. The low-latency processing system includes a processor; and a memory having instructions stored thereon. The low-latency processing system is configured to collect a sequence of frames jointly including information dispersed among at least some frames in the sequence of frames, execute a timing neural network trained to identify an early subsequence of frames in the sequence of frames including at least a portion of the information indicative of the information, and execute a decoding neural network trained to decode the information from the portion of the information in the subsequence of frames, wherein the timing neural network is jointly trained with the decoding neural network to iteratively identify the smallest number of subframes from the beginning of a training sequence of frames containing a portion of training information sufficient to decode the training information.

Abstract: An artificial intelligence (AI) system that includes a processor configured to execute modules of the AI system. The modules comprise a feature extractor, an adversarial noise generator, a compressor and a classifier. The feature extractor is trained to process input data to extract features of the input data for classification of the input data. The adversarial noise generator is trained to generate noise data for distribution of features of the input data such that a misclassification rate of corrupted features that include the extracted features corrupted with the generated noise data is greater than a misclassification rate of the extracted features. The compressor is configured to compress the extracted features. The compressed features are closer to the extracted features than to the corrupted features. The classifier is trained to classify the compressed features.

Abstract: An imaging controller is provided for segmenting instances from depth images including objects to be manipulated by a robot. The imaging controller includes an input interface configured to receive a depth image that includes objects, a memory configured to store instructions and a neural network trained to segment instances from the objects in the depth image, and a processor, coupled with the memory, configured to perform the instructions to segment a pickable instance using the trained neural network.

Abstract: Present disclosure discloses an image processing system and method for manipulating two-dimensional (2D) images of three-dimensional (3D) objects of a predetermined class (e.g., human faces). A 2D input image of a 3D object of the predetermined class is manipulated by manipulating physical properties of the 3D object, such as a 3D shape of the 3D input object, an albedo of the 3D input object, a pose of the 3D input object, and lighting illuminating the 3D input object. The physical properties are extracted from the 2D input image using a neural network that is trained to reconstruct the 2D input image. The 2D input image is reconstructed by disentangling the physical properties from pixels of the 2D input image using multiple subnetworks. The disentangled physical properties produced by the multiple subnetworks are combined into a 2D output image using a differentiable renderer.

Abstract: System and method for generating verisimilar images from real depth images. Train a generative adversarial neural network (GAN) by accessing test depth images having identical instances as instances of a real depth image. Input the test depth images in the generator to generate estimated depth images representing an implicit three-dimensional model of the object. Input, each estimated depth image into a discriminator to obtain a loss and into a pose encoder to obtain a matching loss. Iteratively repeat processes until the losses are minimized to a threshold, to end training. Identify the instances in the real image using the trained GAN pose encoder, to produce a pose transformation matrix for each instance in the real image. Identify pixels in the depth images corresponding to the instances of the real image and merge the pixels for the depth images to form an instance segmentation map for the real depth image.

Abstract: A navigation system for providing driving instructions to a driver of a vehicle traveling on a route is provided. The driving instructions are generated by executing a multimodal fusion method that comprises extracting features from sensor measurements, annotating the features with directions for the vehicle to follow the route with respect to objects sensed by the sensors, and encoding the annotated features with a multimodal attention neural network to produce encodings. The encodings are transformed into a common latent space, and the transformed encodings are fused using an attention mechanism producing an encoded representation of the scene. The method further comprises decoding the encoded representation with a sentence generation neural network to generate a driving instruction and submitting the driving instruction to an output device.

Abstract: Present disclosure discloses an image processing system and method for manipulating two-dimensional (2D) images of three-dimensional (3D) objects of a predetermined class (e.g., human faces). A 2D input image of a 3D object of the predetermined class is manipulated by manipulating physical properties of the 3D object, such as a 3D shape of the 3D input object, an albedo of the 3D input object, a pose of the 3D input object, and lighting illuminating the 3D input object. The physical properties are extracted from the 2D input image using a neural network that is trained to reconstruct the 2D input image. The 2D input image is reconstructed by disentangling the physical properties from pixels of the 2D input image using multiple subnetworks. The disentangled physical properties produced by the multiple subnetworks are combined into a 2D output image using a differentiable renderer.

Abstract: Embodiments of the present disclosure discloses a scene-aware video encoder system. The scene-aware encoder system transforms a sequence of video frames of a video of a scene into a spatio-temporal scene graph. The spatio-temporal scene graph includes nodes representing one or multiple static and dynamic objects in the scene. Each node of the spatio-temporal scene graph describes an appearance, a location, and/or a motion of each of the objects (static and dynamic objects) at different time instances. The nodes of the spatio-temporal scene graph are embedded into a latent space using a spatio-temporal transformer encoding different combinations of different nodes of the spatio-temporal scene graph corresponding to different spatio-temporal volumes of the scene. Each node of the different nodes encoded in each of the combinations is weighted with an attention score determined as a function of similarities of spatio-temporal locations of the different nodes in the combination.

Abstract: Embodiments disclose a method and system for a scene-aware audio-video representation of a scene. The scene-aware audio video representation corresponds to a graph of nodes connected by edges. A node in the graph is indicative of the video features of an object in the scene. An edge in the graph connecting two nodes indicates an interaction of the corresponding two objects in the scene. In the graph, at least one or more edges are associated with audio features of a sound generated by the interaction of the corresponding two objects. The graph of the audio-video representation of the scene may be used to perform a variety of different tasks. Examples of the tasks include one or a combination of an action recognition, an anomaly detection, a sound localization and enhancement, a noisy-background sound removal, and a system control.

Abstract: System and method for generating verisimilar images from real depth images. Train a generative adversarial neural network (GAN) by accessing test depth images having identical instances as instances of a real depth image. Input the test depth images in the generator to generate estimated depth images representing an implicit three-dimensional model of the object. Input, each estimated depth image into a discriminator to obtain a loss and into a pose encoder to obtain a matching loss. Iteratively repeat processes until the losses are minimized to a threshold, to end training. Identify the instances in the real image using the trained GAN pose encoder, to produce a pose transformation matrix for each instance in the real image. Identify pixels in the depth images corresponding to the instances of the real image and merge the pixels for the depth images to form an instance segmentation map for the real depth image.

Abstract: A system includes an imager, a processor, and an output module. The imager is configured to provide a plurality of tissue images. The processor is coupled to the imager and is configured to receive the plurality of images. The processor is coupled to a memory. The memory has instructions for determining classification of a region of tissue associated with the plurality of tissue images. Determining classification includes fusing discriminator outputs from a region covariance descriptor and from a normalized color histogram discriminator. The output module is coupled to the processor. The output module is configured to provide a three dimensional representation of the tissue.

Abstract: A scene captioning system is provided. The scene captioning system includes an interface configured to acquire a stream of scene data signals including frames and sound data, a memory to store a computer-executable scene captioning model including a scene encoder, a timing decoder, a timing detector, and a caption decoder, wherein the audio-visual encoder is shared by the timing decoder and the timing detector and the caption decoder, and a processor, in connection with the memory. The processor is configured to perform steps of extracting scene features from the scene data signals by use of the audio-visual encoder, determining a timing of generating a caption by use of the timing detector, wherein the timing is arranged an early stage of the stream of scene data signals, and generating the caption based on the scene features by using the caption decoder according to the timing.

Abstract: Embodiments of the present disclosure disclose an anomaly detector for detecting an anomaly in a sequence of poses of a human performing an activity. The anomaly detector includes an input interface configured to accept input data indicative of a distribution of the sequence of poses, a memory configured to store a discriminative one-class classifier having a pair of complementary classifiers bounding normal distribution of pose sequences in a reproducing kernel Hilbert space (RKHS), a processor configured to embed the input data into an element of the RKHS and classify the embedded data using the discriminative one-class classifier, and an output interface configured to render a classification result.

Abstract: Embodiments of the present disclosure disclose an anomaly detector for detecting an anomaly in a sequence of poses of a human performing an activity. The anomaly detector includes an input interface configured to accept input data indicative of a distribution of the sequence of poses, a memory configured to store a discriminative one-class classifier having a pair of complementary classifiers bounding normal distribution of pose sequences in a reproducing kernel Hilbert space (RKHS), a processor configured to embed the input data into an element of the RKHS and classify the embedded data using the discriminative one-class classifier, and an output interface configured to render a classification result.

Abstract: A computer-implemented method for training a dialogue response generation system and the dialogue response generation system are provided. The method includes arranging a first multimodal encoder-decoder for the dialogue response generation or video description having a first input and a first output, wherein the first multimodal encoder-decoder has been pretrained by training audio-video datasets with training video description sentences, arranging a second multimodal encoder-decoder for dialog response generation having a second input and a second output, providing first audio-visual datasets with first corresponding video description sentences to the first input of the first multimodal encoder-decoder, wherein the first encoder-decoder generates first output values based on the first audio-visual datasets with the first corresponding description sentences, providing the first audio-visual datasets excluding the first corresponding video description sentences to the second multimodal encoder-decoder.