Abstract: In general, techniques are described that address the limitations of existing conformal prediction methods for cascaded models. In an example, a method includes receiving a first validation data set for validating performance of an upstream model of the two or more cascaded models and receiving a second validation data set for validating performance of a downstream model of the two or more cascaded models wherein the second validation data set is different than the first validation set; estimating system-level errors caused by predictions of the upstream model based on the first validation data set; estimating system-level errors caused by predictions of the downstream model based on the second validation data set; and generating a prediction confidence interval that indicates a confidence for the system based on the system-level errors caused by predictions of the upstream model and based on the system-level errors caused by predictions of the downstream model.

Abstract: A method, apparatus and system for adapting a language model for understanding domain-specific multimodal content include acquiring domain-specific multimodal content for at least one content domain and applying question/answer pairs to the acquired, domain-specific multimodal content for the at least one content domain to train the language model to learn tasks associated with the domain-specific multimodal content for the at least one domain. As such, the trained language model can be implemented to answer questions directed to the domain-specific multimodal content for the at least one domain.

Abstract: Embodiments of the present principles generally relate to methods, apparatuses and systems for determining a measure of conceptual consistency in large language models for understanding of relevant concepts. In some embodiments, a method for measuring conceptual consistency may include prompting an LLM in order to extract answers to background queries and anchor tasks. The method also includes comparing background knowledge facts for a given anchor task associated with known answers with facts extracted from the LLM to determine an LLM performance. The method also includes determining a background knowledge score and an anchor task score based on the LLM's performance. The method also includes determining a conceptual may include score for the LLM by predicting the anchor task score from the background knowledge score. The method also includes outputting an indication of the conceptual may include score.

Abstract: In general, the disclosure describes techniques for joint spatiotemporal Artificial Intelligence (AI) models that can encompass multiple space and time resolutions through self-supervised learning. In an example, a method includes for each of a plurality of multimodal data, generating, by a computing system, using a first machine learning model, a respective modality feature vector representative of content of the multimodal data, wherein each of the generated modality feature vectors has a different modality; processing, by the computing system, each of generated modality feature vectors with a second machine learning model comprising an encoder model to generate event data comprising a plurality of events and/or activities of interest; and analyzing, by the computing system, the event data to generate anomaly data indicative of detected anomalies in the multimodal data.

Abstract: A method, apparatus and system for lifelong reinforcement learning include receiving features of a task, communicating the task features to a learning system, where the learning system learns or performs a task related to the features based on learning or performing similar previous tasks, determining from the features if the task has changed and if so, communicating the features of the changed task to the learning system, where the learning system learns or performs the changed task based on learning or performing similar previous tasks, automatically annotating feature characteristics of received features including differences between the features of the original task and the features of the changed task to enable the learning system to more efficiently learn or perform at least the changed task, and if the task has not changed, processing the task features of a current task by the learning system to learn or perform the current task.

Abstract: A method, apparatus and system for training an embedding space for content comprehension and response includes, for each layer of a hierarchical taxonomy having at least two layers including respective words resulting in layers of varying complexity, determining a set of words associated with a layer of the hierarchical taxonomy, determining a question answer pair based on a question generated using at least one word of the set of words and at least one content domain, determining a vector representation for the generated question and for content related to the at least one content domain of the question answer pair, and embedding the question vector representation and the content vector representations into a common embedding space where vector representations that are related, are closer in the embedding space than unrelated embedded vector representations. Requests for content can then be fulfilled using the trained, common embedding space.

Abstract: In general, the disclosure describes techniques for implementing an MI-based attack detector. In an example, a method includes training a neural network using training data, applying stochastic quantization to one or more layers of the neural network, generating, using the trained neural network, an ensemble of neural networks having a plurality of quantized members, wherein at least one of weights or activations of each of the plurality of quantized members have different bit precision, and combining predictions of the plurality of quantized members of the ensemble to detect one or more adversarial attacks and/or determine performance of the ensemble of neural networks.

Abstract: A method, apparatus and system for moderating multilingual content data, for example, presented during a communication session include receiving or pulling content data that can include multilingual content, classifying, using a first machine learning system, the content data by projecting the content data into a trained embedding space to determine at least one English-language classification for the content data, and determining, using a second machine learning system, if the content data violates at least one predetermined moderation rule, wherein the second machine learning system is trained to determine from English-language classifications determined by the first machine learning system if the content data violates moderation rules. In some embodiments, the method apparatus and system can further include prohibiting a presentation of the content data related to the at least one English-language classification determined to violate the at least one predetermined moderation rule.

Abstract: A computing system comprising a memory configured to store an artificial intelligence (AI) model and an image, and a computation engine executing one or more processors may be configured to perform the techniques for error-based explanations for AI behavior. The computation engine may execute the AI model to analyze the image to output a result. The AI model may, when analyzing the image to output the result, process, based on data indicative of the result, the image to assign an error score to each image feature extracted from the image, and obtain, based on the error scores, an error map. The AI model may next update, based on the error map and to obtain a first updated image, the image to visually indicate the error score assigned to each of the image features, and output one or more of the error scores, the error map, and the first updated image.

Abstract: Techniques are disclosed for identifying multimodal subevents within an event having spatially-related and temporally-related features. In one example, a system receives a Spatio-Temporal Graph (STG) comprising (1) a plurality of nodes, each node having a feature descriptor that describes a feature present in the event, (2) a plurality of spatial edges, each spatial edge describing a spatial relationship between two of the plurality of nodes, and (3) a plurality of temporal edges, each temporal edge describing a temporal relationship between two of the plurality of nodes. Furthermore, the STG comprises at least one of: (1) variable-length descriptors for the feature descriptors or (2) temporal edges that span multiple time steps for the event. A machine learning system processes the STG to identify the multimodal subevents for the event. In some examples, the machine learning system comprises stacked Spatio-Temporal Graph Convolutional Networks (STGCNs), each comprising a plurality of STGCN layers.

Abstract: A method, apparatus and system for zero shot object detection includes, in a semantic embedding space having embedded object class labels, training the space by embedding extracted features of bounding boxes and object class labels of labeled bounding boxes of known object classes into the space, determining regions in an image having unknown object classes on which to perform object detection as proposed bounding boxes, extracting features of the proposed bounding boxes, projecting the extracted features of the proposed bounding boxes into the space, computing a similarity measure between the projected features of the proposed bounding boxes and the embedded, extracted features of the bounding boxes of the known object classes in the space, and predicting an object class label for proposed bounding boxes by determining a nearest embedded object class label to the projected features of the proposed bounding boxes in the space based on the similarity measures.

Abstract: A method, apparatus and system for comprehension-based question answering using a hierarchical taxonomy include receiving a word-based question, associating the word-based question with a layer of the hierarchical taxonomy, in which the hierarchical taxonomy includes at least two layers, each of the at least two layers including respective words resulting in the at least two layers having varying levels complexity, determining which layer of the at least two layers of the hierarchical taxonomy comprises a layer of complexity one level less than the layer of the hierarchical taxonomy associated with the word-based question, and using a pre-trained language model, answering the word-based question using only words associated with the layer of the at least two layers of the hierarchical taxonomy having the one less level of complexity.

Abstract: A method, apparatus and system for adapting a pre-trained network for application to a different dataset includes arranging at least two different types of active adaptation modules in a pipeline configuration, wherein an output of a previous active adaptation module produces an input for a next active adaptation module in the pipeline in the form of adapted network data until a last active adaptation module, and wherein each of the at least two different types of adaptation modules can be switched on or off, determining at least one respective hyperparameter for each of the at least two different types of active adaptation modules, and applying the at least one respective determined hyperparameter to each of the at least two different types of active adaptation modules for processing received data from the pretrained network to determine an adapted network.

Abstract: A method, apparatus and system for training an embedding space for content comprehension and response includes, for each layer of a hierarchical taxonomy having at least two layers including respective words resulting in layers of varying complexity, determining a set of words associated with a layer of the hierarchical taxonomy, determining a question answer pair based on a question generated using at least one word of the set of words and at least one content domain, determining a vector representation for the generated question and for content related to the at least one content domain of the question answer pair, and embedding the question vector representation and the content vector representations into a common embedding space where vector representations that are related, are closer in the embedding space than unrelated embedded vector representations. Requests for content can then be fulfilled using the trained, common embedding space.

Abstract: A method, apparatus and system for visual grounding of a caption in an image include projecting at least two parsed phrases of the caption into a trained semantic embedding space, projecting extracted region proposals of the image into the trained semantic embedding space, aligning the extracted region proposals and the at least two parsed phrases, aggregating the aligned region proposals and the at least two parsed phrases to determine a caption-conditioned image representation and projecting the caption-conditioned image representation and the caption into a semantic embedding space to align the caption-conditioned image representation and the caption. The method, apparatus and system can further include a parser for parsing the caption into the at least two parsed phrases and a region proposal module for extracting the region proposals from the image.

Abstract: A method, apparatus and system for understanding visual content includes determining at least one region proposal for an image, attending at least one symbol of the proposed image region, attending a portion of the proposed image region using information regarding the attended symbol, extracting appearance features of the attended portion of the proposed image region, fusing the appearance features of the attended image region and features of the attended symbol, projecting the fused features into a semantic embedding space having been trained using fused attended appearance features and attended symbol features of images having known descriptive messages, computing a similarity measure between the projected, fused features and fused attended appearance features and attended symbol features embedded in the semantic embedding space having at least one associated descriptive message and predicting a descriptive message for an image associated with the projected, fused features.

Abstract: Techniques are described for neural networks based on Progressive Neural ODEs (PODEs). In an example, a method to progressively train a neural ordinary differential equation (NODE) model comprises processing, by a machine learning system executed by a computing system, first training data, the first training data having a first complexity, to perform training of a first layer for the NODE model; and after performing the first training, processing second training data, the second training data having a second complexity that is higher than the first complexity, to perform training of a second layer for the NODE model.

Abstract: In some examples, a computer-implemented collaboration assessment model identifies actions of each of two or more individuals depicted in video data, identify, based at least on the identified actions of each of the two or more individuals depicted in the video data, first behaviors at a first collaboration assessment level, identify, based at least on the identified actions of each of the two or more individuals depicted in the video data, second behaviors at a second collaboration assessment level different from the first collaboration assessment level, and generate and output, based at least on the first behaviors at the first collaboration assessment level and the second behaviors at the second collaboration assessment level, an indication of at least one of an assessment of a collaboration effort of the two or more individuals or respective assessments of individual contributions of the two or more individuals to the collaboration effort.

Abstract: In general, the disclosure describes techniques for characterizing a dynamical system and a neural ordinary differential equation (NODE)-based controller for the dynamical system. An example analysis system is configured to: obtain a set of parameters of a NODE model used to implement the NODE-based controller, the NODE model trained to control the dynamical system; determine, based on the set of parameters, a system property of a combined system comprising the dynamical system and the NODE-based controller, the system property comprising one or more of an accuracy, safety, reliability, reachability, or controllability of the combined system; and output the system property to modify one or more of the dynamical system or the NODE-based controller to meet a required specification for the combined system.

Abstract: A method, apparatus and system for zero shot object detection includes, in a semantic embedding space having embedded object class labels, training the space by embedding extracted features of bounding boxes and object class labels of labeled bounding boxes of known object classes into the space, determining regions in an image having unknown object classes on which to perform object detection as proposed bounding boxes, extracting features of the proposed bounding boxes, projecting the extracted features of the proposed bounding boxes into the space, computing a similarity measure between the projected features of the proposed bounding boxes and the embedded, extracted features of the bounding boxes of the known object classes in the space, and predicting an object class label for proposed bounding boxes by determining a nearest embedded object class label to the projected features of the proposed bounding boxes in the space based on the similarity measures.