Abstract: The hybrid-head architecture model can be used to train a language model (LM). It uses a combination of attention heads and state space models (SSMs) to improve the speed and efficiency of inferencing a received input sequence. This disclosure combines the high-resolution recall capabilities of attention heads with the efficient context summarization of SSM heads. The model can be separated into a set of layers, and the input sequence can be processed layer by layer. Each layer can have its own number of attention heads and SSM heads. Fine-tuning and optimization can be applied to each layer, as well as normalization and scaling. To further optimize the performance of the hybrid-head architecture model, learnable meta tokens can be used, which act as a learned cache for attention and SSM heads, enhancing the model's focus on salient information. The attention heads and the SSMs can be processed in parallel.

Abstract: Multimodal large language models (MLLMs) have evolved to interpret visual elements, progressing from text prompts for holistic image understanding to sophisticated approaches for region-level understanding. However, a key limitation of existing methods is the reliance on representations that may not consistently capture regions across frames, particularly when aiming for a unified solution for both images and videos. The present disclosure unifies image and video region-level understanding by an LLM via token marks.

Abstract: Image inpainting aims to restore damaged regions of a target image. Because any plausible outcome could be considered valid for this task, reference-based image inpainting has been used in which a reference image (e.g. capturing substantially the same scene as the target image) guides the inpainting process, thereby increasing the probability that the target image is restored to its original state. However, current diffusion models used for image inpainting, even though conditioned on reference images, lack direct awareness of the relationships between the target and reference which results in a loss of faithfulness in the inpainted result. The present disclosure guide the inpainting process of a diffusion model with reference-target image correspondences as constraints, which can preserve the reference-target geometric relationships and thus enhance faithfulness of the inpainted target image to the reference image.

Abstract: Large language models (LLMs) learn via machine learning to understand and generate human-like text, and thus are power when used for various language-based tasks, such as text summarization, translation, and content generation. However, to provide superior performance, the LLM is often of a considerable model size and requires high inference costs. To mitigate the size and execution costs of LLMs, methods have been developed to specifically compress LLMs. However, most existing methods either incur significant accuracy degradation compared to uncompressed models or have high training time, while their adaptability is often constrained by a limited range of hardware-supported compression formats. The present disclosure provides error compensation for a compressed LLM in a training free manner that provides flexibility for diverse performance needs.

Abstract: Apparatuses, systems, and techniques to generate annotations for at least one three-dimensional representation corresponding to a scene based at least in part on at least one two-dimensional image depicting the scene. In at least one embodiment, a set of scene-level labels associated with at least one training scene are used to weakly supervise training of one or more neural networks used to generate the annotations.

Abstract: The processes fine-tune vision-language models (VLMs) on large-scale image caption datasets to amend VLM text feature vectors of attribute-neutral descriptions given attribute-neutralization lists, such that the attribute-neutral descriptions are equidistant to those of attribute-specific descriptions using annotation-free debiasing loss without using attribute labels. Feature vectors for attribute-neutral descriptions can be debiased, whereas the attribute-specific descriptions retain the original information. One or more attribute groups can be used for the attribute-neutralization. There can be more than one VLM, such as for different human languages or different human cultures where some biasing can want to be retained. The processes can be applied to any image group, such as objects, animals, plants, rocks, or other object types, where there is at least one attribute group that contains at least two attributes for neutralization.

Abstract: Referring Video Object Segmentation (RVOS) aims to segment an object referred to by a sentence query throughout an entire video. In contrast to Referring Image Segmentation (RIS), RVOS is particularly faced with dynamic visual challenges, such as position and size variation, pose deformation, object occlusion or exit, and scene variation. Moreover, the referring sentence may contain long-term motions or actions, which may not be easily recognized from a single frame. Existing works that address this challenging task generally require end-to-end training for vision-language models, which can be computationally expensive and time-consuming, while the requirement of dense mask annotations for training impedes the scalability of those approaches. The present disclosure uses grounded prompting to adapt image-based segmentation models to video object segmentation tasks, which can be achieved with relying only on weak supervision.

Abstract: Imitation learning, or artificial intelligence-based learning from demonstration, aims to acquire an agent policy by observing and mimicking the behavior demonstrated in expert demonstrations. Imitation learning can be used to generate reliable and robust learned policies in a variety of tasks involving sequential decision-making, such as autonomous driving and robotics tasks. However, current imitation learning solutions are limited in their ability to generalize states or goals unseen from the expert's demonstrations. The present disclosure integrates a diffusion model into generative adversarial imitation learning, which, in terms of prior solutions, can provide superior performance in generalizing to states or goals unseen from the expert's demonstrations, provide data efficiency for varying the amounts of available expert data, and capture more robust and smoother rewards.

Abstract: Embodiments of the present disclosure relate to fine-tuning a neural network model using a weight-decomposed low-rank adaptation (DoRA). DoRA reduces the number of parameters that are fine-tuned, thereby reducing memory and the time needed to fine-tune the parameters. the Pre-trained weights are decomposed into two components, magnitude and direction, which are separately fine-tuned. The magnitude components are fine-tuned while the direction components remain unchanged (frozen). Then low-rank adaptation (LoRA) is used to fine-tune the direction components, efficiently minimizing the number of trainable parameters. Compared with using LoRA to fine-tune the weights directly, using DoRA exhibits a closer resemblance to full fine-tuning's learning behavior and improves upon LoRA in commonsense reasoning and visual instruction tuning tasks. By employing DoRA, both the learning capacity and training stability of LoRA is enhanced.

Abstract: Semantic segmentation generally refers to a machine learning process that associates a label or category with every pixel in an image. This can be used to recognize a collection of pixels that form distinct categories of objects, which may have applications in autonomous driving for example where the vehicle needs to identify other vehicles, pedestrians, traffic signs, pavement, and other road features from captured images of a surrounding environment. While using pixel-level annotations may be ideal for fully-supervised training of the semantic segmentation model, collecting such annotations is time-consuming and expensive, and therefore limits the scalability and practicality of fully-supervised training methods. The present disclosure enables weakly supervised training of a semantic segmentation model aided by learned prompts embedded with semantic knowledge.

Abstract: A domain adaptation module is used to optimize a first domain derived from a second domain using respective outputs from respective parallel hidden layers of the domains.

Abstract: A domain adaptation module is used to optimize a first domain derived from a second domain using respective outputs from respective parallel hidden layers of the domains.

Abstract: A domain adaptation module is used to optimize a first domain derived from a second domain using respective outputs from respective parallel hidden layers of the domains.

Abstract: According to a network space search method, an expanded search space is partitioned into multiple network spaces. Each network space includes a plurality of network architectures and is characterized by a first range of network depths and a second range of network widths. The performance of the network spaces is evaluated by sampling respective network architectures with respect to a multi-objective loss function. The evaluated performance is indicated as a probability associated with each network space. The method then identifies a subset of the network spaces that has the highest probabilities, and selects a target network space from the subset based on model complexity.

Abstract: A domain adaptation module is used to optimize a first domain derived from a second domain using respective outputs from respective parallel hidden layers of the domains.

Abstract: Embodiments herein treat the action segmentation as a domain adaption (DA) problem and reduce the domain discrepancy by performing unsupervised DA with auxiliary unlabeled videos. In one or more embodiments, to reduce domain discrepancy for both the spatial and temporal directions, embodiments of a Mixed Temporal Domain Adaptation (MTDA) approach are presented to jointly align frame-level and video-level embedded feature spaces across domains, and, in one or more embodiments, further integrate with a domain attention mechanism to focus on aligning the frame-level features with higher domain discrepancy, leading to more effective domain adaptation. Comprehensive experiment results validate that embodiments outperform previous state-of-the-art methods. Embodiments can adapt models effectively by using auxiliary unlabeled videos, leading to further applications of large-scale problems, such as video surveillance and human activity analysis.

Abstract: Embodiments herein treat the action segmentation as a domain adaption (DA) problem and reduce the domain discrepancy by performing unsupervised DA with auxiliary unlabeled videos. In one or more embodiments, to reduce domain discrepancy for both the spatial and temporal directions, embodiments of a Mixed Temporal Domain Adaptation (MTDA) approach are presented to jointly align frame-level and video-level embedded feature spaces across domains, and, in one or more embodiments, further integrate with a domain attention mechanism to focus on aligning the frame-level features with higher domain discrepancy, leading to more effective domain adaptation. Comprehensive experiment results validate that embodiments outperform previous state-of-the-art methods. Embodiments can adapt models effectively by using auxiliary unlabeled videos, leading to further applications of large-scale problems, such as video surveillance and human activity analysis.

Abstract: A domain adaptation module is used to optimize a first domain derived from a second domain using respective outputs from respective parallel hidden layers of the domains.

Abstract: A domain adaptation module is used to optimize a first domain derived from a second domain using respective outputs from respective parallel hidden layers of the domains.

Abstract: A domain adaptation module is used to optimize a first domain derived from a second domain using respective outputs from respective parallel hidden layers of the domains.