Abstract: Certain aspects of the present disclosure provide techniques and apparatus for improved machine learning. In an example method, an input image is accessed, and the input image is processed using an image encoder to generate an image embedding tensor. The image embedding tensor is processed using a mask decoder machine learning model to generate a set of mask embedding tensors. A textual input is processed using a text encoder to generate a text embedding tensor. A set of augmented masks is generated based on aggregating the text embedding tensor with the set of mask embedding tensors.

Abstract: Embodiments described herein provide an open-vocabulary instance segmentation framework that adopts a pre-trained vision-language model to develop a pipeline in detecting novel categories of instances.