Abstract: A method for controlling an artificial intelligence (AI) device can include obtaining an input query, an input image, bounding boxes for objects detected in the input image, object labels corresponding to the bounding boxes, and at least one topic label for a word in the input query, generating at least one word embedding for the at least one topic label, and generating a plurality of word embeddings for the object labels corresponding to the bounding boxes. The method can further include generating output attention maps corresponding to scaled dot product attention matrices based on the at least one word embedding for the at least one topic label from the input query and each of the plurality of word embeddings for the object labels, and combining the output attention maps to generate a final attention map corresponding to the at least one topic label from the input query.

Abstract: A method for controlling an artificial intelligence (AI) device can include obtaining, via a processor in the AI device, a knowledge base including a plurality of nodes, and flattening, via the processor, the knowledge base by transforming the knowledge base into a first plurality of documents to form a first index, each of the first plurality of documents identifying a node within the knowledge base. Also, the method can further include receiving, via the processor, a user query, retrieving, via the processor, a subset of documents based on the first plurality of documents and the user query, and performing, via the processor, a task related to knowledge base question answering (KBQA) based on the subset of documents.

Abstract: A method for controlling an artificial intelligence (AI) device can include receiving, via a processor in the AI device, an input image and a query related to the input image, generating, via the processor, an answer prompt template based on the query, the answer prompt template including a sentence containing a mask token located at a position corresponding to an answer within the sentence, and combining the query and the answer prompt template to generate a string of text including the mask token. Also, the method can further include inputting the string of text to a pre-trained mask language module (MLM) and generating a plurality of scores respectfully corresponding to a plurality of answers, each of the plurality of answers being a candidate for replacing the mask token, determining a selected answer among the plurality of answers based on the plurality of scores, and outputting the selected answer.

Abstract: A method for controlling an artificial intelligence (AI) device can include obtaining, via a processor in the AI device, a plurality of universal modules, receiving, via the processor in the AI device, an input image and a query related to the input image, and selecting, via the processor, a group of universal modules from among the plurality of universal modules. Also, the method can further include determining, via the processor, a layout arrangement for the group of universal modules and connecting the group of universal modules together according to the layout arrangement to form a neural module network (NMN), and outputting, via the processor, an answer based on the NMN, the query and the input image.

Abstract: A method for controlling an artificial intelligence (AI) device can include obtaining, via a processor in the AI device, a knowledge base including a plurality of nodes, and flattening, via the processor, the knowledge base by transforming the knowledge base into a first plurality of documents, each of the first plurality of documents identifying a node within the knowledge base. Also, the method can further include receiving, via the processor, a user query, performing, via the processor, matching based on the user query and the first plurality of documents to generate a ranked results list, reducing, via the processor, the ranked results list based on a reducing operation to generate a reduced list, and outputting, via the processor, linked entities based on the reduced list.