Abstract: Provided is a one-shot solution to visual language reasoning. Example systems described herein decompose the challenge of visual language reasoning into two steps: translation of a graphical depiction of data (e.g., a plot or chart) into text; followed by reasoning over the translated text. In particular, example systems described herein can include a machine-learned visual-to-language conversion model that translates a graphical depiction of a dataset to a set of text descriptive of the dataset. The output of visual-to-language conversion model can then be directly used to prompt a language model, (e.g., a pretrained large language model (LLM)), exploiting the few-shot reasoning capabilities of the language model.

Abstract: Provided is a one-shot solution to visual language reasoning. Example systems described herein decompose the challenge of visual language reasoning into two steps: translation of a graphical depiction of data (e.g., a plot or chart) into text; followed by reasoning over the translated text. In particular, example systems described herein can include a machine-learned visual-to-language conversion model that translates a graphical depiction of a dataset to a set of text descriptive of the dataset. The output of visual-to-language conversion model can then be directly used to prompt a language model, (e.g., a pretrained large language model (LLM)), exploiting the few-shot reasoning capabilities of the language model.

Abstract: Systems and methods for generating structured content using a collaborative generator provide a user interface to a user computing system and receive a prompt from the user computing system via the user interface, the prompt including existing content within an integrated development environment. The systems and methods provide the prompt to a generative model, with the generative model being a machine-learned model trained to process language input prompts to generate a language output. The systems and methods receive a generative output generated by the generative model in response to the prompt, the generative output including generative content divided into one or more generative content cells. Additionally, the systems and methods provide the generative output via the user interface.