Abstract: Disclosed herein are methods, systems, and computer-readable media for generating image captions for training a machine learning model. Current image generation models are hindered by the prevalence of improper or inaccurate captions, which leads to suboptimal training data. This results in less effective image generation models. Disclosed systems and methods involve obtaining a text-to-image dataset including one or more digital image-caption pairs. Systems and methods involve generating a recaptioned dataset by applying an image captioner model to images in the text-to-image dataset. An image captioner model can be trained with an improved image dataset, a first tuning stage, and a second tuning stage, for improved performance.

Abstract: The present technology pertains to influencing the blending of two visual media inputs by first receiving them through a prompt editor. A blending interface is presented, displaying at least one frame from each of the first and second input visual media. The blending is adjusted in response to user input by modifying a blend curve that represents the relative influence of the first visual media compared to the second visual media over time.

Abstract: The present technology pertains to a visual media generative response engine that can create visual media from prompts. The visual media generative response engine can generate visual media in a variety of durations, aspect ratios, and resolutions. Further, the visual media generative response engine is capable of receiving both visual media and text as prompts. Additionally, the present technology pertains to a variety of user interfaces to enable more influence over the output of the visual media generative response engine.

Abstract: Sanitizing data can be a cumbersome task, particularly when the volume of data is large, the content is sensitive, and/or the type of sanitation requires contextual determinations. Sanitizing large amounts of data is tedious and may often require highly trained personnel with clearances and/or other qualifications. In the systems and methods of the present disclosure, language models (LMs) are used to solve these and other technical issues with tools that may allow sanitizing data easily, with high versatility, context awareness, and/or low demand for computational resources. In particular, some of the disclosed systems and methods use a first language model and a second language model (being less resource-intensive than the first language model) to generate sanitized output data with improved efficiency and accuracy. This dual-model approach ensures that sensitive information is handled appropriately while optimizing computer resource usage.

Abstract: The present technology provides an interaction paradigm whereby a prompt source can continue to interact with the generative response engine through a conversational interface while the generative response engine is processing a task, especially a long-running task. A prompt source can provide additional prompts to modify or clarify the task. The prompt source can also provide additional tasks or subtasks. The generative response engine can also provide intermediate responses in the conversational interface. For example, the generative response engine can respond to prompts provided by the prompt source during the performance of the long-running task. The generative response engine can also determine that it should ask for additional details or clarification, and in response to such a determination, the generative response engine can provide intermediate responses in the conversation interface to encourage further input from the prompt source.

Abstract: The present technology provides for the learning of information relevant to a user account by a generative response engine and accessing this information when preparing personalized responses to prompts provided by the user account. A further aspect of the present technology is that the user account does not need to explicitly tell the generative response engine to remember a particular information. Instead, the present technology is configured such that the generative response engine can learn such facts, preferences, or contexts from conversational prompts provided to the chatbot without providing explicit instructions to remember the data. A further aspect of the present technology is that the user account can request that the generative response engine forget some learned facts too.

Abstract: Sanitizing data can be a cumbersome task, particularly when the volume of data is large, the content is sensitive, and/or the type of sanitation requires contextual determinations. Sanitizing large amounts of data is tedious and may often require highly trained personnel with clearances and/or other qualifications. In the systems and methods of the present disclosure, language models (LMs) are used to solve these and other technical issues with tools that may allow sanitizing data easily, with high versatility, context awareness, and/or low demand for computational resources. In particular, some of the disclosed systems and methods use a first language model and a second language model (being less resource-intensive than the first language model) to generate sanitized output data with improved efficiency and accuracy. This dual-model approach ensures that sensitive information is handled appropriately while optimizing computer resource usage.

Abstract: Disclosed herein are methods and systems for generating metadata from content using one or more machine learning models. In an embodiment, a method may include receiving the content through a graphical user interface associated with the large language model, generating a first file by tokenizing the content into an input format for the large language model and merging the tokenized content with a content instruction, inputting the first file into the large language model, generating, using the large language model, metadata from at least the first file, the metadata reflecting a context associated with the content, generating a second file, the second file comprising the metadata, and displaying the generated metadata on the graphical user interface.

Abstract: Accurate prompting improves the operation, efficiency, and output of computer models, such as language models. The disclosed systems and methods improve interaction with computer models by facilitating the generation of accurate prompts for targeted interactions with computer models. For example, disclosed systems can be configured to store prompts and responses generated with computer models in session memories. The system can use information in session memories to generate combined prompts when receiving prompts that refer to previous prompts or answers. The system improves prompt accuracy by generating combined prompts—formed by combining the context from the stored information (e.g., in context sub-prompts) and instructions in the new prompt (e.g., in instructions sub-prompts). The system can generate responses based on the combined prompts allowing the computer models to leverage the context in previous interactions, without burdensome or complicated prompts, for more meaningful or accurate responses.

Abstract: Disclosed herein are methods, systems, and computer-readable media for integrating an application programming interface (API) with a natural language model user interface. In one embodiment, a method includes receiving a registration of an external API via a user interface connected to a natural language model, the natural language model being configured to integrate a plurality of external APIs, accessing a manifest file hosted at a first online location by a publisher of the external API, the manifest file comprising parameters for interfacing with the external API and a second online location of a specification associated with the external API, the parameters and second online location being defined by the publisher of the external API, accessing the specification at the second online location, and integrating the external API with the natural language model based on data from at least one of the manifest file or the specification.

Abstract: Disclosed embodiments may include a method of interacting with a multimodal machine learning model; the method may include providing a graphical user interface associated with a multimodal machine learning model. The method may further include displaying an image to a user in the graphical user interface. The method may also include receiving a textual prompt from the user and then generating input data using the image and the textual prompt. The method may further include generating an output at least in part by applying the input data to the multimodal machine learning model, the multimodal machine learning model configured using prompt engineering to identify a location in the image conditioned on the image and the textual prompt, wherein the output includes a first location indication. The method may also include displaying, in the graphical user interface, an emphasis indicator at the indicated first location in the image.

Abstract: While AI models, like large language models, are powerful tools with multiple applications, they can be complex to use and can require a lot of resources to operate. The disclosed systems and methods provide tools to generate customized models (or AI agents) that are configured with features like tailored knowledge, capabilities, and instructions that make them faster, more efficient, and use less computational resources. AI agents may offer several technical advantages of improved efficiency, resource use, and connectivity. This disclosure describes systems and methods to configure, evaluate, generate, and deploy the custom models that can more efficiently run specific tasks. Disclosed systems and methods are configured to, for example, receive a query to generate a custom model, generate the AI agent custom model with the information in the query, and then resolve user queries more efficiently using the custom model.

Abstract: While AI models, like large language models, are powerful tools with multiple applications, they can be complex to use and can require a lot of resources to operate. The disclosed systems and methods provide tools to generate customized models (or AI agents) that are configured with features like tailored knowledge, capabilities, and instructions that make them faster, more efficient, and use less computational resources. AI agents may offer several technical advantages of improved efficiency, resource use, and connectivity. This disclosure describes systems and methods to configure, evaluate, generate, and deploy the custom models that can more efficiently run specific tasks. Disclosed systems and methods are configured to, for example, receive a query to generate a custom model, generate the AI agent custom model with the information in the query, and then resolve user queries more efficiently using the custom model.

Abstract: The disclosed embodiments may include a method of interacting with a multimodal machine learning model; the method may include providing a graphical user interface associated with a multimodal machine learning model. The method may further include displaying an image to a user in the graphical user interface. The method may also include receiving a textual prompt from the user and then generating input data using the image and the textual prompt. The method may further include generating an output at least in part by applying the input data to the multimodal machine learning model, the multimodal machine learning model configured using prompt engineering to identify a location in the image conditioned on the image and the textual prompt, wherein the output comprises a first location indication. The method may also include displaying, in the graphical user interface, an emphasis indicator at the indicated first location in the image.

Abstract: Disclosed herein are methods, systems, and computer-readable media for regenerating a region of an image with a machine learning model based on a text input. Disclosed embodiments involve accessing a digital input image. Disclosed embodiments involve generating a masked image by removing a masked region from the input image. Disclosed embodiments involve accessing a text input corresponding to an image enhancement prompt. Disclosed embodiments include providing at least one of the input image, the masked region, or the text input to a machine learning model configured to generate an enhanced image. Disclosed embodiments involve generating, with the machine learning model, the enhanced image based on at least one of the input image, the masked region, or the text input.

Abstract: Disclosed herein are methods, systems, and computer-readable media for integrating a particular external application programming interface (API) with a natural language model user interface. In one embodiment, a method includes receiving a first input at the natural language model user interface, determining the first input includes a request to integrate the particular external application programming interface (API) with the natural language model user interface, identifying the particular external API based on the received input, integrating the particular external API with the natural language model user interface, accessing the particular external API based on the first input or a second input at the natural language model user interface, and transmitting, based on the accessing, a response message to the natural language model user interface, the response message including a result of the accessing.

Abstract: The present technology pertains to a generative response engine that can adapt a user interface provided by its front end to receive inputs in a visual format and to provide visual formats in response to prompts. In some embodiments, the generative response engine can provide a greater variety of outputs that can be rendered by the front end. Collectively, the present technology can render dynamic user interface elements in response to prompts received by the generative response engine. Generative response engines that can provide dynamic and multimodal responses that are appropriate to a task are useful for an increased range of tasks.

Abstract: Disclosed herein are methods, systems, and computer-readable media for generating computer code based on natural language input. In an embodiment, a method may comprise one or more of: receiving a docstring representing natural language text specifying a digital programming result; generating, using a trained machine learning model, and based on the docstring, a computer code sample configured to produce respective candidate results; causing the computer code sample to be executed; identifying, based on the executing, a computer code sample configured to produce a particular candidate result associated with the digital programming result; performing at least one of outputting, via a user interface, the identified computer code sample, compiling the identified computer code sample, transmitting the identified computer code sample to a recipient device, storing the identified computer code sample, and/or re-executing the identified computer code sample.

Abstract: Embodiments of the present disclosure may include systems, methods, and computer readable media for generating a vector representation, including receiving a training data set, the training data set including a plurality of paired data samples corresponding to positive example pairs, each positive example pair including a first data unit and a second data unit. Embodiments may also include converting the training data set into at least one first vector of a vector representation. Embodiments may further include accessing one or more negative example pairs to contrast against the positive example pairs. Embodiments may also include converting the one or more negative example pairs into one or more second vectors of the vector representation. Embodiments may further include training an artificial machine learning model to generate additional vectors of the vector representation.

Abstract: Disclosed herein are methods, systems, and computer-readable media for automatically classifying and moderating content. In an embodiment, a method may include receiving input data and one or more content policies, and generating a content taxonomy. The method may also include receiving multi-domain cold start data and generating training data. The method may also include accessing a language model based on the input data and the training data, and iteratively classifying the content of the input data using the language model and the content taxonomy, refining the training data based on the classified content of the input data, refining the language model based on the refined training data, probing the refined language model, and updating the threshold value based on the probing of the refined language model. The method may also include moderating the content of the input data based on the optimized language model and the content taxonomy.