Abstract: In various examples, datasets for training computer vision and/or other artificial intelligence (AI) models may be automatically generated using AI-based techniques. For instance, the systems and methods of the present disclosure may crawl network resources (e.g., webpages) that include information associated with an interactive application (e.g., a wiki page for a gaming application) to obtain “seed” images depicting objects from the interactive application, as well as information (e.g., metadata) corresponding to the seed images. In some examples, a multimodal language model may be used to process the seed images and corresponding information. Based on the processing, the systems and methods may assign one or more labels to the seed images. The labeled seed images may then be used to expand the training dataset by processing and identifying additional images that depict similar objects to those in the labeled seed images.

Abstract: In various examples, techniques for routing queries in multi-backend artificial intelligence systems is described herein. Systems and methods described herein may include a frontend that uses various routing techniques to select backends for routing queries. For instance, the frontend may use at least a keyword routing technique that routes queries to backends based at least on identifiers (e.g., names, codenames, etc.) included in the queries, a classification routing technique that routes queries to backends based at least on performing text classification (e.g., using one or more models, embedding comparisons, etc.), and/or a serialized routing technique that routes queries to backends based at least on an order (e.g., either static or dynamic). In some examples, the backends may be configured to initially respond to the frontend to indicate whether the backends accept or reject queries, such that the frontend is able to reroute queries if needed.

Abstract: In various examples, generative artificial intelligence (AI) models may be used to automatically generate state keys (or “identifiers”) associated with an interactive application. For instance, the systems and methods of the present disclosure may use the AI models to generate the state keys corresponding to essential features of an application. The generated state keys may be structured as key-value pairs, enabling automated application state tracking by mapping values of the essential features to their respective state identifier keys during application runtime. In some examples, the systems and methods of the present disclosure may generate the state keys in a number of ways, such as by using a pretrained AI model with intrinsic knowledge of the application, using a generic AI model to infer the essential features by analyzing a plurality of screenshots captured during application runtime and/or by processing application documentation, wikis, or manuals.

Abstract: Disclosed are apparatuses, systems, and methods for allocating graphics processing unit (GPU) memory resources for executing artificial intelligence (AI) models. The systems and methods can allocate a first portion of memory resources of the GPU to a first set of at least one AI model for a task execution. Upon a need for further data processing, identifying a second set of at least one AI model for continuation of the task execution. The systems and methods can determine that an unallocated portion of the memory resources of the GPU are insufficient for the second set of at least one AI model and can reallocate the first portion of the memory resources of the GPU to the second set of at least one AI model.

Abstract: Disclosed are apparatuses, systems, and techniques for efficient retrieval-augmented generation (RAG) processing that uses flexible content-based document segmentation. The techniques include obtaining, using a processing device, content metadata representative of a distribution of content of a document and segmenting, using the content metadata, the document into a plurality of segments, at least two segments of the plurality of segments having different size. The techniques further include processing, using an embedding model, the plurality of segments to generate a plurality of embeddings and storing the plurality of embeddings in a data store.

Abstract: Disclosed are apparatuses, systems, and techniques for automatically generating commentary to videos that capture sporting activities, computer games, artistic events, political rallies, security-sensitive scenes, and/or any other actions. The techniques include processing a video segment that includes a plurality of video frames, to obtain a description of one or more objects pictured in the video segment and generating, using the obtained description, a prompt for a language model (LM). The techniques further include causing the LM to process the prompt to generate a commentary about an action performed by the one or more objects over a time interval associated with the plurality of video frames.

Abstract: Various examples, systems, and methods are disclosed relating to neural networks for generating images for multiple views while reducing compute usage. A system can perform at least one operation on an image frame in a sequence of frames. The at least one operation can correspond to an image processing pipeline associated with rendering and generation portions of the image frame. The system can select at least one portion of the image frame to render based at least on application data and information from a detector, and can generate remaining portions of the image frame. The system can perform, using at least one machine learning model, interpolation for generating portions of the image frame.

Abstract: Disclosed are apparatuses, systems, and techniques for automatically generating commentary to videos that capture sporting activities, computer games, artistic events, political rallies, security-sensitive scenes, and/or any other actions. The techniques include processing a video segment that includes a plurality of video frames, to obtain a description of one or more objects pictured in the video segment and generating, using the obtained description, a prompt for a language model (LM). The techniques further include causing the LM to process the prompt to generate a commentary about an action performed by the one or more objects over a time interval associated with the plurality of video frames.

Abstract: In various examples, providing virtual assistants for content streaming systems and applications is described herein. For instance, systems and methods are disclosed that use a virtual assistant associated with an application, such as a gaming application, to at least process queries received from a user in order to provide the user with information on how to perform various tasks associated with the application. In some examples, to determine the output information, data associated with the application is processed in order to determine state information describing a current state of the application. Additionally, the query, the state information, and/or additional information may be used to determine contextual information related to the query. One or more language models may then process the query and/or the information to determine the output information associated with the query. The output information may then be provided using various techniques, such as text, graphics, and/or audio.

Abstract: Warp sharding techniques to switch execution between divergent shards on instructions that trigger a long stall, thereby interleaving execution between diverged threads within a warp instead of across warps. The technique may be applied to mitigate pipeline stalls in applications with low warp occupancy and high divergence. Warp data cache locality may also be improved by concentrating memory accesses within a warp rather than spreading them across warps.

Abstract: Accesses between a processor and its external memory is reduced when the processor internally maintains a compressed version of values stored in the external memory. The processor can then refer to the compressed version rather than access the external memory. One compression technique involves maintaining a dictionary on the processor mapping portions of a memory to values. When all of the values of a portion of memory are uniform (e.g., the same), the value is stored in the dictionary for that portion of memory. Thereafter, when the processor needs to access that portion of memory, the value is retrieved from the dictionary rather than from external memory. Techniques are disclosed herein to extend, for example, the capabilities of such dictionary-based compression so that the amount of accesses between the processor and its external memory are further reduced.

Abstract: Some systems compress data utilized by a user mode software without the user mode software being aware of any compression taking place. To maintain that illusion, such systems prevent user mode software from being aware of and/or accessing the underlying compressed states of the data. While such an approach protects proprietary compression techniques used in such systems from being deciphered, such restrictions limit the ability of user mode software to use the underlying compressed forms of the data in new ways. Disclosed herein are various techniques for allowing user-mode software to access the underlying compressed states of data either directly or indirectly. Such techniques can be used, for example, to allow various user-mode software on a single system or on multiple systems to exchange data in the underlying compression format of the system(s) even when the user mode software is unable to decipher the compression format.

Abstract: Accesses between a processor and its external memory is reduced when the processor internally maintains a compressed version of values stored in the external memory. The processor can then refer to the compressed version rather than access the external memory. One compression technique involves maintaining a dictionary on the processor mapping portions of a memory to values. When all of the values of a portion of memory are uniform (e.g., the same), the value is stored in the dictionary for that portion of memory. Thereafter, when the processor needs to access that portion of memory, the value is retrieved from the dictionary rather than from external memory. Techniques are disclosed herein to extend, for example, the capabilities of such dictionary-based compression so that the amount of accesses between the processor and its external memory are further reduced.

Abstract: Warp sharding techniques to switch execution between divergent shards on instructions that trigger a long stall, thereby interleaving execution between diverged threads within a warp instead of across warps. The technique may be applied to mitigate pipeline stalls in applications with low warp occupancy and high divergence. Warp data cache locality may also be improved by concentrating memory accesses within a warp rather than spreading them across warps.

Abstract: Some systems compress data utilized by a user mode software without the user mode software being aware of any compression taking place. To maintain that illusion, such systems prevent user mode software from being aware of and/or accessing the underlying compressed states of the data. While such an approach protects proprietary compression techniques used in such systems from being deciphered, such restrictions limit the ability of user mode software to use the underlying compressed forms of the data in new ways. Disclosed herein are various techniques for allowing user-mode software to access the underlying compressed states of data either directly or indirectly. Such techniques can be used, for example, to allow various user-mode software on a single system or on multiple systems to exchange data in the underlying compression format of the system(s) even when the user mode software is unable to decipher the compression format.

Abstract: Accesses between a processor and its external memory is reduced when the processor internally maintains a compressed version of values stored in the external memory. The processor can then refer to the compressed version rather than access the external memory. One compression technique involves maintaining a dictionary on the processor mapping portions of a memory to values. When all of the values of a portion of memory are uniform (e.g., the same), the value is stored in the dictionary for that portion of memory. Thereafter, when the processor needs to access that portion of memory, the value is retrieved from the dictionary rather than from external memory. Techniques are disclosed herein to extend, for example, the capabilities of such dictionary-based compression so that the amount of accesses between the processor and its external memory are further reduced.

Abstract: Accesses between a processor and its external memory is reduced when the processor internally maintains a compressed version of values stored in the external memory. The processor can then refer to the compressed version rather than access the external memory. One compression technique involves maintaining a dictionary on the processor mapping portions of a memory to values. When all of the values of a portion of memory are uniform (e.g., the same), the value is stored in the dictionary for that portion of memory. Thereafter, when the processor needs to access that portion of memory, the value is retrieved from the dictionary rather than from external memory. Techniques are disclosed herein to extend, for example, the capabilities of such dictionary-based compression so that the amount of accesses between the processor and its external memory are further reduced.

Abstract: Profile-guided optimization is a technique for optimizing execution of computer programs using profile information to improve program runtime performance. Obtaining the profile information can be challenging, especially in live production environments such as high-performance gaming systems. A profiling strategy is provided herein that obtains profile information without requiring extra effort from users. The profiling strategy collects several approximate, lightweight profiles called piecemeal profiles over one or more lifetimes of a computer program, or application. The piecemeal profiles are then used to generate whole program application profiles that can then be used to improve the execution of the application. A piecemeal profile is profile information of a section or portion of an application.

Abstract: A technique selectively avoids memory fetches for partially uniform textures in real time graphics shader programs and instead uses program paths specialized for one or more frequently occurring values. One aspect avoids memory lookups and dependent computations for partially uniform textures through use of pre-constructed coarse-grained representations called value locality maps or dirty tilemaps (DTMs). The decision to use a specialized fast path or not is made dynamically by consulting such coarse-grained dirty tilemap representations. Thread-sharing value reuse can be implemented with or instead of the DTM mechanism.

Abstract: A technique selectively avoids memory fetches for partially uniform textures in real time graphics shader programs and instead uses program paths specialized for one or more frequently occurring values. One aspect avoids memory lookups and dependent computations for partially uniform textures through use of pre-constructed coarse-grained representations called value locality maps or dirty tilemaps (DTMs). The decision to use a specialized fast path or not is made dynamically by consulting such coarse-grained dirty tilemap representations. Thread-sharing value reuse can be implemented with or instead of the DTM mechanism.