Abstract: To enable an accelerator unit to perform one or more operators for a machine-learning model, a processing system is configured to generate a launch kernel using a dynamic operator dispatch mode. For example, a processing unit of the processing system first organizes an operator group of the machine-learning model into a series of nodes that represents the operators in the operator group. Based on this series of nodes, the processing unit retrieves and modifies pre-compiled operators from an operator library stored in a memory of the processing system. The processing unit then generates a launch kernel based on the modified pre-compiled operators.

Abstract: A processing system configured to implement a large language model (LLM) includes an accelerator unit (AU) having hardware configured to perform matrix multiplication operations for the LLM using sets of predetermined matrix dimensions. Further, to help optimize the LLM for the processing system, the processing system includes a processor that modifies one or more matrix multiplication operations of the LLM based the sets of predetermined matrix dimensions supported by the hardware of the AU. The processor then recompiles the LLM using the modified multiplication operations and implements the recompiled LLM.

Abstract: Scheduling kernels on a system with heterogeneous compute circuits includes receiving, by a hardware processor, a plurality of kernels and a graph including a plurality of nodes corresponding to the plurality of kernels. The graph defines a control flow and a data flow for the plurality of kernels. The kernels are implemented within different ones of a plurality of compute circuits coupled to the hardware processor. A set of buffers for performing a job for the graph are allocated based, at least in part, on the data flow specified by the graph. Different ones of the kernels as implemented in the compute circuits are invoked based on the control flow defined by the graph.

Abstract: Scheduling work of a machine learning application includes instantiating kernel objects by a computer processor in response to input of kernel definitions. Each kernel object is of a kernel type indicating a compute circuit. The computer processor generates a graph in a memory. Each node represents a task and specifies an assignment of the task to one or more of the kernel objects, and each edge represents a data dependency. Task queues are created in the memory and assigned to queue tasks represented by the nodes. Kernel objects are assigned to the task queues, and the tasks are enqueued by threads executing the kernel objects, based on assignments of the kernel objects to the task queues and assignments of the tasks to the kernel objects. Tasks are dequeued by the threads, and the compute circuits are activated to initiate processing of the dequeued tasks.

Abstract: Examples herein describe a method for a compiler and hardware-abstraction-layer architecture for a programmable integrated circuit (IC). In one embodiment, a method for mapping and porting a neural network to an integrated circuit (IC) is disclosed. The method includes receiving a network description of the neural network; generating a framework independent network graph based on the network description; performing a plurality of back-end operations on the network graph to generate an execution sequence vector; and configuring the IC based on the execution sequence vector.