Abstract: A method of performing computations for artificial intelligence models may include obtaining an input tensor based on an input to an artificial intelligence model and loading the input tensor into multiple processing devices. The input tensor may be split into multiple input tensor tiles that are distributed among the processing devices such that each of the processing devices does not include an entirety of the input tensor. The method may also include performing multiple tensor operations according to the artificial intelligence model to generate multiple intermediate tensors and an output tensor, one or more of the tensor operations performed using the input tensor.

Abstract: A method of performing tensor operations includes loading two tensors into multiple processing devices that are each split into multiple tensor tiles that are distributed among the processing devices. The method may also include performing a first tensor operation with the two tensors to generate a solution tensor that is split into multiple solution tensor tiles distributed among the processing devices and transferring one or more of the solution tensor tiles amongst one or more of the processing devices without any of the processing devices including the entire solution tensor. The method may also include performing a second tensor operation with the solution tensor and another of the tensors, which is split into multiple tensor tiles that are distributed among the processing devices, to generate another solution tensor. The method may further include repeating the steps of transferring and performing the second tensor operation for each remaining tensor.

Abstract: A tensor parallel group including multiple processing devices separated into a first set of two or more of the processing devices and a second set of two or more of the processing devices. The tensor parallel group may also include multiple communication channels to directly communicatively couple every processing device in the first set of the processing devices with every processing device in the second set of the processing devices without communicatively coupling any of the processing devices in the same set of the processing devices. In these and other embodiments, the processing devices may be configured such that each of the processing devices may be able to communicate with any of the other of the processing devices through at most one other of the processing devices.

Abstract: A method of performing computations for artificial intelligence models may include obtaining an input tensor based on an input to an artificial intelligence model and loading the input tensor into multiple processing devices. The input tensor may be split into multiple input tensor tiles that are distributed among the processing devices such that each of the processing devices does not include an entirety of the input tensor. The method may also include performing multiple tensor operations according to the artificial intelligence model to generate multiple intermediate tensors and an output tensor, one or more of the tensor operations performed using the input tensor.

Abstract: A tensor parallel group including multiple processing devices separated into a first set of two or more of the processing devices and a second set of two or more of the processing devices. The tensor parallel group may also include multiple communication channels to directly communicatively couple every processing device in the first set of the processing devices with every processing device in the second set of the processing devices without communicatively coupling any of the processing devices in the same set of the processing devices. In these and other embodiments, the processing devices may be configured such that each of the processing devices may be able to communicate with any of the other of the processing devices through at most one other of the processing devices.

Abstract: Embodiments herein describe using template files for translating an existing (or base) AI model into a new AI model for a model-specific chipset. That is, instead of requiring a developer to use an AI framework to prepare new code for the new AI model, a compiler can receive a template file which indicates a base AI model (e.g., an AI model that has already been executed on the model-specific chipset) and structural parameters for the new AI model. The compiler can use the structural parameters to modify compilation data corresponding to the base AI model. The compiler can then use the modified compilation data to create code for the new AI model that executes on the model-specific chipset.

Abstract: Embodiments herein describe translating specialized functions for one type of hardware platform into executable code for a different type of hardware platform. For example, a specialized function developed for a first hardware platform (e.g., a GPU or CPU) can be translated into an intermediate representation (IR) containing arguments for a second hardware platform (e.g., a model-specific chipset) by a compiler. The compiler can then convert the IR into executable code for the second hardware platform.

Abstract: Embodiments herein describe an artificial intelligence (AI) hardware platform that includes at least one integrated circuit (IC) with a systolic array and a self-attention circuit. In one example, the systolic array performs operations in a layer of an AI model that do not use data from previous tokens or data sequences processed by the IC, while the self-attention circuit performs operations in the layer of the AI model that do use data from previous tokens or data sequences processed by the IC.

Abstract: Embodiments herein describe a combined systolic array formed by interconnecting multiple ICs (or chips) each containing individual systolic arrays. In one embodiment, the ICs are interconnected using chip-to-chip connections which couple the local systolic arrays in the ICs to each other, thereby forming a larger, combined systolic array.