Abstract: Embodiments herein describe an artificial intelligence (AI) hardware platform that includes at least one integrated circuit (IC) with a systolic array and a self-attention circuit. In one example, the systolic array performs operations in a layer of an AI model that do not use data from previous tokens or data sequences processed by the IC, while the self-attention circuit performs operations in the layer of the AI model that do use data from previous tokens or data sequences processed by the IC.

Abstract: Embodiments herein describe a combined systolic array formed by interconnecting multiple ICs (or chips) each containing individual systolic arrays. In one embodiment, the ICs are interconnected using chip-to-chip connections which couple the local systolic arrays in the ICs to each other, thereby forming a larger, combined systolic array.