Abstract: A method and system are provided a dataflow using a reconfigurable dataflow architecture (RDA) with repeated layered structure having synchronization boundaries between layers for a large language model used in machine learning for artificial intelligence (AI). The method first identifies in the dataflow consecutive calls to a same kernel, wherein the consecutive calls are made in an inference process involving generating a sequence of output tokens based on a given set of input tokens. The method then applies kernel looping to transform the identified consecutive calls in the dataflow into a single call to the kernel by identifying from a list of pattern matched calls ones to include in the single call to the kernel.

Abstract: A system and method provide kernel looping to eliminate synchronization boundaries to achieve peak inference performance in dataflow accelerators. The system provides a dataflow using a reconfigurable dataflow architecture (RDA) with repeated layered structure having synchronization boundaries between layers for a large language model used in machine learning. The method provides kernel looping between layers by first identifying in the dataflow consecutive calls to a same kernel, wherein the consecutive calls are made in an inference process involving generating a sequence of output tokens based on a given set of input tokens. The method then applies kernel looping to transform the identified consecutive calls in the dataflow into a single call to the kernel. The method further modifies the kernel to contain a pipelined outer loop.