Abstract: Provided is a method and system for weight memory mapping for a streaming operation of giant generative artificial intelligence hardware. A weight memory mapping system may include a weight memory configured to store a weight matrix for a pretrained artificial intelligence model; an input register configured to store a plurality of input data; a first hardware operator configured to process a matrix multiplication operation between the plurality of input data and the weight matrix and to compute a lane-level final sum during the progress of the matrix multiplication operation by reusing a partial sum of the matrix multiplication operation; and a second hardware operator configured to preprocess a next matrix multiplication operation during the progress of the matrix multiplication operation using the final sum.

Abstract: Provided is a latency processing unit. The latency processing unit may include a plurality of multiplier-accumulator (MAC) trees configured to perform a matrix product operation for at least one of a plurality of partitions that implement an artificial intelligence (AI) model, streamlined memory access configured to connect each of the plurality of MAC trees to high bandwidth memory in which the at least one partition has been stored through a plurality of channels, a vector execution engine configured to perform an additional operation on results of the operation of the plurality of MAC trees, a local memory unit configured to store the results of the operation of the vector execution engine and an activation value, and an instruction scheduling unit configured to schedule the operations of the plurality of MAC trees and the vector execution engine.