Abstract: Computer implemented methods, systems, and computer program products include program code executing on a processor(s) which obtain a large language model (LLM) for execution. The program code partitions the LLM, where each partition comprises a transformer layer of one or more transformer layers, and wherein the one or more transformer layers are situation between an embedding layer of the LLM and an output layer of the LLM, where layers of the LLM comprise the one or more transformer layers, the embedding layer, and the output layer. The program code performs the inferences with activation-forwarding at each of the one or more transformer layers, the inferences with activation-forwarding.

Abstract: Learned step size quantization in artificial neural network is provided. In various embodiments, a system comprises an artificial neural network and a computing node. The artificial neural network comprises: a quantizer having a configurable step size, the quantizer adapted to receive a plurality of input values and quantize the plurality of input values according to the configurable step size to produce a plurality of quantized input values, at least one matrix multiplier configured to receive the plurality of quantized input values from the quantizer and to apply a plurality of weights to the quantized input values to determine a plurality of output values having a first precision, and a multiplier configured to scale the output values to a second precision.

Abstract: Learned step size quantization in artificial neural network is provided. In various embodiments, a system comprises an artificial neural network and a computing node. The artificial neural network comprises: a quantizer having a configurable step size, the quantizer adapted to receive a plurality of input values and quantize the plurality of input values according to the configurable step size to produce a plurality of quantized input values, at least one matrix multiplier configured to receive the plurality of quantized input values from the quantizer and to apply a plurality of weights to the quantized input values to determine a plurality of output values having a first precision, and a multiplier configured to scale the output values to a second precision.