Abstract: A matrix multiply compute apparatus and method of operation therefor. The apparatus includes a memory configured to store weight matrix elements in a first format including columns of scale factors and mantissa blocks. A first register is configured to receive the scale factors of each weight matrix column via a crossbar coupled to the memory, the converter is configured to determine max exponents using the scale factors of the columns, and a second register is configured to store the max exponents. Also, the first register is configured to receive the mantissa blocks of the columns, and the converter is configured to determine converted blocks using all of the scale factors and the mantissa blocks. A weight buffer receives the converted blocks and max exponents as the elements in a second format that is used by a compute device to determine matrix multiply outputs, which are stored in an output buffer.