Abstract: A technique to compute statistics of data elements include serially inputting the data elements into a compute channel. The compute channel can generate a first running mean and a first running variance associated with data elements of the vector having odd sequence indices, and a second running mean and a second running variance associated with data elements of the vector having even sequence indices. Subsequent to serially inputting data elements into the compute channel, the first running mean and the second running mean are aggregated to generate a mean associated with the data elements of the vector, and the first running variance and the second running variance are aggregated to generate a variance associated with the data elements of the vector.

Abstract: Techniques for compressing a neural network model by mixing compression ratios (sparsity patterns) are described. The weight tensor of a neural network model is divided into weight groups. The pruning cost of compressing the weight values according to a compression ratio is determined for each weight group, and a pruning cost distribution for the compression ratio is generated from the pruning costs of the weight groups. A cost threshold can then be selected from the pruning cost distribution, and weight groups having a pruning cost below the selected cost threshold are compressed according to the compression ratio. The remaining weight groups can be compressed using one or more less aggressive compression ratios. The cost threshold can be adjusted to tune the overall sparsity and accuracy of the compressed neural network.

Abstract: A technique for matching the throughput between writing into and reading from a memory can include receiving, in parallel, computational results in a high precision format for storing into the memory at a first frequency, and storing the computational results in the memory. The technique may further include rounding the computational results using round-to-the-nearest-even or stochastic rounding to down-convert the computational results from the high precision format to a low precision format in parallel, and outputting the computational results in the low precision format in parallel from the memory at a second frequency.

Abstract: A processing engine array is provided with an interconnect mode of operation to use the array as an interconnect to move data elements to different locations in memory such as to perform a matrix transpose operation. In this interconnect mode of operation, although computations are still being performed in the array, the computations are not carried out to modify or change the values of the data elements, but are instead carried out to rearrange the data elements in memory. As such, the computations carried out in the interconnect mode of operation can deviate from the expected behavior of floating-point calculations. A mode selection signal can be used to provide the proper outputs of the processing elements of the array depending on the mode of operation.

Abstract: A technique for packing matrix multiplications for concurrent execution in an integrated circuit device may include obtaining a description of a neural network model, and generating an intermediate representation of the neural network model. Matrix multiplication instructions in the intermediate representation of the neural network model can then be vectorized for concurrent execution on an integrated circuit device, and machine instructions can be generated for the integrated circuit device based on the vectorized matrix multiplication instructions.

Abstract: Techniques to speed up decoding of compressed data objects may include offloading a decoding function to a decoder accelerator. The techniques may include a processor parsing the header including one or more code tables from a compressed data object, loading code data for one or more code tables into the decoder accelerator, and providing the encoded data from the compressed data object to the decoder accelerator via a decoder bus interface. The decoder accelerator decodes the encoded data into decoded data blocks. The processor then receives the decoded data blocks from the decoder accelerator via the decoder bus interface, generates pre-transformation data blocks based on the decoded data blocks by performing inverse domain transformation, and converts the pre-transformation data blocks into a decompressed data object.

Abstract: To provide native hardware support for complex operations such as a beam search operation, an integrated circuit device can be implemented with multiple computational circuit blocks coupled in series to form a pipeline. Each computational circuit block includes an arithmetic logic unit (ALU) circuit having a first numeric input, a second numeric input, a primary result output, and a secondary output. The ALU circuit is programmable to perform an arithmetic operation on the first numeric input and the second numeric input to generate the primary result output, and provide one of the first numeric input or the second numeric input to the secondary output, which is fed back to the ALU circuit.

Abstract: Systems and methods are provided to perform multiply-accumulate operations of reduced precision numbers in a systolic array. Each row of the systolic array can receive reduced inputs from a respective reducer. The reducer can receive a particular input and generate multiple reduced inputs from the input. The reduced inputs can include reduced input data elements and/or a reduced weights. The systolic array may lack support for inputs with a first bit-length and the reducers may reduce the bit-length of a given input from the first bit-length to a second shorter bit-length and provide multiple reduced inputs with second shorter bit-length to the array. The systolic array may perform multiply-accumulate operations on each unique combination of the multiple reduced input data elements and the reduced weights to generate multiple partial outputs. The systolic array may sum the partial outputs to generate the output.

Abstract: A technique for arranging matrix multiplications for concurrent execution in an integrated circuit device may include obtaining a representation of a data dependency graph of a neural network model. The data dependency graph may include having an accumulation group (AG) pack of accumulation groups (AGs), in which each of the AGs has one or more matrix multipartition instructions. A representation of a memory location base partition constraint graph of the AG pack can be generated, and an AG row group constraint graph can be generated based on the memory location base partition constraint graph. The AGs of the AG pack can then be assigned to tiles in an integrated circuit device based on the AG row group constraint graph.

Abstract: Techniques for replacing input values being loaded into a computational circuit are described. Small input values such as denormal numbers can be replaced with alternative values such as zeros to reduce switching activity in the computational circuit, and thus reduce power consumption. In applications such as most neural networks, the impact on the prediction results when replacing small numbers with zeros can be negligible. In applications where high precision computations may be desirable, the input values can be loaded into the computation circuit without modification.

Abstract: Systems and methods are provided to perform multiply-accumulate operations of reduced precision numbers in a systolic array. Each row of the systolic array can receive reduced inputs from a respective reducer. The reducer can receive a particular input and generate multiple reduced inputs from the input. The reduced inputs can include reduced input data elements and/or a reduced weights. The systolic array may lack support for inputs with a first bit-length and the reducers may reduce the bit-length of a given input from the first bit-length to a second shorter bit-length and provide multiple reduced inputs with second shorter bit-length to the array. The systolic array may perform multiply-accumulate operations on each unique combination of the multiple reduced input data elements and the reduced weights to generate multiple partial outputs. The systolic array may sum the partial outputs to generate the output.

Abstract: To improve utilization of a systolic array, each row of the array is provided with two or more general purpose row input buses. Each of the general purpose row input data buses can be operable to transfer either feature map (FMAP) input elements or weight values into the processing elements of the corresponding row of the array. The flexibility of the general purpose row input buses allows weights to be loaded in parallel into the array to speed up weight loading times to reduce the overall latency.

Abstract: A technique to program a compute channel having multiple computational circuit blocks coupled in series in a pipeline can include receiving a machine instruction for the compute channel. The machine instruction is decoded to obtain an opcode, and the opcode can be used as an index to access an opcode entry in an opcode table. The opcode entry contains a pointer to a microoperation, and the pointer can be used to access a microoperation represented by a control entry in a control table and a datapath configuration entry in a datapath table. The microoperation can then be issued to the compute channel by configuring the compute channel with the control entry and the datapath configuration entry.

Abstract: A compute channel can have multiple computational circuit blocks coupled in series to form a pipeline. The compute channel can perform a computation on an input tensor to generate an output tensor based on an instruction. When the computational does not require all of the computational circuit blocks, the throughput of the compute channel can be increased by splitting the data elements of the input tensor into multiple input data streams. The multiple input data streams are provided to respective subsets of one or more computational circuit blocks in the pipeline using bypass circuitry of the computational circuit blocks, and the computation can be performed on multiple input data streams in the respective subsets of one or more computational circuit blocks to generate multiple output data streams corresponding to the output tensor.

Abstract: A compute channel having a compute pipeline of compute stages can be configured using a configuration pipeline with a control table and a datapath table. The control table stores control entries corresponding to respective microoperations, and each control entry includes control information for the compute channel. A datapath table stores datapath configuration entries corresponding to respective microoperations, and each datapath configuration entry has a datapath configuration that includes computational circuit block configurations to configure respective computational circuit blocks in the compute pipeline of the compute channel. Control logic can issue a microoperation to the compute channel by configuring the compute channel according to the control information of the microoperation obtained from the control table, and by inputting the datapath configuration of the microoperation obtained from the datapath table into the configuration pipeline of the compute channel.

Abstract: To improve performance of a computational array, the architecture of the array can be modified to allow the processing engines of a column to operate in parallel and the clock frequency of the array to be increased. The processing engines of each column of the array can be grouped into a series of row groups. The processing engines of each row group can be loaded with input values, and computations on the input values can be carried out in parallel to generate the column output. One or more flip-flop stages can be inserted into the computational logic of each of the processing engines. The computational logic can then be distributed across the flip-flop stages to reduce the propagation delay between flip-flop stages of the processing engine, hence allowing the clock frequency of the array to be increased.

Abstract: A systolic array can implement an architecture tailored to perform matrix multiplications on sparse matrices. Each processing element in the systolic array may include a register configured to store a value, and a multiplexor configured to select an input element from multiple input data buses based on metadata associated with the value. Each processing element may also include a multiplier configured to multiply the selected input element with the value to generate a multiplication result, and an adder configured to add the multiplication result to a partial sum input to generate a partial sum output.

Abstract: In one example, an apparatus comprises: a memory array having an array of memory elements arranged in rows and columns, each memory element being configured to store a data element; and a memory access circuit configured to: perform a row write operation to store a first group of data elements at a first row of the array of memory elements; perform a column read operation at a first column of the array of memory elements to obtain a second group of data elements; and perform a column write operation to store a third group of data elements at the first column of the array of memory elements to replace the second group of data elements.

Abstract: To take advantage of the architecture of a systolic array tailored to perform sparse matrix multiplications, a weight matrix can be converted into a set of constrained fine-grained sparse weight matrices. The conversion process may include receiving a request to perform a matrix multiplication operation with a weight matrix, and determining that the weight matrix satisfies a sparsity condition to convert the weight matrix into a set of constrained fine-grained sparse weight matrices. The weight matrix can then be converted into a set of constrained fine-grained sparse weight matrices. Computer instructions can then be generated for an integrated circuit device to perform the requested matrix multiplication operation as a set of sparse matrix multiplication operations using the set of constrained fine-grained sparse weight matrices.

Abstract: A technique for performing a tensor operation includes inputting concatenated data words of a first input tensor and concatenated data words of a second input tensor into a compute channel having a plurality of compute stages coupled in series. The concatenated data words of the first input tensor and the second input tensor represented in a first datatype can be converted into data elements represented in a second datatype using a first subset of the compute stages. A binary operation can be performed on each data element represented in the second datatype from the first input tensor with a corresponding data element represented in the second datatype from the second input tensor to generate output data elements of an output tensor represented in the second datatype using a second subset of the compute stages. The output data elements of the output tensor can then be outputted from the compute channel.