Abstract: An integrated circuit device for vector computations can include multiple computational circuit blocks coupled in series in a pipeline. Each computational circuit block can include an arithmetic logic unit (ALU) circuit having a first numeric input, a second numeric input, and a primary result output. The ALU circuit is programmable to perform a computational operation on the first numeric input and the second numeric input to generate the primary result output. Each computational circuit block can also include an output register to receive the primary result output of the ALU and provide the primary result output of the ALU circuit as a primary output of the computational circuit block, and one or more feedback registers to receive the primary result output of the ALU and provide the primary result output of the ALU circuit to a previous computational circuit block of the pipeline.

Abstract: An integrated circuit device for decoding acceleration can include a table storage subsystem for storing information associated with a code table. The table storage subsystem includes code table storage for storing symbol entries each containing a code symbol, maximum code storage for storing maximum values for different code lengths, and offset storage for storing offset values. Comparators corresponding to the maximum values can concurrently compare input data to each of the maximum values, and a selection circuit can select, based on comparison results of the comparators, an index value from index values derived from the input data and the offset values. The index value can then be used to select a symbol entry from the code table storage for decoding a code in the input data.

Abstract: To improve utilization of a systolic array, each row of the array is provided with a number of row input data buses. Each of the row input data buses can be operable to transfer either feature map (FMAP) input elements or weight values into the processing elements of the corresponding row of the array. By using such row input data buses, concurrent matrix multiplications as well as faster background loading can be achieved in the array.

Abstract: A technique to compute statistics of data elements include serially inputting the data elements into a compute channel. The compute channel can generate a first running mean and a first running variance associated with data elements of the vector having odd sequence indices, and a second running mean and a second running variance associated with data elements of the vector having even sequence indices. Subsequent to serially inputting data elements into the compute channel, the first running mean and the second running mean are aggregated to generate a mean associated with the data elements of the vector, and the first running variance and the second running variance are aggregated to generate a variance associated with the data elements of the vector.

Abstract: Techniques for compressing a neural network model by mixing compression ratios (sparsity patterns) are described. The weight tensor of a neural network model is divided into weight groups. The pruning cost of compressing the weight values according to a compression ratio is determined for each weight group, and a pruning cost distribution for the compression ratio is generated from the pruning costs of the weight groups. A cost threshold can then be selected from the pruning cost distribution, and weight groups having a pruning cost below the selected cost threshold are compressed according to the compression ratio. The remaining weight groups can be compressed using one or more less aggressive compression ratios. The cost threshold can be adjusted to tune the overall sparsity and accuracy of the compressed neural network.

Abstract: A technique for matching the throughput between writing into and reading from a memory can include receiving, in parallel, computational results in a high precision format for storing into the memory at a first frequency, and storing the computational results in the memory. The technique may further include rounding the computational results using round-to-the-nearest-even or stochastic rounding to down-convert the computational results from the high precision format to a low precision format in parallel, and outputting the computational results in the low precision format in parallel from the memory at a second frequency.

Abstract: A processing engine array is provided with an interconnect mode of operation to use the array as an interconnect to move data elements to different locations in memory such as to perform a matrix transpose operation. In this interconnect mode of operation, although computations are still being performed in the array, the computations are not carried out to modify or change the values of the data elements, but are instead carried out to rearrange the data elements in memory. As such, the computations carried out in the interconnect mode of operation can deviate from the expected behavior of floating-point calculations. A mode selection signal can be used to provide the proper outputs of the processing elements of the array depending on the mode of operation.

Abstract: A technique for packing matrix multiplications for concurrent execution in an integrated circuit device may include obtaining a description of a neural network model, and generating an intermediate representation of the neural network model. Matrix multiplication instructions in the intermediate representation of the neural network model can then be vectorized for concurrent execution on an integrated circuit device, and machine instructions can be generated for the integrated circuit device based on the vectorized matrix multiplication instructions.

Abstract: Techniques to speed up decoding of compressed data objects may include offloading a decoding function to a decoder accelerator. The techniques may include a processor parsing the header including one or more code tables from a compressed data object, loading code data for one or more code tables into the decoder accelerator, and providing the encoded data from the compressed data object to the decoder accelerator via a decoder bus interface. The decoder accelerator decodes the encoded data into decoded data blocks. The processor then receives the decoded data blocks from the decoder accelerator via the decoder bus interface, generates pre-transformation data blocks based on the decoded data blocks by performing inverse domain transformation, and converts the pre-transformation data blocks into a decompressed data object.

Abstract: To provide native hardware support for complex operations such as a beam search operation, an integrated circuit device can be implemented with multiple computational circuit blocks coupled in series to form a pipeline. Each computational circuit block includes an arithmetic logic unit (ALU) circuit having a first numeric input, a second numeric input, a primary result output, and a secondary output. The ALU circuit is programmable to perform an arithmetic operation on the first numeric input and the second numeric input to generate the primary result output, and provide one of the first numeric input or the second numeric input to the secondary output, which is fed back to the ALU circuit.

Abstract: Systems and methods are provided to perform multiply-accumulate operations of reduced precision numbers in a systolic array. Each row of the systolic array can receive reduced inputs from a respective reducer. The reducer can receive a particular input and generate multiple reduced inputs from the input. The reduced inputs can include reduced input data elements and/or a reduced weights. The systolic array may lack support for inputs with a first bit-length and the reducers may reduce the bit-length of a given input from the first bit-length to a second shorter bit-length and provide multiple reduced inputs with second shorter bit-length to the array. The systolic array may perform multiply-accumulate operations on each unique combination of the multiple reduced input data elements and the reduced weights to generate multiple partial outputs. The systolic array may sum the partial outputs to generate the output.

Abstract: A technique for arranging matrix multiplications for concurrent execution in an integrated circuit device may include obtaining a representation of a data dependency graph of a neural network model. The data dependency graph may include having an accumulation group (AG) pack of accumulation groups (AGs), in which each of the AGs has one or more matrix multipartition instructions. A representation of a memory location base partition constraint graph of the AG pack can be generated, and an AG row group constraint graph can be generated based on the memory location base partition constraint graph. The AGs of the AG pack can then be assigned to tiles in an integrated circuit device based on the AG row group constraint graph.

Abstract: Techniques for replacing input values being loaded into a computational circuit are described. Small input values such as denormal numbers can be replaced with alternative values such as zeros to reduce switching activity in the computational circuit, and thus reduce power consumption. In applications such as most neural networks, the impact on the prediction results when replacing small numbers with zeros can be negligible. In applications where high precision computations may be desirable, the input values can be loaded into the computation circuit without modification.

Abstract: Systems and methods are provided to perform multiply-accumulate operations of reduced precision numbers in a systolic array. Each row of the systolic array can receive reduced inputs from a respective reducer. The reducer can receive a particular input and generate multiple reduced inputs from the input. The reduced inputs can include reduced input data elements and/or a reduced weights. The systolic array may lack support for inputs with a first bit-length and the reducers may reduce the bit-length of a given input from the first bit-length to a second shorter bit-length and provide multiple reduced inputs with second shorter bit-length to the array. The systolic array may perform multiply-accumulate operations on each unique combination of the multiple reduced input data elements and the reduced weights to generate multiple partial outputs. The systolic array may sum the partial outputs to generate the output.

Abstract: To improve utilization of a systolic array, each row of the array is provided with two or more general purpose row input buses. Each of the general purpose row input data buses can be operable to transfer either feature map (FMAP) input elements or weight values into the processing elements of the corresponding row of the array. The flexibility of the general purpose row input buses allows weights to be loaded in parallel into the array to speed up weight loading times to reduce the overall latency.

Abstract: A technique to program a compute channel having multiple computational circuit blocks coupled in series in a pipeline can include receiving a machine instruction for the compute channel. The machine instruction is decoded to obtain an opcode, and the opcode can be used as an index to access an opcode entry in an opcode table. The opcode entry contains a pointer to a microoperation, and the pointer can be used to access a microoperation represented by a control entry in a control table and a datapath configuration entry in a datapath table. The microoperation can then be issued to the compute channel by configuring the compute channel with the control entry and the datapath configuration entry.

Abstract: A compute channel can have multiple computational circuit blocks coupled in series to form a pipeline. The compute channel can perform a computation on an input tensor to generate an output tensor based on an instruction. When the computational does not require all of the computational circuit blocks, the throughput of the compute channel can be increased by splitting the data elements of the input tensor into multiple input data streams. The multiple input data streams are provided to respective subsets of one or more computational circuit blocks in the pipeline using bypass circuitry of the computational circuit blocks, and the computation can be performed on multiple input data streams in the respective subsets of one or more computational circuit blocks to generate multiple output data streams corresponding to the output tensor.

Abstract: A compute channel having a compute pipeline of compute stages can be configured using a configuration pipeline with a control table and a datapath table. The control table stores control entries corresponding to respective microoperations, and each control entry includes control information for the compute channel. A datapath table stores datapath configuration entries corresponding to respective microoperations, and each datapath configuration entry has a datapath configuration that includes computational circuit block configurations to configure respective computational circuit blocks in the compute pipeline of the compute channel. Control logic can issue a microoperation to the compute channel by configuring the compute channel according to the control information of the microoperation obtained from the control table, and by inputting the datapath configuration of the microoperation obtained from the datapath table into the configuration pipeline of the compute channel.

Abstract: To improve performance of a computational array, the architecture of the array can be modified to allow the processing engines of a column to operate in parallel and the clock frequency of the array to be increased. The processing engines of each column of the array can be grouped into a series of row groups. The processing engines of each row group can be loaded with input values, and computations on the input values can be carried out in parallel to generate the column output. One or more flip-flop stages can be inserted into the computational logic of each of the processing engines. The computational logic can then be distributed across the flip-flop stages to reduce the propagation delay between flip-flop stages of the processing engine, hence allowing the clock frequency of the array to be increased.

Abstract: A systolic array can implement an architecture tailored to perform matrix multiplications on sparse matrices. Each processing element in the systolic array may include a register configured to store a value, and a multiplexor configured to select an input element from multiple input data buses based on metadata associated with the value. Each processing element may also include a multiplier configured to multiply the selected input element with the value to generate a multiplication result, and an adder configured to add the multiplication result to a partial sum input to generate a partial sum output.