Abstract: Some embodiments provide an integrated circuit (IC). The IC includes a microprocessor circuit for loading configuration for a neural network and generating instructions for executing the neural network based on the configuration. The IC includes a neural network inference circuit for executing a neural network for input data according to instructions received from the central processing circuit. The IC includes an input processing circuit for receiving data and preparing input data for the neural network inference circuit. The IC includes a unified memory accessible by the microprocessor circuit, neural network inference circuit, and input processing circuit.

Abstract: Some embodiments provide a neural network inference circuit (NNIC) for executing a network having multiple layers. The NNIC includes multiple circuit sets. Each circuit set includes a dot product circuit to compute dot products between weight values and activation values for at least a subset of a first set of the layers, a math function circuit to compute values based on computations using activation values for at least a subset of a second set of the layers, and a post-processing circuit to receive (i) values output by the dot product circuit and (ii) values output by the math function circuit and to perform post-processing operations on the received values. The NNIC includes a set of accumulation circuits. Each accumulation circuit is to accumulate outputs of math function circuits for layers of the second set of layers that perform matrix multiplication of sets of activation values output by previous layers.

Abstract: Some embodiments provide a neural network inference circuit for executing a neural network that includes computation nodes. Each respective computation node of a set of the computation nodes includes (i) a respective linear function that includes a respective dot product of input values for the computation node and weight values for the computation node and (ii) a respective non-linear activation function. The neural network inference circuit includes a set of dot product circuits to compute the dot product for a computation node and a post-processing circuit to compute (i) a result of the linear function for the computation node based on the dot product for the computation node and (ii) an output for the computation node by applying a piecewise linear function to the result of the linear function for the computation node to apply the non-linear activation function for the computation node.

Abstract: Some embodiments provide a neural network inference circuit (NNIC) for executing a neural network (NN) that includes computation nodes at multiple layers. The NNIC includes a set of processing circuits for executing the computation nodes of the NN, a set of memories for storing data used by the processing circuits to execute the NN layers, and a read controller for retrieving the data from the memories for use by the processing circuits. The data is stored in the memories as multiple varying-size blocks. The read controller receives read instructions for a requested block of data to be used by the processing circuits for one or more computation nodes. The read instructions include a base memory address for multiple blocks of data, a size of the requested block of data, and a location of the requested block of data within the multiple blocks of data.

Abstract: A bonded structure is disclosed. The bonded structure can include a first element that has a first plurality of contact pads. The first plurality of contact pads includes a first contact pad and a second redundant contact pad. The bonded structure can also include a second element directly bonded to the first element without an intervening adhesive. The second element has a second plurality of contact pads. The second plurality of contact pads includes a third contact pad and a fourth redundant contact pad. The first contact pad is configured to connect to the third contact pad. The second contact pad is configured to connect to the fourth contact pad. The bonded structure can include circuitry that has a first state in which an electrical signal is transferred to the first contact pad and a second state in which the electrical signal is transferred to the second contact pad.

Abstract: A bonded structure is disclosed. The bonded structure can include a first element that has a first plurality of contact pads. The first plurality of contact pads includes a first contact pad and a second redundant contact pad. The bonded structure can also include a second element directly bonded to the first element without an intervening adhesive. The second element has a second plurality of contact pads. The second plurality of contact pads includes a third contact pad and a fourth redundant contact pad. The first contact pad is configured to connect to the third contact pad. The second contact pad is configured to connect to the fourth contact pad. The bonded structure can include circuitry that has a first state in which an electrical signal is transferred to the first contact pad and a second state in which the electrical signal is transferred to the second contact pad.

Abstract: Some embodiments provide an IC for implementing a machine-trained network with multiple layers. The IC includes a set of circuits to compute a dot product of (i) a first number of input values computed by other circuits of the IC and (ii) a set of predefined weight values, several of which are zero, with a weight value for each of the input values. The set of circuits includes (i) a dot product computation circuit to compute the dot product based on a second number of inputs and (ii) for each input value, at least two sets of wires for providing the input value to at least two of the dot product computation circuit inputs. The second number is less than the first number. Each input value with a corresponding weight value that is not equal to zero is provided to a different one of the dot product computation circuit inputs.

Abstract: Some embodiments provide an IC for implementing a machine-trained network with multiple layers. The IC includes a set of circuits to compute a dot product of (i) a first number of input values computed by other circuits of the IC and (ii) a set of predefined weight values, several of which are zero, with a weight value for each of the input values. The set of circuits includes (i) a dot product computation circuit to compute the dot product based on a second number of inputs and (ii) for each input value, at least two sets of wires for providing the input value to at least two of the dot product computation circuit inputs. The second number is less than the first number. Each input value with a corresponding weight value that is not equal to zero is provided to a different one of the dot product computation circuit inputs.

Abstract: Some embodiments provide a method for a circuit that executes a neural network including multiple nodes. The method loads a set of weight values for a node into a set of weight value buffers, a first set of bits of each input value of a set of input values for the node into a first set of input value buffers, and a second set of bits of each of the input values into a second set of input value buffers. The method computes a first dot product of the weight values and the first set of bits of each input value and a second dot product of the weight values and the second set of bits of each input value. The method shifts the second dot product by a particular number of bits and adds the first dot product with the bit-shifted second dot product to compute a dot product for the node.

Abstract: Some embodiments provide a neural network inference circuit (NNIC) for executing a neural network that includes multiple computation nodes at multiple layers. The NNIC includes multiple core circuits including memories for storing input values for the computation nodes. The NNIC includes a set of post-processing circuits for computing output values of the computation nodes. The output values for a first layer are for storage in the core circuits as input values for a second layer. The NNIC includes an output bus that connects the post-processing circuits to the core circuits. The output bus is for (i) receiving a set of output values from the post-processing circuits, (ii) transporting the output values of the set to the core circuits based on configuration data specifying a core circuit at which each of the output values is to be stored, and (iii) aligning the output values for storage in the core circuits.

Abstract: Some embodiments provide a method that receives a specification of a neural network for execution by an integrated circuit. The integrated circuit includes a neural network inference circuit for executing the neural network to generate an output based on an input, an input processing circuit for providing the input to the neural network inference circuit, a microprocessor circuit for controlling the neural network inference circuit and the input processing circuit, and a unified memory accessible by the microprocessor circuit, the neural network inference circuit, and the input processing circuit. The method determines usage of the unified memory by the neural network inference circuit while executing the neural network. Based on the determined usage by the neural network inference circuit, the method allocates portions of the unified memory to the microprocessor circuit and input processing circuit.

Abstract: Some embodiments provide a neural network inference circuit (NNIC) for executing a neural network that includes multiple computation nodes at multiple layers. The NNIC includes a set of clusters of core computation circuits and a channel, connecting the core computation circuits, that includes separate segments corresponding to each of the clusters. The NNIC includes a fabric controller circuit, a cluster controller circuit for each of the clusters, and a core controller circuit for each of the core computation circuits. The fabric controller circuit receives high-level neural network instructions from a microprocessor and parses the high-level neural network instructions.

Abstract: Some embodiments provide a neural network inference circuit for executing a neural network that includes multiple layers of computation nodes. At least a subset of the layers include non-convolutional layers. The neural network inference circuit includes multiple cores with memories that store input values for the layers. The cores are grouped into multiple clusters. For each cluster, the neural network inference circuit includes a set of processing circuits for receiving input values from the cores of the cluster and executing the computation nodes of the non-convolutional layers.

Abstract: Some embodiments provide a method for executing a neural network. The method writes a first input to a first set of physical memory banks in a unified memory shared by an input processing circuit and a neural network inference circuit that executes the neural network. While the neural network inference circuit is executing the network a first time to generate a first output for the first input, the method writes a second input to a second set of physical memory banks in the unified memory. The neural network inference circuit executes a same set of instructions to read the first input from the first set of memory banks in order to execute the network the first time and to read the second input from the second set of memory banks in order to execute the network a second time to generate a second output for the second input.

Abstract: For a neural network inference circuit that executes a neural network including multiple computation nodes at multiple layers for which data is stored in a plurality of memory banks, some embodiments provide a method for dynamically putting memory banks into a sleep mode of operation to conserve power. The method tracks the accesses to individual memory banks and, if a certain number of clock cycles elapse with no access to a particular memory bank, sends a signal to the memory bank indicating that it should operate in a sleep mode. Circuit components involved in dynamic memory sleep, in some embodiments, include a core RAM pipeline, a core RAM sleep controller, a set of core RAM bank select decoders, and a set of core RAM memory bank wrappers.

Abstract: Some embodiments provide a method for a neural network inference circuit that executes a neural network including multiple computation nodes at multiple layers. Each computation node of a set of the computation nodes includes a dot product of input values and weight values. The method reads a set of encoded weight data for a set of weight values from a memory of the neural network inference circuit. The method decodes the encoded weight data to generate decoded weight data for the set of weight values. The method stores the decoded weight data in a buffer. The method uses the decoded weight data to execute a set of computation nodes. Each computation node of the set of computation nodes includes a dot product between the set of weight values and a different set of input values.

Abstract: Some embodiments provide a neural network inference circuit (NNIC) for executing a neural network that includes multiple computation nodes at multiple layers. The NNIC includes multiple core circuits including memories for storing input values for the computation nodes. The NNIC includes a set of post-processing circuits for computing output values of the computation nodes. The output values for a first layer are for storage in the core circuits as input values for a second layer. The NNIC includes an output bus that connects the post-processing circuits to the core circuits. The output bus is for (i) receiving a set of output values from the post-processing circuits, (ii) transporting the output values of the set to the core circuits based on configuration data specifying a core circuit at which each of the output values is to be stored, and (iii) aligning the output values for storage in the core circuits.

Abstract: The present disclosure provides chip architectures for FPGAs and other routing implementations that provide for increased memory with high bandwidth, in a reduced size, accessible with reduced latency. Such architectures include a first layer in advanced node and a second layer in legacy node. The first layer includes an active die, active circuitry, and a configurable memory, and the second layer includes a passive die with wiring. The second layer is bonded to the first layer such that the wiring of the second layer interconnects with the active circuitry of the first layer and extends an amount of wiring possible in the first layer.

Abstract: A bonded structure is disclosed. The bonded structure can include a first element that has a first plurality of contact pads. The first plurality of contact pads includes a first contact pad and a second redundant contact pad. The bonded structure can also include a second element directly bonded to the first element without an intervening adhesive. The second element has a second plurality of contact pads. The second plurality of contact pads includes a third contact pad and a fourth redundant contact pad. The first contact pad is configured to connect to the third contact pad. The second contact pad is configured to connect to the fourth contact pad. The bonded structure can include circuitry that has a first state in which an electrical signal is transferred to the first contact pad and a second state in which the electrical signal is transferred to the second contact pad.

Abstract: For a neural network inference circuit that executes a neural network including multiple computation nodes at multiple layers for which data is stored in a plurality of memory banks, some embodiments provide a method for dynamically putting memory banks into a sleep mode of operation to conserve power. The method tracks the accesses to individual memory banks and, if a certain number of clock cycles elapse with no access to a particular memory bank, sends a signal to the memory bank indicating that it should operate in a sleep mode. Circuit components involved in dynamic memory sleep, in some embodiments, include a core RAM pipeline, a core RAM sleep controller, a set of core RAM bank select decoders, and a set of core RAM memory bank wrappers.