Abstract: Some embodiments provide a method for executing a neural network. The method writes a first input to a first set of physical memory banks in a unified memory shared by an input processing circuit and a neural network inference circuit that executes the neural network. While the neural network inference circuit is executing the network a first time to generate a first output for the first input, the method writes a second input to a second set of physical memory banks in the unified memory. The neural network inference circuit executes a same set of instructions to read the first input from the first set of memory banks in order to execute the network the first time and to read the second input from the second set of memory banks in order to execute the network a second time to generate a second output for the second input.

Abstract: Some embodiments provide a method for training multiple parameters of a machine-trained (MT) network subject to a sparsity constraint that requires a threshold portion of the parameters to be equal to zero. A first set of the parameters subject to the sparsity constraint are grouped into groups of parameters. For each parameter of a second set of the parameters subject to the sparsity constraint, the method determines an accuracy penalty associated with the parameter being set to zero. For each group of parameters in the first set of parameters, the method determines a minimum accuracy penalty for each possible number of parameters in the group being set to zero. The method uses the determined accuracy penalties to set to the value zero at least the threshold portion of the plurality of parameters.

Abstract: Some embodiments provide a method for generating neural network program instructions for a neural network inference circuit to execute a neural network. The neural network inference circuit includes a particular amount of available memory. The method receives a specification of the neural network including multiple layers. The method determines (i) a required amount of weight memory for the neural network and (ii) required amounts of activation memory for each of a set of layers of the neural network. When the required amount of weight memory and the required amount of activation memory for at least one layer is greater than the particular amount of available memory, the method generates the program instructions for the neural network inference circuit to execute a first set of the layers of the neural network multiple times for different blocks of input data and execute a second set of the layers in a single pass.

Abstract: Some embodiments provide a method for a neural network inference circuit that executes a neural network including multiple computation nodes at multiple layers. Each computation node of a set of the computation nodes includes a dot product of input values and weight values. The method reads a set of encoded weight data for a set of weight values from a memory of the neural network inference circuit. The method decodes the encoded weight data to generate decoded weight data for the set of weight values. The method stores the decoded weight data in a buffer. The method uses the decoded weight data to execute a set of computation nodes. Each computation node of the set of computation nodes includes a dot product between the set of weight values and a different set of input values.

Abstract: Some embodiments provide a method for training a machine-trained (MT) network. The method uses a set of training inputs to train parameters of the MT network according to an initial loss function. The method uses a set of validation inputs to compute an error measure for the MT network as trained by the first set of training inputs. The method modifies the loss function for subsequent training of the MT network based on the computed error measure. The method uses the set of training inputs to train the parameters of the MT network according to the modified loss function.

Abstract: Some embodiments of the invention provide a method for configuring a network with multiple nodes. Each node generates an output value based on received input values and a set of weights that are previously trained to each have an initial value. For each weight, the method calculates a factor that represents a loss of accuracy to the network due to changing the weight from its initial value to a different value in a set of allowed values for the weight. Based on the factors, the method identifies a subset of the weights that have factors with values below a threshold. The method changes the values of each weight from its initial value to one of the values in its set of allowed values. The values of the identified subset are all changed to zero. The method trains the weights beginning with the changed values for each weight.

Abstract: Some embodiments provide a neural network inference circuit (NNIC) for executing a neural network that includes multiple computation nodes at multiple layers. The NNIC includes multiple core circuits including memories for storing input values for the computation nodes. The NNIC includes a set of post-processing circuits for computing output values of the computation nodes. The output values for a first layer are for storage in the core circuits as input values for a second layer. The NNIC includes an output bus that connects the post-processing circuits to the core circuits. The output bus is for (i) receiving a set of output values from the post-processing circuits, (ii) transporting the output values of the set to the core circuits based on configuration data specifying a core circuit at which each of the output values is to be stored, and (iii) aligning the output values for storage in the core circuits.

Abstract: Some embodiments provide a method for training parameters of a network. The method receives a network with layers of nodes. Each node of a set of the layers computes an output value based on a set of input values and a set of trained weight values. A first layer of the network includes a first number of filters. The method replaces the first layer with a second layer having a second number of filters that is less than the first number and a third layer, following the second layer, having the first number of filters. Each weight value in the filters of the second and third layers is restricted to a set of allowed quantized weight values. A total number of weight values in the filters of the second and third layers is less than a total number of weight values in the filters of the first layer.

Abstract: Some embodiments provide a method for training parameters of a network. the method receives a machine-trained (MT) network with multiple layers of computation nodes. Each computation node of a set of the layers computes an output value based on a set of input values and a set of trained weight values. A first layer of the MT network includes a first number of filters. The method replaces the first layer with (i) a second layer having a second number of filters that is less than the first number of filters and (ii) a third layer having the first number of filters. Output values of computation nodes of the second layer are quantized and the third layer using the quantized output values of the second layer as input values.

Abstract: Some embodiments provide a method for configuring a machine-trained (MT) network that includes multiple configurable weights to train. The method propagates a set of inputs through the MT network to generate a set of output probability distributions. Each input has a corresponding expected output probability distribution. The method calculates a value of a continuously-differentiable loss function that includes a term approximating an extremum function of the difference between the expected output probability distributions and generated set of output probability distributions. The method trains the weights by back-propagating the calculated value of the continuously-differentiable loss function.

Abstract: Some embodiments provide a method for training a network including layers that each includes multiple nodes. The method identifies a set of related layers of the network. Each node in one of the related layers has corresponding nodes in each of the other related layers. Each set of corresponding nodes receives a same set of inputs and applies different sets of weights to the inputs to generate an output. The method identifies an element-wise addition layer including nodes that each add outputs of a different set of corresponding nodes from the related layers to generate a sum. The method uses a set of outputs generated by the nodes of each related layer to determine batch normalization parameters specific to each layer of the set of related layers. The method uses data generated by the element-wise addition layer to determine batch normalization parameters for the set of related layers.

Abstract: Some embodiments of the invention provide a novel method for training a multi-layer node network. Some embodiments train the multi-layer network using a set of inputs generated with random misalignments incorporated into the training data set. In some embodiments, the training data set is a synthetically generated training set based on a three-dimensional ground truth model as it would be sensed by a sensor array from different positions and with different deviations from ideal alignment and placement. Some embodiments dynamically generate training data sets when a determination is made that more training is required. Training data sets, in some embodiments, are generated based on training data sets for which the multi-layer node network has produced bad results.

Abstract: Some embodiments of the invention provide a machine-trained method that selects an output from a plurality of outputs by processing an input. The method uses layers of machine-trained processing nodes to process the input to produce a multi-dimensional codeword. The method generates a set of affinity scores with each affinity score identifying the proximity of the produced codeword to a codeword in a first set of previously defined codewords. The method compares the set of affinity scores generated for the produced codeword with sets of affinity scores previously generated for the first-set codewords that express the proximity of the first-set codewords to a second set of codewords. The method identifies the first-set codeword that has the affinity score set that best matches the affinity score set generated for the produced codeword. The method selects the associated output of the identified first-set codeword as the output of the network.

Abstract: Some embodiments provide a method for training a machine-trained (MT) network that processes input data using network parameters. The method maps a set of input instances to a set of output values by propagating the set of input instances through the MT network. The set of input instances include input instances for each of multiple categories. The method selects multiple input instances as anchor instances. For each anchor instance, the method computes a loss function as a comparison between the output value for the anchor instance and each output value for an input instance in a different category than the anchor. The method computes a total loss function for the MT network as a sum of the loss function computed for each anchor instance. The method trains the network parameters using the computed total loss function.

Abstract: Some embodiments of the invention provide a method for configuring a machine-trained (MT) network that includes multiple nodes. Each node of a set of the nodes generates an output value based on received input values and a set of configurable weights. The method propagates a set of inputs through the MT network to generate a set of outputs, with each input having a corresponding expected output. The method calculates a value of a loss function comprising (i) a first term that measures a difference between each generated output and its corresponding expected output and (ii) a second term that constrains the weights to discrete sets of allowed values and accounts for an increase in the first term due to constraining the weights to the discrete sets of values. The method uses the calculated value of the loss function to train the weights of the MT network.

Abstract: Some embodiments provide a method for executing a layer of a neural network, for a circuit that restricts a number of weight values used per layer. The method applies a first set of weights to a set of inputs to generate a first set of results. The first set of weights are restricted to a first set of allowed values. For each of one or more additional sets of weights, the method applies the respective additional set of weights to the same set of inputs to generate a respective additional set of results. The respective additional set of weights is restricted to a respective additional set of allowed values that is related to the first set of allowed values and the other additional sets of allowed values. The method generates outputs for the particular layer by combining the first set of results with each respective additional set of results.

Abstract: Some embodiments of the invention provide a method for implementing a temporal convolution network (TCN) that includes several layers of machine-trained processing nodes. While processing one set of inputs that is provided to the TCN at a particular time, some of the processing nodes of the TCN use intermediate values computed by the processing nodes for other sets of inputs that were provided to the TCN at earlier times. To speed up the operation of the TCN and improve its efficiency, the method of some embodiments stores intermediate values computed by the TCN processing nodes for earlier sets of TCN inputs, so that these values can later be used for processing later set of TCN inputs.

Abstract: Some embodiments provide a compiler for optimizing the implementation of a machine-trained network (e.g., a neural network) on an integrated circuit (IC). The compiler of some embodiments receives a specification of a machine-trained network including multiple layers of computation nodes and generates a graph representing options for implementing the machine-trained network in the IC. The compiler, as part of generating the graph, in some embodiments, determines whether any set of channels contains no non-zero values (i.e., contains only zero values). For sets of channels that include no non-zero values, some embodiments perform a zero channel removal operation to remove all-zero channels wherever possible. In some embodiments, zero channel removal operations include removing input channels, removing output channels, forward propagation, and backward propagation of channels and constants.

Abstract: Some embodiments of the invention provide a method for implementing a temporal convolution network (TCN) that includes several layers of machine-trained processing nodes. While processing one set of inputs that is provided to the TCN at a particular time, some of the processing nodes of the TCN use intermediate values computed by the processing nodes for other sets of inputs that were provided to the TCN at earlier times. To speed up the operation of the TCN and improve its efficiency, the method of some embodiments stores intermediate values computed by the TCN processing nodes for earlier sets of TCN inputs, so that these values can later be used for processing later set of TCN inputs.

Abstract: For a neural network inference circuit that executes a neural network including multiple computation nodes at multiple layers for which data is stored in a plurality of memory banks, some embodiments provide a method for dynamically putting memory banks into a sleep mode of operation to conserve power. The method tracks the accesses to individual memory banks and, if a certain number of clock cycles elapse with no access to a particular memory bank, sends a signal to the memory bank indicating that it should operate in a sleep mode. Circuit components involved in dynamic memory sleep, in some embodiments, include a core RAM pipeline, a core RAM sleep controller, a set of core RAM bank select decoders, and a set of core RAM memory bank wrappers.