Abstract: Methods, apparatus, systems and articles of manufacture for loss-error-aware quantization of a low-bit neural network are disclosed. An example apparatus includes a network weight partitioner to partition unquantized network weights of a first network model into a first group to be quantized and a second group to be retrained. The example apparatus includes a loss calculator to process network weights to calculate a first loss. The example apparatus includes a weight quantizer to quantize the first group of network weights to generate low-bit second network weights. In the example apparatus, the loss calculator is to determine a difference between the first loss and a second loss. The example apparatus includes a weight updater to update the second group of network weights based on the difference. The example apparatus includes a network model deployer to deploy a low-bit network model including the low-bit second network weights.

Abstract: Methods, apparatus, systems and articles of manufacture for loss-error-aware quantization of a low-bit neural network are disclosed. An example apparatus includes a network weight partitioner to partition unquantized network weights of a first network model into a first group to be quantized and a second group to be retrained. The example apparatus includes a loss calculator to process network weights to calculate a first loss. The example apparatus includes a weight quantizer to quantize the first group of network weights to generate low-bit second network weights. In the example apparatus, the loss calculator is to determine a difference between the first loss and a second loss. The example apparatus includes a weight updater to update the second group of network weights based on the difference. The example apparatus includes a network model deployer to deploy a low-bit network model including the low-bit second network weights.

Abstract: Systems, methods, apparatuses, and computer program products to receive a plurality of binary weight values for a binary neural network sampled from a policy neural network comprising a posterior distribution conditioned on a theta value. An error of a forward propagation of the binary neural network may be determined based on a training data and the received plurality of binary weight values. A respective gradient value may be computed for the plurality of binary weight values based on a backward propagation of the binary neural network. The theta value for the posterior distribution may be updated using reward values computed based on the gradient values, the plurality of binary weight values, and a scaling factor.

Abstract: Apparatuses, methods, and GPUs are disclosed for universal loss-error-aware quantization (ULQ) of a neural network (NN). In one example, an apparatus includes data storage to store data including activation sets and weight sets, and a network processor coupled to the data storage. The network processor is configured to implement the ULQ by constraining a low-precision NN model based on a full-precision NN model, to perform a loss-error-aware activation quantization to quantize activation sets into ultra-low-bit versions with given bit-width values, to optimize the NN with respect to a loss function that is based on the full-precision NN model, and to perform a loss-error-aware weight quantization to quantize weight sets into ultra-low-bit versions.

Abstract: Methods, apparatus, systems and articles of manufacture for loss-error-aware quantization of a low-bit neural network are disclosed. An example apparatus includes a network weight partitioner to partition unquantized network weights of a first network model into a first group to be quantized and a second group to be retrained. The example apparatus includes a loss calculator to process network weights to calculate a first loss. The example apparatus includes a weight quantizer to quantize the first group of network weights to generate low-bit second network weights. In the example apparatus, the loss calculator is to determine a difference between the first loss and a second loss. The example apparatus includes a weight updater to update the second group of network weights based on the difference. The example apparatus includes a network model deployer to deploy a low-bit network model including the low-bit second network weights.

Abstract: Methods, systems, apparatus, and articles of manufacture are disclosed to train a neural network. An example apparatus includes an architecture evaluator to determine an architecture type of a neural network, a knowledge branch implementor to select a quantity of knowledge branches based on the architecture type, and a knowledge branch inserter to improve a training metric by appending the quantity of knowledge branches to respective layers of the neural network.

Abstract: Methods and apparatus relating to techniques for incremental network quantization. In an example, an apparatus comprises logic, at least partially comprising hardware logic to partition a plurality of model weights in a deep neural network (DNN) model into a first group of weights and a second group of weights, convert each weight in the first group of weights to a power of two, and repeatedly retrain the DNN model while converting a subset of weights in the second group to a power of two or zero. Other embodiments are also disclosed and claimed.