Abstract: A method of manufacturing an apparatus is provided. The apparatus is formed on a wafer or a package. The apparatus includes a polynomial generator, a plurality of matrix generators connected to an output of the polynomial generator, and a convolution generator connected to an output of the plurality of matrix generators. The apparatus is tested using one or more electrical to optical converters, one or more optical splitters, and one or more optical to electrical converters.

Abstract: An electronic apparatus performs a method of quantizing a neural network. The method includes: clipping a value used within the neural network beyond a range from a minimum value to a maximum value; simulating a quantization process using the clipped value; updating the minimum value and the maximum value during a training of the neural network to optimize the quantization process; and quantizing the value used within the neural network according to the updated minimum value and the maximum value. In some embodiments, the method of quantizing a neural network further includes minimizing the range during the training.

Abstract: A method includes: providing a deep neural networks (DNN) model comprising a plurality of layers, each layer of the plurality of layers includes a plurality of nodes; sampling a change of a weight for each of a plurality of weights based on a distribution function, each weight of the plurality of weights corresponds to each node of the plurality of nodes; updating the weight with the change of the weight multiplied by a sign of the weight; and training the DNN model by iterating the steps of sampling the change and updating the weight. The plurality of weights has a high rate of sparsity after the training.

Abstract: A system and a method to normalize a deep neural network (DNN) in which a mean of activations of the DNN is set to be equal to about 0 for a training batch size of 8 or less, and a variance of the activations of the DNN is set to be equal to about a predetermined value for the training batch size. A minimization module minimizes a sum of a network loss of the DNN plus a sum of a product of a first Lagrange multiplier times the mean of the activations squared plus a sum of a product of a second Lagrange multiplier times a quantity of the variance of the activations minus one squared.

Abstract: A system and a method generate a neural network that includes at least one layer having weights and output feature maps that have been jointly pruned and quantized. The weights of the layer are pruned using an analytic threshold function. Each weight remaining after pruning is quantized based on a weighted average of a quantization and dequantization of the weight for all quantization levels to form quantized weights for the layer. Output feature maps of the layer are generated based on the quantized weights of the layer. Each output feature map of the layer is quantized based on a weighted average of a quantization and dequantization of the output feature map for all quantization levels. Parameters of the analytic threshold function, the weighted average of all quantization levels of the weights and the weighted average of each output feature map of the layer are updated using a cost function.

Abstract: A system and a method generate a neural network that includes at least one layer having weights and output feature maps that have been jointly pruned and quantized. The weights of the layer are pruned using an analytic threshold function. Each weight remaining after pruning is quantized based on a weighted average of a quantization and dequantization of the weight for all quantization levels to form quantized weights for the layer. Output feature maps of the layer are generated based on the quantized weights of the layer. Each output feature map of the layer is quantized based on a weighted average of a quantization and dequantization of the output feature map for all quantization levels. Parameters of the analytic threshold function, the weighted average of all quantization levels of the weights and the weighted average of each output feature map of the layer are updated using a cost function.

Abstract: A method includes: providing a deep neural networks (DNN) model comprising a plurality of layers, each layer of the plurality of layers includes a plurality of nodes; sampling a change of a weight for each of a plurality of weights based on a distribution function, each weight of the plurality of weights corresponds to each node of the plurality of nodes; updating the weight with the change of the weight multiplied by a sign of the weight; and training the DNN model by iterating the steps of sampling the change and updating the weight. The plurality of weights has a high rate of sparsity after the training.

Abstract: A system and method that provides balanced pruning of weights of a deep neural network (DNN) in which weights of the DNN are partitioned into a plurality of groups, a count of a number of non-zero weights is determined in each group, a variance of the count of weights in each group is determined, a loss function of the DNN is minimized using Lagrange multipliers with a constraint that the variance of the count of weights in each group is equal to 0, and the weights and the Lagrange multipliers are retrained by back-propagation.

Abstract: A technique to prune weights of a neural network using an analytic threshold function h(w) provides a neural network having weights that have been optimally pruned. The neural network includes a plurality of layers in which each layer includes a set of weights w associated with the layer that enhance a speed performance of the neural network, an accuracy of the neural network, or a combination thereof. Each set of weights is based on a cost function C that has been minimized by back-propagating an output of the neural network in response to input training data. The cost function C is also minimized based on a derivative of the cost function C with respect to a first parameter of the analytic threshold function h(w) and on a derivative of the cost function C with respect to a second parameter of the analytic threshold function h(w).

Abstract: A technique to prune weights of a neural network using an analytic threshold function h(w) provides a neural network having weights that have been optimally pruned. The neural network includes a plurality of layers in which each layer includes a set of weights w associated with the layer that enhance a speed performance of the neural network, an accuracy of the neural network, or a combination thereof. Each set of weights is based on a cost function C that has been minimized by back-propagating an output of the neural network in response to input training data. The cost function C is also minimized based on a derivative of the cost function C with respect to a first parameter of the analytic threshold function h(w) and on a derivative of the cost function C with respect to a second parameter of the analytic threshold function h(w).

Abstract: A system and method for pruning. A neural network includes a plurality of long short-term memory cells, each of which includes an input having a weight matrix Wc, an input gate having a weight matrix Wi, a forget gate having a weight matrix Wf, and an output gate having a weight matrix Wo. In some embodiments, after initial training, one or more of the weight matrices Wi, Wf, and Wo are pruned, and the weight matrix Wc is left unchanged. The neural network is then retrained, the pruned weights being constrained to remain zero during retraining.

Abstract: A method of manufacturing an apparatus is provided. The apparatus is formed on a wafer or a package, The apparatus includes a polynomial generator, a plurality of matrix generators connected to an output of the polynomial generator, and a convolution generator connected to an output of the plurality of matrix generators. The apparatus is tested using one or more electrical to optical converters, one or more optical splitters, and one or more optical to electrical converters.

Abstract: A method of manufacturing an apparatus and a method of constructing an integrated circuit are provided. The method of manufacturing an apparatus includes forming the apparatus on a wafer or a package with at least one other apparatus, wherein the apparatus comprises a polynomial generator, a first matrix generator, a second matrix generator, a third matrix generator, and a convolution generator; and testing the apparatus, wherein testing the apparatus comprises testing the apparatus using one or more electrical to optical converters, one or more optical splitters that split an optical signal into two or more optical signals, and one or more optical to electrical converters.

Abstract: A system and a method generate a neural network that includes at least one layer having weights and output feature maps that have been jointly pruned and quantized. The weights of the layer are pruned using an analytic threshold function. Each weight remaining after pruning is quantized based on a weighted average of a quantization and dequantization of the weight for all quantization levels to form quantized weights for the layer. Output feature maps of the layer are generated based on the quantized weights of the layer. Each output feature map of the layer is quantized based on a weighted average of a quantization and dequantization of the output feature map for all quantization levels. Parameters of the analytic threshold function, the weighted average of all quantization levels of the weights and the weighted average of each output feature map of the layer are updated using a cost function.

Abstract: A recurrent neural network that predicts blood glucose level includes a first long short-term memory (LSTM) network and a second LSTM network. The first LSTM network may include an input to receive near-infrared (NIR) radiation data and includes an output. The second LSTM network may include an input to receive the output of the first LSTM network and an output to output blood glucose level data based on the NIR radiation data input to the first LSTM network.

Abstract: A system and method for pruning. A neural network includes a plurality of long short-term memory cells, each of which includes an input having a weight matrix Wc, an input gate having a weight matrix Wi, a forget gate having a weight matrix Wf, and an output gate having a weight matrix Wo. In some embodiments, after initial training, one or more of the weight matrices Wi, Wf, and Wo are pruned, and the weight matrix Wc is left unchanged. The neural network is then retrained, the pruned weights being constrained to remain zero during retraining.

Abstract: A system and method for pruning. A neural network includes a plurality of long short-term memory cells, each of which includes an input having a weight matrix Wc, an input gate having a weight matrix Wi, a forget gate having a weight matrix Wf, and an output gate having a weight matrix Wo. In some embodiments, after initial training, one or more of the weight matrices Wi, Wf, and Wo are pruned, and the weight matrix Wc is left unchanged. The neural network is then retrained, the pruned weights being constrained to remain zero during retraining.

Abstract: A system and a method to normalize a deep neural network (DNN) in which a mean of activations of the DNN is set to be equal to about 0 for a training batch size of 8 or less, and a variance of the activations of the DNN is set to be equal to about a predetermined value for the training batch size. A minimization module minimizes a sum of a network loss of the DNN plus a sum of a product of a first Lagrange multiplier times the mean of the activations squared plus a sum of a product of a second Lagrange multiplier times a quantity of the variance of the activations minus one squared.

Abstract: A system and method that provides balanced pruning of weights of a deep neural network (DNN) in which weights of the DNN are partitioned into a plurality of groups, a count of a number of non-zero weights is determined in each group, a variance of the count of weights in each group is determined, a loss function of the DNN is minimized using Lagrange multipliers with a constraint that the variance of the count of weights in each group is equal to 0, and the weights and the Lagrange multipliers are retrained by back-propagation.

Abstract: A method of manufacturing an apparatus and a method of constructing an integrated circuit are provided. The method of manufacturing an apparatus includes forming the apparatus on a wafer or a package with at least one other apparatus, wherein the apparatus comprises a polynomial generator, a first matrix generator, a second matrix generator, a third matrix generator, and a convolution generator; and testing the apparatus, wherein testing the apparatus comprises testing the apparatus using one or more electrical to optical converters, one or more optical splitters that split an optical signal into two or more optical signals, and one or more optical to electrical converters.