Abstract: A method for multiple copies of a set of multi-kernel set operations in a hardware accelerated neural network includes a word line for receiving a pixel value of an input image. A bit line communicates a modified pixel value. An analog memory cell including a first capacitor stores a first kernel weight of a first kernel in one of a plurality of kernel sets such that the pixel value is operated on by the first kernel weight to produce the modified pixel value. A charge connection connects the first capacitor to at least a second capacitor storing a second kernel weight of a related kernel of a second one of the plurality of kernel sets such that charge is shared between the first capacitor and at least the second capacitor to normalize the first kernel weight and the second kernel weight.

Abstract: Methods and systems of implementing a convolutional neural network are described. In an example, a structure may receive input signals and distribute the input signals to a plurality of unit cells. The structure may include a plurality of multi-kernel modules that may include a respective set of unit cells. A unit cell may correspond to an element of a kernel being implemented in the convolutional neural network and may include a storage component configured to store a weight of a corresponding element of the kernel. A first pass gate of the unit cell may be activated to pass a stored weight of the unit cell to a plurality of operation circuits in the corresponding unit cell, such that the stored weight may be applied to the input signals. The structure may generate a set of outputs based on the application of the stored weights to the input signals.

Abstract: Techniques for auto weight scaling a bounded weight range of RPU devices with the size of the array during ANN training are provided. In one aspect, a method of ANN training includes: initializing weight values winit in the array to a random value, wherein the array represents a weight matrix W with m rows and n columns; calculating a scaling factor ? based on a size of the weight matrix W; providing digital inputs x to the array; dividing the digital inputs x by a noise and bound management factor ? to obtain adjusted digital inputs x?; performing a matrix-vector multiplication of the adjusted digital inputs x? with the array to obtain digital outputs y?; multiplying the digital outputs y? by the noise and bound management factor ?; and multiplying the digital outputs y? by the scaling factor ? to provide digital outputs y of the array.

Abstract: A resistive processing unit cell includes a weight storage device to store a weight value of the resistive processing unit cell, and multiple circuit blocks. Each circuit block includes a weight update circuit coupled to dedicated update control lines, and a weight read circuit coupled to dedicated read control lines. The circuit blocks are configured to operate in parallel to (i) perform separate weight read operations in which each read circuit generates a read current based on a stored weight value, and outputs the read current on the dedicated read control lines of the read circuit, and (ii) perform separate weight update operations in which each update circuit receives respective update control signals on the dedicated update control lines, generates update currents based on the respective update control signals, and applies the update current to the weight storage device to adjust the weight value based on the update current.

Abstract: Mechanisms are provided for acceleration of convolutional neural networks on analog arrays. Input ports receive image signals from frames in an input image. Input memory arrays store the image signals received from the input ports into a respective input memory location to create a plurality of image sub-regions in input memory arrays. A distributor associated each of a set of analog array tiles in an analog array to a part of image sub-regions of the input memory arrays, so that one or more of a set of analog memory components is associated with the image signals in a distribution order to create a respective output signal. An assembler stores each of the respective output signals into one of a set of memory outputs in an output order that is determined by the distribution order.

Abstract: Advanced noise and signal management techniques for RPU arrays during ANN training are provided. In one aspect of the invention, a method for ANN training includes: providing an array of RPU devices with pre-normalizers and post-normalizers; computing and pre-normalizing a mean and standard deviation of all elements of an input vector x to the array that belong to the set group of each of the pre-normalizers; and computing and post-normalizing the mean ? and the standard deviation ? of all elements of an output vector y that belong to the set group of each of the post-normalizers.

Abstract: An apparatus and method are provided for saturation prevention of a current integrator in a Resistive Processing Unit-based (RPU-based) accelerator. The apparatus includes a set of hardware switches. The apparatus further includes a voltage generator, operatively coupled between an input terminal and an output terminal of the current integrator, reducing a magnitude of an output voltage at the output terminal of the current integrator during a current integration operation by selectively applying a non-zero initial voltage to the current integrator prior to the current integration operation, responsive to an operating state of the set of hardware switches.

Abstract: A resistive processing unit cell includes a weight storage device to store a weight value of the resistive processing unit cell, and multiple circuit blocks. Each circuit block includes a weight update circuit coupled to dedicated update control lines, and a weight read circuit coupled to dedicated read control lines. The circuit blocks are configured to operate in parallel to (i) perform separate weight read operations in which each read circuit generates a read current based on a stored weight value, and outputs the read current on the dedicated read control lines of the read circuit, and (ii) perform separate weight update operations in which each update circuit receives respective update control signals on the dedicated update control lines, generates update currents based on the respective update control signals, and applies the update current to the weight storage device to adjust the weight value based on the update current.

Abstract: An apparatus and method are provided for saturation prevention of a current integrator in a Resistive Processing Unit-based (RPU-based) accelerator. The apparatus includes a set of hardware switches. The apparatus further includes a voltage generator, operatively coupled between an input terminal and an output terminal of the current integrator, reducing a magnitude of an output voltage at the output terminal of the current integrator during a current integration operation by selectively applying a non-zero initial voltage to the current integrator prior to the current integration operation, responsive to an operating state of the set of hardware switches.

Abstract: Techniques for auto weight scaling a bounded weight range of RPU devices with the size of the array during ANN training are provided. In one aspect, a method of ANN training includes: initializing weight values winit in the array to a random value, wherein the array represents a weight matrix W with m rows and n columns; calculating a scaling factor ? based on a size of the weight matrix W; providing digital inputs x to the array; dividing the digital inputs x by a noise and bound management factor ? to obtain adjusted digital inputs x?; performing a matrix-vector multiplication of the adjusted digital inputs x? with the array to obtain digital outputs y?; multiplying the digital outputs y? by the noise and bound management factor ?; and multiplying the digital outputs y? by the scaling factor ? to provide digital outputs y of the array.

Abstract: Advanced noise and signal management techniques for RPU arrays during ANN training are provided. In one aspect of the invention, a method for ANN training includes: providing an array of RPU devices with pre-normalizers and post-normalizers; computing and pre-normalizing a mean and standard deviation of all elements of an input vector x to the array that belong to the set group of each of the pre-normalizers; and computing and post-normalizing the mean p and the standard deviation a of all elements of an output vector y that belong to the set group of each of the post-normalizers.

Abstract: Zero-shifting techniques in analog crosspoint arrays are provided. In one aspect, an analog array-based vector-matrix multiplication includes: a weight array connected to a reference array, each including a crossbar array having a set of conductive row wires and a set of conductive column wires intersecting the set of conductive row wires, and optimizable crosspoint devices at intersections of the set of conductive column wires and the set of conductive row wires. A method for analog array-based vector-matrix computing is also provided that includes: applying repeated voltage pulses to the crosspoint devices in the weight array until all of the crosspoint devices in the weight array converge to their own symmetry point; and copying conductance values for each crosspoint device from the weight array to the reference array.

Abstract: Methods and systems of implementing a convolutional neural network are described. In an example, a structure may receive input signals and distribute the input signals to a plurality of unit cells. The structure may include a plurality of multi-kernel modules that may include a respective set of unit cells. A unit cell may correspond to an element of a kernel being implemented in the convolutional neural network and may include a storage component configured to store a weight of a corresponding element of the kernel. A first pass gate of the unit cell may be activated to pass a stored weight of the unit cell to a plurality of operation circuits in the corresponding unit cell, such that the stored weight may be applied to the input signals. The structure may generate a set of outputs based on the application of the stored weights to the input signals.

Abstract: A method for multiple copies of a set of multi-kernel set operations in a hardware accelerated neural network includes a word line for receiving a pixel value of an input image. A bit line communicates a modified pixel value. An analog memory cell including a first capacitor stores a first kernel weight of a first kernel in one of a plurality of kernel sets such that the pixel value is operated on by the first kernel weight to produce the modified pixel value. A charge connection connects the first capacitor to at least a second capacitor storing a second kernel weight of a related kernel of a second one of the plurality of kernel sets such that charge is shared between the first capacitor and at least the second capacitor to normalize the first kernel weight and the second kernel weight.

Abstract: Zero-shifting techniques in analog crosspoint arrays are provided. In one aspect, an analog array-based vector-matrix multiplication includes: a weight array connected to a reference array, each including a crossbar array having a set of conductive row wires and a set of conductive column wires intersecting the set of conductive row wires, and optimizable crosspoint devices at intersections of the set of conductive column wires and the set of conductive row wires. A method for analog array-based vector-matrix computing is also provided that includes: applying repeated voltage pulses to the crosspoint devices in the weight array until all of the crosspoint devices in the weight array converge to their own symmetry point; and copying conductance values for each crosspoint device from the weight array to the reference array.

Abstract: Mechanisms are provided for acceleration of convolutional neural networks on analog arrays. Input ports receive image signals from frames in an input image. Input memory arrays store the image signals received from the input ports into a respective input memory location to create a plurality of image sub-regions in input memory arrays. A distributor associated each of a set of analog array tiles in an analog array to a part of image sub-regions of the input memory arrays, so that one or more of a set of analog memory components is associated with the image signals in a distribution order to create a respective output signal. An assembler stores each of the respective output signals into one of a set of memory outputs in an output order that is determined by the distribution order.