Abstract: Methods and neural network accelerator for online selection of number formats for network parameters of a neural network.

Abstract: A method and data processing system for implementing inference using an attention-based neural network in a neural network accelerator comprising fixed function hardware. An input sequence for the neural network is padded to a fixed length. A padding mask is generated, identifying the part of the padded input sequence that contains the padding values. An attention mask is generated from the padding mask, using an outer product operation. The padded input sequence and the attention mask are processed to perform the inference using the attention-based neural network. Also disclosed are methods and data processing systems for selecting numerical formats for use in such a neural network, and methods and data processing systems for training such a neural network.

Abstract: A computer-implemented method for selecting numerical formats suitable for use in configuring a hardware implementation of an attention-based neural network. A dataset of test input sequences for the neural network is obtained. Each test input sequence is padded with padding values. For each padded input sequence, a padding mask is generated identifying the part of the padded input sequence that contains the padding values. An attention mask is generated from each padding mask, using an outer product operation. The padded input sequences and attention masks are processed through the neural network. During the processing, statistics are collected, describing ranges of values obtained at various layers of the neural network. Numerical formats are selected for the various layers based on the collected statistics.

Abstract: A computer-implemented method of selecting a number format for use in configuring a hardware implementation of a bidirectional recurrent neural network (BRNN) for operation on a sequence of inputs. A received BRNN representation is implemented as a test neural network equivalent to the BRNN over a sequence of input tensors, each step of the test neural network being for operation on (a) an input tensor of the sequence, (b) a corresponding backward state tensor generated in respect of a subsequent input tensor of the sequence, and (c) a corresponding forward state tensor generated in respect of a preceding input tensor of the sequence. The test neural network includes a forward recurrent neural network (RNN) for operation on the forward state tensors over the input tensors of the sequence; and a backward RNN for operation on the backward state tensors over the input tensors of the sequence.

Abstract: A method of implementing in hardware a bidirectional recurrent neural network (BRNN) for operation on a sequence of inputs, each step of the BRNN being for operation on (a) an input of the sequence, (b) corresponding backward state generated in respect of a subsequent input of the sequence, and (c) corresponding forward state generated in respect of a preceding input of the sequence. A representation of the BRNN is transformed into a derivative neural network equivalent to the BRNN over the sequence of inputs. The derivative neural network includes a forward recurrent neural network (RNN) for operation on the forward state over the inputs of the sequence, and a backward recurrent neural network (RNN) for operation on the backward state over the inputs of the sequence. The derivative neural network is implemented in hardware so as to perform the BRNN on the sequence of inputs.

Abstract: A method and data processing system for implementing a neural network containing at least one matrix multiplication operation. The matrix multiplication operation is mapped to a graph of neural network operations including at least one element-wise operation. The at least one element-wise operation is implemented in fixed-function hardware of a neural network accelerator.

Abstract: Methods for implementing an exponential operation, and a softmax neural network layer, in neural network accelerator hardware, and a data processing system for implementing the exponential operation and a data processing system for implementing the softmax layer. The exponential operation or softmax layer is mapped to a plurality of elementary neural network operations, and the neural network accelerator hardware evaluates these operations, to produce the result of the operation or layer respectively.

Abstract: A method and data processing system implement a neural network containing at least one matrix multiplication operation. The matrix multiplication operation is mapped to a graph of neural network operations including at least one transformation and at least one convolution. The at least one convolution is implemented in fixed-function hardware of a neural network accelerator.