Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing sequence data using neural networks with linear recurrence and feedforward units. In one aspect, a system for performing a machine learning task on a network input to generate a network output is provided. The system includes one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement a neural network configured to perform the machine learning task. The neural network includes a number of layer blocks each including: (i) a linear recurrent layer, and (ii) one or more feedforward layers. Each layer block is configured to perform operations including: receiving an input sequence for the layer block; and generating an output sequence for the layer block.

Abstract: A method, device, and computer-readable storage medium for generating a text representation of a speech sample, including receiving an audio sample, encoding the audio sample based on left context of the audio sample with a structured state-space sequence model and a conformer, the structured state-space sequence model being initialized with a diagonal matrix of recurrent weights and trained with a set of training data, decoding the encoded audio sample, and generating a transcript of the audio sample based on the decoding.