Abstract: Systems and methods for pipelined neural network processing with continuous and asynchronous updates are described. A method for processing a neural network comprising L layers, where L is an integer greater than two, includes partitioning the L layers among a set of computing resources configured to process forward passes and backward passes associated with each of the L layers. The method further includes initiating processing of the forward passes and the backward passes using the set of computing resources. The method further includes upon completion of a first set of forward passes and a first set of backward passes associated with a first layer of the L layers, initiating update of parameters associated with the first layer when gradients are available for updating the parameters associated with the first layer without waiting to calculate gradients associated with any of remaining L layers.

Abstract: Methods for gradient accumulation with free momentum are performed by systems and devices during neural network model training. An accumulator that includes a processor circuit and a memory element generates free momentum between passes of a neural network model training process. The processor circuit receives a difference weight (gradient) and generates a first input by applying a weighting parameter thereto. The processor circuit obtains a prior weight from the memory element and generates a second input by applying another weighting parameter thereto. The processor circuit generates a filtered input with momentum by filtering the first and second input. The memory element generates a stored next pass weight by accumulating the filtered input with the prior weight. A computing resource then processes the next pass of the neural network model training using the stored next pass weight. The methods, systems, and devices are applicable to pipelined model parallelism training processes.

Abstract: Improved convolutional neural network-based machine learning models are disclosed herein. A convolutional neural network is configured to decompose feature maps generated based on a data item to be classified. The feature maps are decomposed into a first and second subsets. The first subset is representative of high frequency components of the data item, and the second subset is representative of low frequency components of the data item. The second subset is upsampled and is combined with the first subset. The combined feature maps are convolved with a filter to extract a set of features associated with the data item. The first subset is also downsampled and combined with the second subset. The combined feature maps are convolved with a filter to extract another set of features. The data item is classified based on the sets of features extracted based on the convolution operations.

Abstract: Methods for gradient accumulation with free momentum are performed by systems and devices during neural network model training. An accumulator that includes a processor circuit and a memory element generates free momentum between passes of a neural network model training process. The processor circuit receives a difference weight (gradient) and generates a first input by applying a weighting parameter thereto. The processor circuit obtains a prior weight from the memory element and generates a second input by applying another weighting parameter thereto. The processor circuit generates a filtered input with momentum by filtering the first and second input. The memory element generates a stored next pass weight by accumulating the filtered input with the prior weight. A computing resource then processes the next pass of the neural network model training using the stored next pass weight. The methods, systems, and devices are applicable to pipelined model parallelism training processes.

Abstract: Improved convolutional neural network-based machine learning models are disclosed herein. A convolutional neural network is configured to decompose feature maps generated based on a data item to be classified. The feature maps are decomposed into a first and second subsets. The first subset is representative of high frequency components of the data item, and the second subset is representative of low frequency components of the data item. The second subset is upsampled and is combined with the first subset. The combined feature maps are convolved with a filter to extract a set of features associated with the data item. The first subset is also downsampled and combined with the second subset. The combined feature maps are convolved with a filter to extract another set of features. The data item is classified based on the sets of features extracted based on the convolution operations.

Abstract: Systems and methods for pipelined neural network processing with continuous and asynchronous updates are described. A method for processing a neural network comprising L layers, where L is an integer greater than two, includes partitioning the L layers among a set of computing resources configured to process forward passes and backward passes associated with each of the L layers. The method further includes initiating processing of the forward passes and the backward passes using the set of computing resources. The method further includes upon completion of a first set of forward passes and a first set of backward passes associated with a first layer of the L layers, initiating update of parameters associated with the first layer when gradients are available for updating the parameters associated with the first layer without waiting to calculate gradients associated with any of remaining L layers.

Abstract: In embodiments, apparatuses, methods and storage media are described that are associated with training adaptive speech recognition systems (“ASR”) using audio and text obtained from captioned video. In various embodiments, the audio and caption may be aligned for identification, such as according to a start and end time associated with a caption, and the alignment may be adjusted to better fit audio to a given caption. In various embodiments, the aligned audio and caption may then be used for training if an error value associated with the audio and caption demonstrates that the audio and caption will aid in training the ASR. In various embodiments, filters may be used on audio and text prior to training. Such filters may be used to exclude potential training audio and text based on filter criteria. Other embodiments may be described and claimed.

Abstract: In embodiments, apparatuses, methods and storage media are described that are associated with training adaptive speech recognition systems (“ASR”) using audio and text obtained from captioned video. In various embodiments, the audio and caption may be aligned for identification, such as according to a start and end time associated with a caption, and the alignment may be adjusted to better fit audio to a given caption. In various embodiments, the aligned audio and caption may then be used for training if an error value associated with the audio and caption demonstrates that the audio and caption will aid in training the ASR. In various embodiments, filters may be used on audio and text prior to training. Such filters may be used to exclude potential training audio and text based on filter criteria. Other embodiments may be described and claimed.

Abstract: In embodiments, apparatuses, methods and storage media are described that are associated with recognition of speech based on sequences of named entities. Language models may be trained as being associated with sequences of named entities. A language model may be selected for speech recognition after identification of one or more sequences of named entities by an initial language model. After identification of the one or more sequences of named entities, weights may be assigned to the one or more sequences of named entities. These weights may be utilized to select a language module and/or update the initial language model to one that is associated with the identified one or more sequences of named entities. In various embodiments, the language model may be repeatedly updated until the recognized speech converges sufficiently to satisfy a predetermined threshold. Other embodiments may be described and claimed.