Abstract: Techniques for training a machine learning model are described herein. For example, the techniques may include implementing a cross batch normalization layer that generates a cross batch normalization layer output based on a first layer output during training of the neural network. The training may be based on a local batch of training examples of a global batch including the local batch and at least one remote batch of training examples. The cross batch normalization layer output may include normalized components of the first layer output determined based on global normalization statistics for the global batch. Such techniques may be used to train a neural network over distributed machines by synchronizing batches between such machines.

Abstract: Techniques for training a machine learning model are described herein. For example, the techniques may include implementing a cross batch normalization layer that generates a cross batch normalization layer output based on a first layer output during training of the neural network. The training may be based on a local batch of training examples of a global batch including the local batch and at least one remote batch of training examples. The cross batch normalization layer output may include normalized components of the first layer output determined based on global normalization statistics for the global batch. Such techniques may be used to train a neural network over distributed machines by synchronizing batches between such machines.