Abstract: A combination of two or more trained machine learning models can exhibit a combined accuracy greater than the accuracy of any one of the constituent models. However, this increase accuracy comes at additional computational cost. Cascades of machine learning models are provided herein that result in increased model accuracy and/or reduced model compute cost. These benefits are obtained by conditionally executing one or more of the models of the cascade based on the estimated correctness of already-executed models. The estimated correctness can be obtained as an additional output of the already-executed model(s) or could be determined as an entropy, maximum class probability, maximum class logit, or other function of the output(s) of the already-executed model(s). The expected computational cost of executing the model cascade is reduced by only executing the downstream model(s) when the upstream model(s) has resulted in an output whose accuracy is suspect.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing network inputs by applying augmentations to internal representations of the network inputs.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for improving the representation of items of a vocabulary in an embedding space for use in machine learning models. An embedding matrix is generated wherein each row in the embedding matrix is a vector of elements and corresponds to an item of a vocabulary. A score is assigned to each vector in the embedding matrix indicating a probability of its corresponding vector being used in the machine learning model. The scores are iteratively updated by sampling a proper subset of vectors and updating the elements of each respective vector in the proper subset of vectors based on the respective scores of vectors. The score of each vector are then updated based on a loss function of the machine learning model. The embedding matrix is then re-structured based on the updated scores of the vectors.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for neural network compression. In one aspect, a method comprises receiving a neural network and identifying a particular set of multiple weights of the neural network. Multiple anchor points are determined based on current values of the particular set of weights of the neural network. The neural network is trained by, at each of multiple training iterations, performing operations comprising adjusting the values of the particular set of weights by backpropagating gradients of a loss function. The loss function comprises a first loss function term based on a prediction accuracy of the neural network and a second loss function term based on a similarity of the current values of the particular set of weights to the anchor points. After training, the values of the particular set of weights are quantized based on the anchor points.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training neural networks. In one aspect, a system includes a neural network shrinking engine that is configured to receive a neural network being trained and generate a reduced neural network by a shrinking process. The shrinking process includes training the neural network based on a shrinking engine loss function that includes terms penalizing active neurons of the neural network and removing inactive neurons from the neural network. The system includes a neural network expansion engine that is configured to receive the neural network being trained and generate an expanded neural network by an expansion process including adding new neurons to the neural network and training the neural network based on an expanding engine loss function. The system includes a training subsystem that generates reduced neural networks and expanded neural networks.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining neural network architectures.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training neural networks. In one aspect, a system includes a neural network shrinking engine that is configured to receive a neural network being trained and generate a reduced neural network by a shrinking process. The shrinking process includes training the neural network based on a shrinking engine loss function that includes terms penalizing active neurons of the neural network and removing inactive neurons from the neural network. The system includes a neural network expansion engine that is configured to receive the neural network being trained and generate an expanded neural network by an expansion process including adding new neurons to the neural network and training the neural network based on an expanding engine loss function. The system includes a training subsystem that generates reduced neural networks and expanded neural networks.

Abstract: A combination of two or more trained machine learning models can exhibit a combined accuracy greater than the accuracy of any one of the constituent models. However, this increase accuracy comes at additional computational cost. Cascades of machine learning models are provided herein that result in increased model accuracy and/or reduced model compute cost. These benefits are obtained by conditionally executing one or more of the models of the cascade based on the estimated correctness of already-executed models. The estimated correctness can be obtained as an additional output of the already-executed model(s) or could be determined as an entropy, maximum class probability, maximum class logit, or other function of the output(s) of the already-executed model(s). The expected computational cost of executing the model cascade is reduced by only executing the downstream model(s) when the upstream model(s) has resulted in an output whose accuracy is suspect.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training neural networks. In one aspect, a system includes a neural network shrinking engine that is configured to receive a neural network being trained and generate a reduced neural network by a shrinking process. The shrinking process includes training the neural network based on a shrinking engine loss function that includes terms penalizing active neurons of the neural network and removing inactive neurons from the neural network. The system includes a neural network expansion engine that is configured to receive the neural network being trained and generate an expanded neural network by an expansion process including adding new neurons to the neural network and training the neural network based on an expanding engine loss function. The system includes a training subsystem that generates reduced neural networks and expanded neural networks.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for neural network compression. In one aspect, a method comprises receiving a neural network and identifying a particular set of multiple weights of the neural network. Multiple anchor points are determined based on current values of the particular set of weights of the neural network. The neural network is trained by, at each of multiple training iterations, performing operations comprising adjusting the values of the particular set of weights by backpropagating gradients of a loss function. The loss function comprises a first loss function term based on a prediction accuracy of the neural network and a second loss function term based on a similarity of the current values of the particular set of weights to the anchor points. After training, the values of the particular set of weights are quantized based on the anchor points.

Abstract: Implementations disclose filtering wind noises in video content. A method includes receiving video content comprising an audio component and a video component, detecting, by a processing device, occurrence of a wind noise artifact in a segment of the audio component, identifying an intensity of the wind noise artifact, wherein the intensity is based on a signal-to-noise ratio of the wind noise artifact, selecting, by the processing device, a wind noise replacement operation based on the identified intensity of the wind noise artifact, and applying, by the processing device, the selected wind noise replacement operation to the segment of the audio component to remove the wind noise artifact from the segment.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training neural networks. In one aspect, a system includes a neural network shrinking engine that is configured to receive a neural network being trained and generate a reduced neural network by a shrinking process. The shrinking process includes training the neural network based on a shrinking engine loss function that includes terms penalizing active neurons of the neural network and removing inactive neurons from the neural network. The system includes a neural network expansion engine that is configured to receive the neural network being trained and generate an expanded neural network by an expansion process including adding new neurons to the neural network and training the neural network based on an expanding engine loss function. The system includes a training subsystem that generates reduced neural networks and expanded neural networks.

Abstract: Implementations disclose filtering wind noises in video content. A method includes receiving video content comprising an audio component and a video component, detecting, by a processing device, occurrence of a wind noise artifact in a segment of the audio component, identifying an intensity of the wind noise artifact, wherein the intensity is based on a signal-to-noise ratio of the wind noise artifact, selecting, by the processing device, a wind noise replacement operation based on the identified intensity of the wind noise artifact, and applying, by the processing device, the selected wind noise replacement operation to the segment of the audio component to remove the wind noise artifact from the segment.

Abstract: Implementations disclose filtering wind noises in video content. A method includes receiving video content comprising an audio component and a video component, detecting, by a processing device, occurrence of a wind noise artifact in a segment of the audio component, identifying duration of the wind noise artifact and intensity of the wind noise artifact, selecting, by the processing device, a wind noise replacement operation based on the identified duration and intensity of the wind noise artifact, and applying, by the processing device, the selected wind noise replacement operation to the segment of the audio component to remove the wind noise artifact from the segment.

Abstract: Implementations disclose filtering wind noises in video content. A method includes receiving video content comprising an audio component and a video component, detecting, by a processing device, occurrence of a wind noise artifact in a segment of the audio component, identifying duration of the wind noise artifact and intensity of the wind noise artifact, selecting, by the processing device, a wind noise replacement operation based on the identified duration and intensity of the wind noise artifact, and applying, by the processing device, the selected wind noise replacement operation to the segment of the audio component to remove the wind noise artifact from the segment.