Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing images using neural networks. One of the methods includes receiving a network input; processing the network input through a gater neural network to generate a gating vector that includes a respective value for each of a plurality of filters; determining, from the gating vector and for each of the plurality of filters, whether the filter is active or inactive; and processing the network input through the main convolutional neural network to generate an image processing output, comprising, for each convolutional layer in the first plurality of convolutional layers: receiving an input feature map for the convolutional layer; and generating an output feature map, the generating comprising: for each filter of the convolutional layer that is inactive: setting the output channel for the filter to have all zero elements.