Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for accounting for long-tail training data.

Abstract: A method includes receiving, by a computing device, training data to train a neural network, wherein the training data comprises a plurality of inputs and a plurality of corresponding labels. The method also includes mapping, by a representation learner of the neural network, the plurality of inputs to a plurality of feature vectors. The method additionally includes training a kernelized classification layer of the neural network to perform nonlinear classification of an input feature vector into one of a plurality of classes, wherein the kernelized classification layer is based on a kernel which enables the nonlinear classification, and wherein the kernel is selected from a space of positive definite kernels based on application of a nonlinear softmax loss function to the plurality of feature vectors and the plurality of corresponding labels. The method further includes outputting a trained neural network comprising the representation learner and the trained kernelized classification layer.