Abstract: Techniques for learning from highly-diverse datasets are provided. In one embodiment, the system is provided that comprises a memory that stores computer executable components. The system can comprise a processor that executes the computer executable components stored in the memory. The computer executable components can comprise a neural network component that creates a neural network comprising a router that routes the neural network to a first layer of neurons that comprises a plurality of neurons. The computer executable components can comprise a training component that performs a plurality of successive training iterations on the neural network, a first iteration of the plurality of successive training iterations comprising both training the router to route among the plurality of neurons of the first layer of neurons, and training a first neuron of the plurality of neurons of the first layer of neurons to produce a given output from a given input.