Abstract: A scalable neural network system may include a root processor and a plurality of neural network processors with a tree of synchronizing sub-systems connecting them together. Each synchronization sub-system may connect one parent to a plurality of children. Furthermore, each of the synchronizing sub-systems may simultaneously distribute weight updates from the root processor to the plurality of neural network processors, while statistically combining corresponding weight gradients from its children into single statistical weight gradients. A generalized network of sensor-controllers may have a similar structure.

Abstract: A multi-processor system for batched pattern recognition may utilize a plurality of different types of neural network processors and may perform batched sets of pattern recognition jobs on a two-dimensional array of inner product units (IPUs) by iteratively applying layers of image data to the IPUs in one dimension, while streaming neural weights from an external memory to the IPUs in the other dimension. The system may also include a load scheduler, which may schedule batched jobs from multiple job dispatchers, via initiators, to one or more batched neural network processors for executing the neural network computations.

Abstract: A multi-processor system for data processing may utilize a plurality of different types of neural network processors to perform, e.g., learning and pattern recognition. The system may also include a scheduler, which may select from the available units for executing the neural network computations, which units may include standard multi-processors, graphic processor units (GPUs), virtual machines, or neural network processing architectures with fixed or reconfigurable interconnects.