Abstract: A system includes a rack with multiple hardware acceleration devices and multiple modular controllers coupled together into a single system implementing one or more servers. Each modular hardware acceleration device includes multiple hardware accelerators, such as graphical processing units, field programmable gate arrays or other specialized processing circuits. In each modular hardware acceleration device, hardware accelerators are communicatively coupled to a multi-port connection device, such as a switch, and also communicatively coupled to at least two external ports. A modular controller of a particular server coordinates operation of hardware accelerators of multiple hardware acceleration devices included in the particular server to provide advanced processing capabilities. Hardware accelerators may be dynamically assigned to particular processing servers to adjust processing capabilities of those servers.