Abstract: Some embodiments provide a method of executing a distributed application with multiple EPUs that perform computations for the application. The EPUs are connected through a network having network elements. The method iteratively provides instructions to the EPUs to perform computations associated with the application. The method configures each EPU to store results of the computations performed by the EPU in a memory connected to the network. Computation results of at least two different EPUs are stored in at least two different memories connected to the network. The method configures, for each computation result in memory, at least one network element associated with the EPU to determine, based on a set of metrics that are defined to avoid creating congestion in the network, whether the result should be forwarded through the network and to forward the result through the network only after determining that the result should be forwarded.

Abstract: Some embodiments provide a method of managing communication between graphics processing units (GPUs) through a network having multiple forwarding elements. The method uses a set of servers to generate instructions for specifying forwarding behaviors for the forwarding elements to forward data messages containing results of computations performed by the GPUs. The GPUs perform the computations in order to collectively execute a distributed application. The method distributes the generated instructions to the forwarding elements to configure the forwarding elements to implement the forwarding behavior.