Abstract: Some disclosed embodiments are directed to computing systems having different accelerators such that a first set of accelerators has a greater memory capability than a second set accelerators, while the second set of accelerators has a greater processing capability than the first set of accelerators. A machine learning model having different dense layers and sparse layers is distributed on the different accelerators such that the dense layers are distributed on one or more accelerators selected from the first set of accelerators and the sparse layers are distributed on one or more accelerators in the second set of accelerators.

Abstract: Some disclosed embodiments are directed to computing systems having different accelerators such that a first set of accelerators has a greater memory capability than a second set accelerators, while the second set of accelerators has a greater processing capability than the first set of accelerators. A machine learning model having different dense layers and sparse layers is distributed on the different accelerators such that the dense layers are distributed on one or more accelerators selected from the first set of accelerators and the sparse layers are distributed on one or more accelerators in the second set of accelerators.

Abstract: Methods are provided for identifying and redistributing the experts of MOE machine learning models that are distributed on different accelerators. Systems identify a set of input tokens to be routed to the plurality of experts and identify a routing assignment of the set of input tokens to the plurality of experts. After identifying a current distribution of the plurality of experts on the plurality of accelerators, systems determine a new distribution of the plurality of experts on the plurality of accelerators which will result in an improved processing efficiency of the set of input tokens by the plurality of accelerators based on the routing assignment of the set of tokens to the plurality of experts as compared to the current distribution of the plurality of experts on the plurality of accelerators. The new distribution of the plurality of experts is also applied to realize the anticipated improvements in processing efficiencies.

Abstract: Some disclosed embodiments are directed to computing systems having different accelerators such that a first set of accelerators has a greater memory capability than a second set accelerators, while the second set of accelerators has a greater processing capability than the first set of accelerators. A machine learning model having different dense layers and sparse layers is distributed on the different accelerators such that the dense layers are distributed on one or more accelerators selected from the first set of accelerators and the sparse layers are distributed on one or more accelerators in the second set of accelerators.