Abstract: A method includes training, using a first computing system having a first configuration, a first machine learning model having a machine learning model architecture, and training, using a second computing system having a different second configuration, a second machine learning model having the machine learning model architecture. The method also includes determining, for a shared training operation performed by both the first computing system and the second computing system, a similarity measure that represents a similarity between: a first training output generated by the first computing system during performance of the shared training operation during training of the first machine learning model; and a second training output generated by the second computing system during performance of the shared training operation during training of the second machine learning model.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for debugging correctness issues in training machine learning models. In one aspect, a method comprises training a first machine learning model using a first computing system having a first configuration; training a second machine learning model using a second computing system having a second configuration, wherein the second configuration of the second computing system is different than the first configuration of the first computing system; and determining, for each of a plurality of shared training operations that are performed by both the first computing system and the second computing system, a respective similarity measure that measures a similarity between: a first training output generated by the first computing system by performing the shared training operation, and a second training output generated by the second computing system by performing the shared training operation.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for debugging correctness issues in training machine learning models. In one aspect, a method comprises training a first machine learning model using a first computing system having a first configuration; training a second machine learning model using a second computing system having a second configuration, wherein the second configuration of the second computing system is different than the first configuration of the first computing system; and determining, for each of a plurality of shared training operations that are performed by both the first computing system and the second computing system, a respective similarity measure that measures a similarity between: a first training output generated by the first computing system by performing the shared training operation, and a second training output generated by the second computing system by performing the shared training operation.