Abstract: Disclosed are a training method and apparatus for distributed training of a neural network, the training apparatus including processors configured to perform distributed training, wherein each of the processors is further configured to perform a forward direction operation for layers of the neural network, determine a loss of the neural network based on the forward direction operation, determine a local gradient for each layer of the neural network by performing a backward direction operation for the layers of the neural network based on the loss, determine whether to perform gradient clipping for a local gradient determined for a previous layer, in response to determining a local gradient for a current layer through the backward direction operation, determine an aggregated gradient based on the backward direction operation and the gradient clipping performed by each of the processors, and update parameters of the neural network based on the aggregated gradient.

Abstract: A processor-implemented method with data loading includes: dividing a training data set into a plurality of subsets based on sizes of a plurality of data files included in the training data set; loading, from each of the plurality of subsets, a portion of data files in the subset to a plurality of processors based on a proportion of a number of data files of the plurality of subsets in the subset and a batch size of distributed training; and reallocating, based on sizes of data files loaded to processors in a same group among the plurality of processors, the loaded data files to the processors in the same group.