Abstract: A data processing system comprising a plurality of processing nodes, each comprising at least one memory configured to store an array of data items, wherein each of the plurality of processing nodes is configured to execute compute instructions during a compute phase and following a precompiled synchronisation barrier, enter at least one exchange phase. During the at least one exchange phase, a series of collective operations are carried out. Each processing node is configured to perform a reduce scatter collective in at least one first dimension. Using the results of the reduce scatter collective, each processing node performs an allreduce in a second dimension. The processing nodes then perform an all-gather collective in the at least one first dimension using the results of the allreduce.

Abstract: In a data processing system, at least one processing node is configured to perform computations for a multi-stage process whilst at least one other processor performs the load/unload operations required to calculate a subsequent stage of the multi stage process. An exchange of data then occurs between the processing nodes. At a later time, at least one processing node performs calculations using the data loaded from storage, whilst at least one other processor performs the load/unload operations required to calculate a subsequent stage of the multi stage process.

Abstract: A data processing system comprising a plurality of processing nodes that are arranged to update a model in a parallel manner Each of the processing nodes starts with a different set of updates to model parameters. Each of the processing nodes is configured to perform one or more reduce-scatter collectives so as to exchange and reduce the updates. Having done so, each processing node is configured to apply the reduced set of updates to obtain an updated set of model parameters. The processing nodes then exchange the updated model parameters using an all-gather so that each processing node ends up with the same model parameters at the end of the process.

Abstract: A data processing system comprising a plurality of processing nodes, each comprising at least one memory configured to store an array of data items, wherein each of the plurality of processing nodes is configured to execute compute instructions during a compute phase and following a precompiled synchronisation barrier, enter at least one exchange phase. During the at least one exchange phase, a series of collective operations are carried out. Each processing node is configured to perform a reduce scatter collective in at least one first dimension. Using the results of the reduce scatter collective, each processing node performs an allreduce in a second dimension. The processing nodes then perform an all-gather collective in the at least one first dimension using the results of the allreduce.

Abstract: In a data processing system, at least one processing node is configured to perform computations for a multi-stage process whilst at least one other processor performs the load/unload operations required to calculate a subsequent stage of the multi stage process. An exchange of data then occurs between the processing nodes. At a later time, at least one processing node performs calculations using the data loaded from storage, whilst at least one other processor performs the load/unload operations required to calculate a subsequent stage of the multi stage process.