Abstract: Technologies are disclosed herein for dynamically generating communication primitives for use in model parameter synchronization during data-parallel DNN training by packing directed spanning trees. An interconnect topology for communication between GPUs in a computing system is determined. A quantity of directed spanning trees are generated for transmitting data between the GPUs using the interconnect topology and packed. The directed spanning trees define the connections between GPUs that are to be utilized for the transmission and the amount of data to be transmitted on each connection. Program code is generated for implementing the data transfer defined by the directed spanning trees. When the program code is executed, the directed spanning trees are used to pipeline the transmission of chunks of data, such as model parameters used during data-parallel DNN training, between the GPUs. The program code can also determine an optimal chunk size for data to be transferred between the GPUs.

Abstract: Mechanisms to modify a dataflow execution graph that processes a data stream. An intermediate dataflow execution graph is used during modification of the dataflow execution graph from one configuration (the old dataflow execution graph) to the next (the new dataflow execution graph). Data messages of the data stream may continue to feed into the intermediate dataflow execution graph, thereby reducing latency and maintaining throughput during reconfiguration of the dataflow execution graph. Control message(s) that are structured to accomplish the reconfiguration is/are also passed into the intermediate dataflow execution graph during reconfiguration. As the control message(s) are all processed by the intermediate dataflow execution graph, the intermediate dataflow execution graph assumes the topology of the new dataflow execution graph.

Abstract: A sparse array is partitioned into first partitions and a second array is partitioned into second partitions based on an invariant relationship between the sparse array and the second array. The sparse array and the second array are associated with a computation involving the sparse array and the second array. The first partitions and the corresponding second partitions are distributed to workers. A different first partition and corresponding second partition is distributed to each of the workers. Third partitions of the sparse array and corresponding fourth partitions of the second array are determined based on the invariant relationship and measurements of load are received from each of the workers. At least one of the first partitions and the corresponding second partition is different from one of the third partitions and the corresponding fourth partition. The at least one of the first partitions and the corresponding second partition that is different is redistributed among the workers.

Abstract: A plurality of array partitions are defined for use by a set of tasks of the program run-time. The array partitions can be determined from one or more arrays that are utilized by the program at run-time. Each of the plurality of computing devices are assigned to perform one or more tasks in the set of tasks. By assigning each of the plurality of computing devices to perform one or more tasks, an objective to reduce data transfer amongst the plurality of computing devices can be implemented.

Abstract: A plurality of array partitions are defined for use by a set of tasks of the program run-time. The array partitions can be determined from one or more arrays that are utilized by the program at run-time. Each of the plurality of computing devices are assigned to perform one or more tasks in the set of tasks. By assigning each of the plurality of computing devices to perform one or more tasks, an objective to reduce data transfer amongst the plurality of computing devices can be implemented.

Abstract: A sparse array is partitioned into first partitions and a second array is partitioned into second partitions based on an invariant relationship between the sparse array and the second array. The sparse array and the second array are associated with a computation involving the sparse array and the second array. The first partitions and the corresponding second partitions are distributed to workers. A different first partition and corresponding second partition is distributed to each of the workers. Third partitions of the sparse array and corresponding fourth partitions of the second array are determined based on the invariant relationship and measurements of load are received from each of the workers. At least one of the first partitions and the corresponding second partition is different from one of the third partitions and the corresponding fourth partition. The at least one of the first partitions and the corresponding second partition that is different is redistributed among the workers.

Abstract: A plurality of array partitions are defined for use by a set of tasks of the program run-time. The array partitions can be determined from one or more arrays that are utilized by the program at run-time. Each of the plurality of computing devices are assigned to perform one or more tasks in the set of tasks. By assigning each of the plurality of computing devices to perform one or more tasks, an objective to reduce data transfer amongst the plurality of computing devices can be implemented.

Abstract: A continuing analytics method includes distributing continuous analytics tasks among a number of workers. The workers execute the tasks on data elements stored in a distributed data storage system. Executing a task changes the data elements. In response to the change, a worker that executed a task invokes an update to the data storage system. The worker then increments a version number related to the changed data element, updates the data elements, and notifies other workers of the updated data element.