Abstract: Techniques for accelerating the training of machine learning (ML) models in the presence of network bandwidth constraints via data instance compression. For example, consider a scenario in which (1) a first computer system is configured to train a ML model on a training dataset that is stored on a second computer system remote from the first computer system, and (2) one or more network bandwidth constraints place a cap on the amount of data that may be transmitted between the two computer systems per training iteration. In this and other similar scenarios, the techniques of the present disclosure enable the second computer system to send, according to one of several schemes, a batch of compressed data instances to the first computer system at each training iteration, such that the data size of the batch is less than or equal to the data cap.

Abstract: Techniques for implementing importance sampling via machine learning (ML)-based gradient approximation are provided. In one set of embodiments, these techniques include (1) training a deep neural network (DNN) on a training dataset using stochastic gradient descent and (2) in parallel with (1), training a separate ML model (i.e., gradient approximation model) that is designed to predict gradient norms (or gradients) for the data instances in the training dataset. The techniques further include (3) applying the gradient approximation model to the training dataset on a periodic basis to generate gradient norm/gradient predictions for the data instances in the training dataset and (4) using the gradient norm/gradient predictions to update sampling probabilities for the data instances. The updated sampling probabilities can then be accessed during the ongoing training of the DNN (i.e., step (1)) to perform importance sampling of data instances and thereby accelerate the training procedure.