Abstract: A method includes receiving a trained deep neural network (DNN) having multiple layers represented by respective weight matrices. The DNN is pruned by, for at least one layer represented by a respective weight matrix including multiple weights, converting the weight matrix into a structured sparse weight matrix by (a) defining multiple M-element line-blocks in the weight matrix, each line-block including the weights along an M-element portion of a row of the weight matrix, and respective multiple M-element column-blocks in a transpose of the weight matrix (b) specifying a subset of at least N weights in the weight matrix that are to be nulled in each M-element line-block and respective at least N weights to be nulled in each M-element column-block of the transpose of the weight matrix, and (c) nulling the weights in the subset. A deep learning operation is performed using the pruned DNN.

Abstract: A DNN can be compressed by pruning one or more tensors for a deep learning operation. A first pruning parameter and a second pruning parameter are determined for a tensor. A vector having a size of the second pruning parameter may be extracted from the tensor. Pruning probabilities may be determined for the elements in the vector. One or more elements in the vector are selected based on the pruning probabilities. Alternatively, a matrix, in lieu of the vector, may be extracted from the tensor. Pruning probabilities may be determined for the columns in the matrix. One or more columns are selected based on their pruning probabilities. The number of the selected element(s) or column(s) may equal the first pruning parameter. The tensor can be modified by modifying the value(s) of the selected element(s) or column(s) and setting the value(s) of one or more unselected elements or columns to zero.