Abstract: Methods and systems for compressing a matrix are described. The matrix, having a plurality of rows formed by a respective plurality of vectors, is partitioned into a plurality of submatrices, each submatrix containing sub-vectors from a respective group of one or more contiguous columns of the matrix. For each given submatrix, the sub-vectors are clustered into a plurality of clusters. For each given cluster, a centroid and a variance are computed and stored, based on the sub-vectors belonging to the given cluster. A mapping relating each vector to a respective cluster in each submatrix is stored. The stored centroids, stored variances and stored mapping form a set of compressed data for reconstruction of the matrix.

Abstract: Methods and systems for training a neural network model using domain mixing and multi-teacher knowledge distillation are described. Tokens, including a unique token, are inputted to an encoder of the neural network model. A unique embedding vector encoded from the unique token is inputted to an adaptor network to generate domain probabilities. A domain mixing embedding vector, determined from the unique embedding vector, is inputted to a predictor of the neural network model, to generate a predicted output. A final loss is computed using a domain mixing loss computed from the domain probabilities and a ground-truth domain of the data sample, and using an output prediction loss computed from the predicted output and a ground-truth label of the data sample. Parameters of the neural network model and adaptor network are updated using the final loss.

Abstract: An agnostic combinatorial knowledge distillation (CKD) method for transferring trained knowledge of neural model from a complex model (teacher) to a less complex model (student) is described. In addition to training the student to generate a final output that approximates both the teacher's final output and a ground truth of a training input, the method further maximizes knowledge transfer by training hidden layers of the student to generate outputs that approximate a representation of a subset of teacher hidden layers are mapped to each of the student hidden layers for a given training input.

Abstract: Methods and systems for compressing a matrix are described. The matrix, having a plurality of rows formed by a respective plurality of vectors, is partitioned into a plurality of submatrices, each submatrix containing sub-vectors from a respective group of one or more contiguous columns of the matrix. For each given submatrix, the sub-vectors are clustered into a plurality of clusters. For each given cluster, a centroid and a variance are computed and stored, based on the sub-vectors belonging to the given cluster. A mapping relating each vector to a respective cluster in each submatrix is stored. The stored centroids, stored variances and stored mapping form a set of compressed data for reconstruction of the matrix.