Abstract: Methods and systems for compressing a neural network (NN) which performs an inference task and for performing computations of a Kronecker layer of a Kronecker NN are described. Data samples are obtained from a training dataset. The input data of the data samples are inputted into a trained NN to generate NN predictions for the input data. Further, the input data are inputted into a Kronecker NN to generate Kronecker NN predictions for the input data. Two losses are computed: a knowledge distillation loss, based on outputs generated by a layer of the NN and a corresponding Kronecker layer of the Kronecker NN and a loss for Kronecker layer, based on the Kronecker NN predictions and ground-truth labels of the data samples. The two losses are combined into a total loss, which is propagated through the Kronecker NN to adjust values of learnable parameters of the Kronecker NN.

Abstract: Methods and processors for using a Neural Network (NN) are disclosed. The method includes, during a first iteration, determining a first sub-network of the NN and training the first sub-network. The method includes during a second iteration a second sub-network of the NN and training the second sub-network. The method includes during an inference iteration of the NN, selecting a target sub-network amongst the first sub-network and the second sub-network, and generating an inference output by employing only the target sub-network of the NN on inference data for reducing computational resources of the processor for generating the inference output.

Abstract: Methods and systems for compressing a matrix are described. The matrix, having a plurality of rows formed by a respective plurality of vectors, is partitioned into a plurality of submatrices, each submatrix containing sub-vectors from a respective group of one or more contiguous columns of the matrix. For each given submatrix, the sub-vectors are clustered into a plurality of clusters. For each given cluster, a centroid and a variance are computed and stored, based on the sub-vectors belonging to the given cluster. A mapping relating each vector to a respective cluster in each submatrix is stored. The stored centroids, stored variances and stored mapping form a set of compressed data for reconstruction of the matrix.

Abstract: Method and system of training a student neural network (SNN) model. A first training phase is performed over a plurality of epochs during which a smoothing factor to teacher neural network (TNN) model outputs to generate smoothed TNN model outputs, a first loss is computed based on the SNN model outputs and the smoothed TNN model outputs, and an updated set of the SNN model parameters is computed with an objective of reducing the first loss in a following epoch of the first training phase. The soothing factor is adjusted over the plurality of epochs of the first training phase to reduce a smoothing effect on the generated smoothed TNN model outputs. A second training phase is performed based on the SNN model outputs and a set of predefined expected outputs for the plurality of input data samples.

Abstract: Method and system of training a student neural network using adversarial learning and knowledge distillation, including: training a generator to generate adversarial data samples for respective training data samples by masking parts of the training data samples with an objective of maximizing a divergence between output predictions generated by the student neural network and a teacher neural network model for the adversarial data samples; and training the student neural network based on objectives of (i) minimizing a divergence between output predictions generated by the student neural network and the teacher neural network model for the adversarial data samples, and (ii) minimizing a divergence between output predictions generated by the student neural network and the teacher neural network model for the training data samples.

Abstract: According to embodiments, an encoder neural network receives a one-hot representation of a real text. The encoder neural network outputs a latent representation of the real text. A decoder neural network receives random noise data or artificial code generated by a generator neural network from random noise data. The decoder neural network outputs softmax representation of artificial text. The decoder neural network receives the latent representation of the real text. The decoder neural network outputs a reconstructed softmax representation of the real text. A hybrid discriminator neural network receives a first combination of the soft-text and the latent representation of the real text and a second combination of the softmax representation of artificial text and the artificial code. The hybrid discriminator neural network outputs a probability indicating whether the second combination is similar to the first combination. Additional embodiments for utilizing latent representation are also disclosed.

Abstract: A method and machine translation system for bi-directional translation of textual sequences between a first language and a second language are described. The machine translation system includes a first autoencoder configured to receive a vector representation of a first textual sequence in the first language and encode the vector representation of the first textual sequence into a first sentence embedding. The machine translation system also includes a sum-product network (SPN) configured to receive the first sentence embedding and generate a second sentence embedding by maximizing a first conditional probability of the second sentence embedding given the first sentence embedding and a second autoencoder receiving the second sentence embedding, the second autoencoder being trained to decode the second sentence embedding into a vector representation of a second textual sequence in the second language.

Abstract: Methods and systems for compressing a neural network which performs an inference task and for performing computations of a Kronecker layer of a Kroenke NN are described. A batch of data samples are obtained from a training dataset. The input data of the data samples are inputted into a trained neural network to forward propagate the input data through the trained neural network and generate neural network predictions for the input data. Further, the input data are inputted into a Kronecker neural network to forward propagate the input data through the Kronecker neural network to generate Kronecker neural network predictions for the input data. Afterwards, two losses are computed: a knowledge distillation loss and a loss for Kronecker layer. The knowledge distillation loss is based on outputs generated by a layer of the neural network and a corresponding Kronecker layer of the Kronecker neural network.

Abstract: Methods, devices and processor-readable media for re-weighting to improve knowledge distillation are described. A reweighting module may be used to determine relative weights to assign to a ground truth label and dark knowledge distilled from the teacher (i.e. the teacher output logits used as soft labels). A meta-reweighting method is described to optimize the weights for a given labeled data sample.

Abstract: Methods and systems for training a neural network model using domain mixing and multi-teacher knowledge distillation are described. Tokens, including a unique token, are inputted to an encoder of the neural network model. A unique embedding vector encoded from the unique token is inputted to an adaptor network to generate domain probabilities. A domain mixing embedding vector, determined from the unique embedding vector, is inputted to a predictor of the neural network model, to generate a predicted output. A final loss is computed using a domain mixing loss computed from the domain probabilities and a ground-truth domain of the data sample, and using an output prediction loss computed from the predicted output and a ground-truth label of the data sample. Parameters of the neural network model and adaptor network are updated using the final loss.

Abstract: Methods, devices and processor-readable media for knowledge distillation using intermediate representations are described. A student model is trained using a Dropout-KD approach in which intermediate layer selection is performed efficiently such that the skip, search, and overfitting problems in intermediate layer KD may be solved. Teacher intermediate layers are selected randomly at each training epoch, with the layer order preserved to avoid breaking information flow. Over the course of multiple training epochs, all of the teacher intermediate layers are used for knowledge distillation. A min-max data augmentation method is also described based on the intermediate layer selection of the Dropout-KD training method.

Abstract: In accordance to embodiments, an encoder neural network is configured to receive a one-hot representation of a real text and output a latent representation of the real text generated from the one-hot representation of the real text. A decoder neural network is configured to receive the latent representation of the real text, and output a reconstructed softmax representation of the real text from the latent representation of the real text, the reconstructed softmax representation of the real text is a soft-text. A generator neural network is configured to generate artificial text based on random noise data. A discriminator neural network is configured to receive the soft-text and receive a softmax representation of the artificial text, and output a probability indicating whether the softmax representation of the artificial text received by the discriminator neural network is not from the generator neural network.

Abstract: Methods and systems for compressing a matrix are described. The matrix, having a plurality of rows formed by a respective plurality of vectors, is partitioned into a plurality of submatrices, each submatrix containing sub-vectors from a respective group of one or more contiguous columns of the matrix. For each given submatrix, the sub-vectors are clustered into a plurality of clusters. For each given cluster, a centroid and a variance are computed and stored, based on the sub-vectors belonging to the given cluster. A mapping relating each vector to a respective cluster in each submatrix is stored. The stored centroids, stored variances and stored mapping form a set of compressed data for reconstruction of the matrix.

Abstract: An agnostic combinatorial knowledge distillation (CKD) method for transferring trained knowledge of neural model from a complex model (teacher) to a less complex model (student) is described. In addition to training the student to generate a final output that approximates both the teacher's final output and a ground truth of a training input, the method further maximizes knowledge transfer by training hidden layers of the student to generate outputs that approximate a representation of a subset of teacher hidden layers are mapped to each of the student hidden layers for a given training input.

Abstract: A method and machine translation system for bi-directional translation of textual sequences between a first language and a second language are described. The machine translation system includes a first autoencoder configured to receive a vector representation of a first textual sequence in the first language and encode the vector representation of the first textual sequence into a first sentence embedding. The machine translation system also includes a sum-product network (SPN) configured to receive the first sentence embedding and generate a second sentence embedding by maximizing a first conditional probability of the second sentence embedding given the first sentence embedding and a second autoencoder receiving the second sentence embedding, the second autoencoder being trained to decode the second sentence embedding into a vector representation of a second textual sequence in the second language.

Abstract: A method of and system for compressing a deep neural network model using knowledge distillation.

Abstract: In at least one broad aspect, described herein are systems and methods in which a latent representation shared between two languages is built and/or accessed, and then leveraged for the purpose of text generation in both languages. Neural text generation techniques are applied to facilitate text generation, and in particular the generation of sentences (i.e., sequences of words or subwords) in both languages, in at least some embodiments.

Abstract: A method and system for augmenting a training dataset for a generative adversarial network (GAN). The training dataset includes labelled data samples and unlabelled data samples. The method includes: receiving generated samples generated using a first neural network of the GAN and the unlabelled samples of training dataset; determining a decision value for a sample from a decision function, wherein the sample is a generated sample of the generated samples or an unlabelled sample of the unlabelled samples of the training dataset; comparing the decision value to a threshold; in response to determining that the decision value exceeds the threshold: predicting a label for a sample; assigning the label to the sample; and augmenting the training dataset to include the sample with the assigned label as a labelled sample.

Abstract: Method and system for performing semi-supervised regression with a generative adversarial network (GAN) that includes a generator comprising a first neural network and a discriminator comprising a second neural network, comprising: outputting, from the first neural network, generated samples derived from a random noise vector; inputting, to the second neural network, the generated samples, a plurality of labelled training samples, and a plurality of unlabelled training samples; and outputting, from the second neural network, a predicted continuous label for each of a plurality of the generated samples and unlabelled samples.

Abstract: According to embodiments, an encoder neural network receives a one-hot representation of a real text. The encoder neural network outputs a latent representation of the real text. A decoder neural network receives random noise data or artificial code generated by a generator neural network from random noise data. The decoder neural network outputs softmax representation of artificial text. The decoder neural network receives the latent representation of the real text. The decoder neural network outputs a reconstructed softmax representation of the real text. A hybrid discriminator neural network receives a first combination of the soft-text and the latent representation of the real text and a second combination of the softmax representation of artificial text and the artificial code. The hybrid discriminator neural network outputs a probability indicating whether the second combination is similar to the first combination. Additional embodiments for utilizing latent representation are also disclosed.