Abstract: A system and method are provided for generating a trained model to filter data sets for filtering hate speech. The method includes obtaining an unfiltered corpus of data, obtaining a set of trigger phrases, and using the set of trigger phrases to generate a trained model which comprises at least one conditional likelihood of the trigger phrases conditioned on documents in the corpus of data. A system and method are also provided for filtering data sets for hate speech using pre-trained models. The method includes obtaining a pretrained model generated using a set of trigger phrases and which comprises at least one conditional likelihood of the trigger phrases conditioned on document in a corpus of data used to generate the pretrained model; using the pretrained model to filter an unfiltered dataset and generate a filtered dataset; and outputting the filtered dataset.

Abstract: A system for training the neural network using dropout with slicing operations preserves the regularization effects of dropout, while speeding up computations and reducing the memory requirements of training the neural network. Instead of randomly dropping weights connected to neurons in a neural network, the system slices contiguous memory segments of weight matrices. For transformer models, the approach first receives input data that consist of a sequence of elements. Based on the input data, input embedding vectors with positional encoding are generated. Then the transformer model is trained by passing the input embedding vectors through various neural network layers. While passing through linear layers, some of the weight matrices are sliced (e.g., masked) such that a contiguous section of a weight matrix is kept unsliced and used for training and the rest of the weight matrix is not accessed.

Abstract: A system for training the neural network using dropout with slicing operations preserves the regularization effects of dropout, while speeding up computations and reducing the memory requirements of training the neural network. Instead of randomly dropping weights connected to neurons in a neural network, the system slices contiguous memory segments of weight matrices. For transformer models, the approach first receives input data that consist of a sequence of elements. Based on the input data, input embedding vectors with positional encoding are generated. Then the transformer model is trained by passing the input embedding vectors through various neural network layers. While passing through linear layers, some of the weight matrices are sliced (e.g., masked) such that a contiguous section of a weight matrix is kept unsliced and used for training and the rest of the weight matrix is not accessed.

Abstract: The present disclosure relates to a system, method and non-transitory computer readable medium for training language models. The exemplary method includes obtaining a first language model. The method includes using a determined set of weights of the first language model to initialize a second language model. The first and second language model are different model types. The method includes applying the second language model to perform an operation.

Abstract: A method of training a neural network model and related systems are disclosed. The method includes training the neural network model by factorising, based on a singular value decomposition scheme, a first plurality of nodes of the neural network model into a low rank neural network model comprising a second plurality of nodes. Each node of the second plurality of nodes is defined at least in part by at least one weight matrix, and the factorisation is based on a matrix decomposition scheme constrained by one or more directionality criteria.

Abstract: A system and method are provided for generating a trained model to filter data sets for filtering hate speech. The method includes obtaining an unfiltered corpus of data, obtaining a set of trigger phrases, and using the set of trigger phrases to generate a trained model which comprises at least one conditional likelihood of the trigger phrases conditioned on documents in the corpus of data. A system and method are also provided for filtering data sets for hate speech using pre-trained models. The method includes obtaining a pretrained model generated using a set of trigger phrases and which comprises at least one conditional likelihood of the trigger phrases conditioned on document in a corpus of data used to generate the pretrained model; using the pretrained model to filter an unfiltered dataset and generate a filtered dataset; and outputting the filtered dataset.