Abstract: This application relates to the artificial intelligence field, and discloses a neural network search method and a related apparatus. The neural network search method includes: constructing attention heads in transformer layers by sampling a plurality of candidate operators during model search, to construct a plurality of candidate neural networks, and comparing performance of the plurality of candidate neural networks to select a target neural network with higher performance. In this application, a transformer model is constructed with reference to model search, so that a new attention structure with better performance than an original self-attention mechanism can be generated, and effect in a wide range of downstream tasks is significantly improved.

Abstract: A computer-implemented method is provided for model training performed by a processing system. The method comprises determining a set of first weights based on a first matrix associated with a source model, determining a set of second weights based on the set of first weights, forming a second matrix associated with a target model based on the set of first weights and the set of second weights, initializing the target model based on the second matrix, and training the target model.

Abstract: This application relates to the field of artificial intelligence, and provides a model training method. The method includes: obtaining a to-be-trained first neural network model, where the first neural network model includes a first operator, and the first operator is used to perform a product operation on input data and a target weight matrix; replacing the first operator in the first neural network model with a second operator, to obtain a second neural network model, where the second operator is used to perform a product operation on input data and a plurality of sub-weight matrices, and the plurality of sub-weight matrices are obtained by performing matrix factorization on the target weight matrix; and performing model training on the second neural network model to obtain a target neural network model.

Abstract: A model compression method is provided, which can be applied to the field of artificial intelligence. The method includes: obtaining a first neural network model, a second neural network model, and a third neural network model; processing first to-be-processed data using the first neural network model, to obtain a first output; processing the first to-be-processed data using the third neural network model, to obtain a second output; determining a first target loss based on the first output and the second output, and updating the second neural network model based on the first target loss, to obtain an updated second neural network model; and compressing the updated second neural network model to obtain a target neural network model. The model generated based on the method has higher processing precision.

Abstract: A text processing model training method, and a text processing method and apparatus in the natural language processing field in the artificial intelligence field are disclosed. The training method includes: obtaining training text; separately inputting the training text into a teacher model and a student model to obtain sample data output by the teacher model and prediction data output by the student model; the sample data includes a sample semantic feature and a sample label; the prediction data includes a prediction semantic feature and a prediction label; and the teacher model is a pre-trained language model used for text classification; and training a model parameter of the student model based on the sample data and the prediction data, to obtain a target student model. The method enables the student model to effectively perform knowledge transfer, thereby improving accuracy of a text processing result of the student model.