Abstract: Methods and systems for model compression include determining importance values for respective parameters in a pre-trained model corresponding to general knowledge of the pre-trained model. Loss values are determined for removal of the parameters based on the importance values and a regularization term corresponding to domain-specific knowledge. Parameters are pruned from the pre-trained model based on the loss values to create a pruned model.

Abstract: Systems and methods for optimizing large language models (LLM) with domain-oriented model compression. Importance weights for general knowledge in a trained LLM, pretrained with deep learning, can be determined by computing the error when removing a weight from the trained LLM. The trained LLM can be iteratively optimized to obtain a domain-compressed LLM with domain knowledge while maintaining general knowledge by: fine-tuning the trained LLM iteratively with domain knowledge using the importance weights for general knowledge to obtain a fine-tuned LLM; determining importance weights for domain knowledge in the LLM with a regularization term by using gradient descent to optimize parameters when the fine-tuned LLM is trained with domain knowledge; and pruning learned knowledge based on importance weights for domain knowledge. A corrective action can be performed on a monitored entity using the domain-compressed LLM.