Abstract: Systems and methods for privacy-preserving transformer model training are provided. The system includes one or more data repositories in a computer network or cloud infrastructure having data stored therein. The system anonymizes the data in the one or more documents, and trains a transformer model on the data outside of the network. The data includes sensitive information. Anonymizing the data includes extracting the data from the one or more documents and irreversibly transforming the data in the one or more documents into context-preserving tensors. Training the transformer model on the data comprises using the context-preserving tensors instead of the data to train the transformer model on the data.