Abstract: A cross-modality knowledge transfer system is provided for adapting one or more source model networks to one or more target model networks. The system is configured to perform steps of providing the TI paired datasets through the source feature encoders of the one or more source model networks, extracting TI source features and TI source moments from the TI paired data by the BN layers of the one or more source model networks, providing the TI paired datasets and the unlabeled TR datasets through the one or more target model networks to extract TI target features and TR target moments, training jointly all the feature encoders of the one or more target model networks by matching the extracted TI target features and TR target moments with the TI source features and TI source moments along with mixing weights, and forming a final target model network by combining the trained one or more target model networks.