Abstract: The direction-of-arrival estimation and mutual coupling calibration method and system with arbitrary sensor geometry and unknown mutual moupling are disclosed, a direction-of-arrival estimation and mutual coupling calibration method with arbitrary sensor geometry and unknown mutual moupling, comprising the following steps: acquiring an array signal, and then estimating a covariance matrix according to the array signal, a noise subspace is obtained by eigendecomposition of the covariance matrix; determining the angle search range of DOA, and generating a group of grids according to the angle search range of DOA, based on the noise subspace and the mutual coupling matrix between the elements, and obtaining a spectrum function corresponding to each grid; Estimating DOA based on the peak value of spectrum function corresponding to each grid, and obtaining the mutual coupling coefficient between the array elements from the DOA.