Direction of arrival (DOA) estimation problem for multiple-input multiple-output (MIMO) radar with unknown mutual coupling is studied, and an algorithm for the DOA estimation based on root multiple signal classification (MUSIC) is proposed. Firstly, according to the Toeplitz structure of the mutual coupling matrix, output data of some specified sensors are selected to eliminate the influence of the mutual coupling. Then the reduced-dimension transformation is applied to make the computation burden lower as well as obtain a Vandermonde structure of the direction matrix. Finally, Root-MUSIC can be adopted for the angle estimation. The angle estimation performance of the proposed algorithm is better than that of estimation of signal parameters via rotational invariance techniques (ESPRIT)-like algorithm and MUSIC-like algorithm. Furthermore, the proposed algorithm has lower complexity than them. The simulation results verify the effectiveness of the algorithm, and the theoretical estimation error of the algorithm is also derived
