Classical methods of DOA estimation such as the MUSIC algorithm are based on
estimating the signal and noise subspaces from the sample covariance matrix.
For a small number of samples, such methods are exposed to performance
breakdown, as the sample covariance matrix can largely deviate from the true
covariance matrix. In this paper, the problem of DOA estimation performance
breakdown is investigated. We consider the structure of the sample covariance
matrix and the dynamics of the root-MUSIC algorithm. The performance breakdown
in the threshold region is associated with the subspace leakage where some
portion of the true signal subspace resides in the estimated noise subspace. In
this paper, the subspace leakage is theoretically derived. We also propose a
two-step method which improves the performance by modifying the sample
covariance matrix such that the amount of the subspace leakage is reduced.
Furthermore, we introduce a phenomenon named as root-swap which occurs in the
root-MUSIC algorithm in the low sample size region and degrades the performance
of the DOA estimation. A new method is then proposed to alleviate this problem.
Numerical examples and simulation results are given for uncorrelated and
correlated sources to illustrate the improvement achieved by the proposed
methods. Moreover, the proposed algorithms are combined with the pseudo-noise
resampling method to further improve the performance.Comment: 37 pages, 10 figures, Submitted to the IEEE Transactions on Signal
Processing in July 201