Transceiver design for interference MIMO relay systems with direct links

In this paper, we consider interference multiple-input multiple-output (MIMO) relay systems where multiple pairs of transmitters-receivers simultaneously communicate through a single relay node and where all nodes are equipped with multiple antennas. We propose a new iterative algorithm to jointly optimize the relay precoding matrix and the receiver matrices of such a system based on the minimum sum mean-squared error (MSE) criterion. The optimal structure of the relay precoding matrix is developed to decrease the computational complexity of transceiver optimization. Simulation results demonstrate that the proposed transceiver optimization algorithm has a betterMSE and biterror-rate (BER) performance and a faster convergence rate than existing works