Square-root Nyquist filter design for QAM-based filter bank multicarrier systems

Filter bank multicarrier systems with quadrature amplitude modulation (FBMC/QAM) have drawn attentions to get the advantage of complex symbol transmission, as well as very low out of band radiation and relaxed synchronization requirements for asynchronous scenarios. In order to make this system viable for practical deployment, the biggest challenge is designing appropriate filters to minimize the interference between adjacent subcarriers, while maintaining the Nyquist property of the filter. We show that the deviation from the Nyquist property can be compensated through the fractional shift of the filtered symbols, which provides flexibility to optimize the stopband of the filter. The proposed design method shows advantages over the state of the art designs, and provides guidance for the filter design in practical FBMC/QAM systems