Abstract. Deep learning-based successive interference cancellation (DL-SIC) for uplink multiple-input multiple-output -non-orthogonal multiple access (MIMO-NOMA) system tries to optimize the usersโ bit error rate (BER) and total mean square error (MSE) performance with higher order modulation schemes. The recent work of DL-SIC receiver design for users with a QPSK modulation scheme is investigated in this thesis to validate its performance as a potential alternative approach to traditional SIC receivers for NOMA users. Then, a DL-SIC receiver design for higher order modulation with less dependence on modulation order in the output layer is proposed, which enables us to decode the users with different modulation schemes. In our proposed design, we employ two deep neural networks (DNNs) for each SIC step. The system model is considered an M-antenna base station (BS) that serves two uplink users with a single antenna in the Rayleigh fading channel. The equivalent conventional minimum mean square error-based SIC (MMSE-SIC) and zero-forcing-based SIC (ZF-SIC) receivers are implemented as a baseline comparison.
The simulation results showed that the BER performance of the proposed DL-SIC receiver for both users with QPSK modulation results in a 10 dB gain between BER of 10^(-2) and 10^(-3) compared to the ZF-SIC receiver. Furthermore, the performance difference between the proposed scheme and ZF-SIC is significantly high when both users transmit with 16QAM. Overall, the proposed DL-SIC receiver performs better in all signal-to-noise ratio (SNR) regions than the equivalent ZF-SIC receivers and also aids in mitigating the SIC error propagation problem. In addition, it improves the processing latency due to the benefits of the parallelized computing architecture and decreases the complexity of traditional SIC receivers
