Intelligent AutoEncoder-Based Modulation: Optimizing Transmission Performance over Non-Ideal Wireless Channels

Abstract

This paper presents an End-to-End wireless transceiver architecture based on deep AutoEncoder (AE) networks that jointly optimizes the transmitter and receiver as a single differentiable system, replacing the conventional cascade of independently designed signal processing blocks. The channel model incorporates three concurrent non-ideal impairments: nonlinear distortion from the power amplifier (PA) characterized by the Rapp model, progressive carrier frequency offset (CFO), and flat Rayleigh fading. Through the training process and testing scenarios across three progressively evolving architectures, namely single-symbol constellation shaping, multi-symbol blind CFO compensation, and implicit neural forward error correction (Neural FEC), the obtained results confirm that the AE is capable of autonomously learning PAresilient signal constellations, performing blind CFO estimation without pilot signals, and unifying the modulation and channel coding processes into a single optimal system representation. Monte Carlo BER simulations show that the proposed architecture achieves 3–5 dB SNR gain over conventional 16-QAM with ZF equalization, provides 2–3 dB gain relative to ideally CFO-compensated 16-QAM, and the Neural FEC configuration successfully performs the channel coding function, exhibiting effective error correction performance