This thesis explores engineering studies and designs aiming to meeting the requirements of enhancing capacity and energy efficiency for next generation communication networks. Challenges of spectrum scarcity and energy constraints are addressed and new technologies are proposed, analytically investigated and examined.
The thesis commences by reviewing studies on spectrally and energy-efficient techniques, with a special focus on non-orthogonal multicarrier modulation, particularly spectrally efficient frequency division multiplexing (SEFDM). Rigorous theoretical and mathematical modelling studies of SEFDM are presented. Moreover, to address the potential application of SEFDM under the 5th generation new radio (5G NR) heterogeneous numerologies, simulation-based studies of SEFDM coexisting with orthogonal frequency division multiplexing (OFDM) are conducted. New signal formats and corresponding transceiver structure are designed, using a Hilbert transform filter pair for shaping pulses. Detailed modelling and numerical investigations show that the proposed signal doubles spectral efficiency without performance degradation, with studies of two signal formats; uncoded narrow-band internet of things (NB-IoT) signals and unframed turbo coded multi-carrier signals. The thesis also considers using constellation shaping techniques and SEFDM for capacity enhancement in 5G system. Probabilistic shaping for SEFDM is proposed and modelled to show both transmission energy reduction and bandwidth saving with advantageous flexibility for data rate adaptation. Expanding on constellation shaping to improve performance further, a comparative study of multidimensional modulation techniques is carried out. A four-dimensional signal, with better noise immunity is investigated, for which metaheuristic optimisation algorithms are studied, developed, and conducted to optimise bit-to-symbol mapping. Finally, a specially designed machine learning technique for signal and system design in physical layer communications is proposed, utilising the application of autoencoder-based end-to-end learning. Multidimensional signal modulation with multidimensional constellation shaping is proposed and optimised by using machine learning techniques, demonstrating significant improvement in spectral and energy efficiencies
