In this paper, we propose a deep learning-based signal detector called
TransD3D-IM, which employs the Transformer framework for signal detection in
the Dual-mode index modulation-aided three-dimensional (3D) orthogonal
frequency division multiplexing (DM-IM-3D-OFDM) system. In this system, the
data bits are conveyed using dual-mode 3D constellation symbols and active
subcarrier indices. As a result, this method exhibits significantly higher
transmission reliability than current IM-based models with traditional maximum
likelihood (ML) detection. Nevertheless, the ML detector suffers from high
computational complexity, particularly when the parameters of the system are
large. Even the complexity of the Log-Likelihood Ratio algorithm, known as a
low-complexity detector for signal detection in the DM-IM-3D-OFDM system, is
also not impressive enough. To overcome this limitation, our proposal applies a
deep neural network at the receiver, utilizing the Transformer framework for
signal detection of DM-IM-3D-OFDM system in Rayleigh fading channel. Simulation
results demonstrate that our detector attains to approach performance compared
to the model-based receiver. Furthermore, TransD3D-IM exhibits more robustness
than the existing deep learning-based detector while considerably reducing
runtime complexity in comparison with the benchmarks