We propose two 8-dimensional (8D) modulation formats (8D-2048PRS-T1 and
8D-2048PRS-T2) with a spectral efficiency of 5.5 bit/4D-sym, where the 8
dimensions are obtained from two time slots and two polarizations. Both formats
provide a higher tolerance to nonlinearity by selecting symbols with
nonidentical states of polarization (SOPs) in two time slots. The performance
of these novel 8D modulation formats is assessed in terms of the effective
signal-to-noise ratio (SNR) and normalized generalized mutual information.
8D-2048PRS-T1 is more suitable for high SNRs, while 8D-2048PRS-T2 is shown to
be more tolerant to nonlinearities. A sensitivity improvement of at least 0.25
dB is demonstrated by maximizing normalized generalized mutual information
(NGMI). For a long-haul nonlinear optical fiber transmission system, the
benefit of mitigating the nonlinearity is demonstrated and a reach increase of
6.7% (560 km) over time-domain hybrid four-dimensional two-amplitude
eight-phase shift keying (TDH-4D-2A8PSK) is observed