This paper presents back-to-back and transmission performance comparisons between single-polarization 120 Gbit/s RZ-DQP-ASK and RZ-D8PSK formats in a single-wavelength system. In the back-to-back configuration, we determine the OSNR requirements of the two formats and compare the results with those obtained analytically and numerically. In the transmission experiments, we examine signal constellations to quantify the significance of intra-channel fiber nonlinearities and investigate the BER performance of the two formats over different reaches (up to 480 km). The results show that the performance of DQP-ASK is superior to D8PSK in back-to-back configuration but is inferior over long distances
