In this paper, we investigate the hybrid beamforming design with sub-connected structure for millimeter-wave (mmWave) massive multi-user multiple-input multiple-output (MU-MIMO) amplify-and-forward (AF) relay systems. Considering the constant-modulus and block-diagonal constraints, an iterative algorithm is proposed to sequentially design the analog beamforming of the base station, relay, and users. Next, the relay baseband combiner is designed by transforming the highly complicated non-convex mutual information maximization problem into an easily tractable weighted minimum mean squared error one. To mitigate the inter-user interference, a successive serial interference cancellation based and a piecewise successive approximation based methods are developed for the uniform sparse distribution and dense distribution scenarios, respectively. Simulation results demonstrate that the two proposed hybrid beamforming schemes can achieve good performance in terms of sum-rate and energy efficiency, and outperform benchmarks significantly. In addition, the two proposed schemes are shown to be robust to imperfect channel state information (CSI), even though they are studied based on perfect CSI
