Scalp-recorded electrophysiological Envelope Following Responses (EFRs) have recently gained popularity as an objective measure of supra-threshold hearing deficits (e.g., as caused by cochlear synaptopathy). To explore the frequency specificity of EFRs in diagnosing deficits, we recorded EFRs to sinusoidally amplitude-modulated (AM) white noise carriers with different bandwidths in two participant groups: young normal-hearing control group and a group self-reported hearing difficulties in noisy listening environments. Despite individual variability in EFR strength, the group-means revealed minor response differences in stimuli with different low-frequency content. The limited contribution of auditory frequency channels below 2 kHz to the EFR was confirmed through EFR simulations from a biophysically-inspired model of the human auditory periphery. Further investigation showed that the minor low-frequency contribution stems from the lack of AM coding in individual low-frequency channels, although the overall spike-rate increases for stimulation with wider carrier bands. In conclusion, despite the broadband cochlear excitation, the broadband EFR mostly reflects summed AM coding strength in frequency channels above 2 kHz. To increase the frequency sensitivity of the EFRs it is necessary to carefully investigate the bandwidth and modulation frequency of the stimulus