Premature-born adults exhibit lasting white matter alterations as demonstrated by widespread reduction in fractional anisotropy (FA) based on diffusion-weighted imaging (DWI). FA reduction, however, is non-specific for microscopic underpinnings such as aberrant myelination or fiber density (FD). Using recent advances in DWI, we tested the hypothesis of reduced FD in premature-born adults and investigated its link with the degree of prematurity and cognition. 73 premature- and 89 mature-born adults aged 25–27 years underwent single-shell DWI, from which a FD measure was derived using convex optimization modeling for microstructure informed tractography (COMMIT). Premature-born adults exhibited lower FD in numerous tracts including the corpus callosum and corona radiata compared to mature-born adults. These FD alterations were associated with both the degree of prematurity, as assessed via gestational age and birth weight, as well as with reduced cognition as measured by full-scale IQ. Finally, lower FD overlapped with lower FA, suggesting lower FD underlie unspecific FA reductions. Results provide evidence that premature birth leads to lower FD in adulthood which links with lower full-scale IQ. Data suggest that lower FD partly underpins FA reductions of premature birth but that other processes such as hypomyelination might also take place