INTRODUCTION: Microstructural alterations as assessed by diffusion tensor imaging (DTI) are key findings in both Alzheimer's disease (AD) and small vessel disease (SVD). We determined the contribution of each of these conditions to diffusion alterations. METHODS: We studied six samples (N = 365 participants) covering the spectrum of AD and SVD, including genetically defined samples. We calculated diffusion measures from DTI and free water imaging. Simple linear, multivariable random forest, and voxel-based regressions were used to evaluate associations between AD biomarkers (amyloid beta, tau), SVD imaging markers, and diffusion measures. RESULTS: SVD markers were strongly associated with diffusion measures and showed a higher contribution than AD biomarkers in multivariable analysis across all memory clinic samples. Voxel-wise analyses between tau and diffusion measures were not significant. DISCUSSION: In memory clinic patients, the effect of SVD on diffusion alterations largely exceeds the effect of AD, supporting the value of diffusion measures as markers of SVD