Previous neuroimaging research indicates that white matter injury and integrity, measured respectively by white matter hyperintensities (WMH) and fractional anisotropy (FA) obtained from diffusion tensor imaging (DTI), differ with aging and cerebrovascular disease (CVD) and are associated with episodic memory deficits in cognitively normal older adults. However, knowledge about tract-specific relationships between WMH, FA, and episodic memory in aging remains limited. We hypothesized that white matter connections between frontal cortex and subcortical structures as well as connections between frontal and temporo-parietal cortex would be most affected. In the current study, we examined relationships between WMH, FA and episodic memory in 15 young adults, 13 elders with minimal WMH and 15 elders with extensive WMH, using an episodic recognition memory test for object-color associations. Voxel-based statistics were used to identify voxel clusters where white matter measures were specifically associated with variations in episodic memory performance, and white matter tracts intersecting these clusters were analyzed to examine white matter-memory relationships. White matter injury and integrity measures were significantly associated with episodic memory in extensive regions of white matter, located predominantly in frontal, parietal, and subcortical regions. Template based tractography indicated that white matter injury, as measured by WMH, in the uncinate and inferior longitudinal fasciculi were significantly negatively associated with episodic memory performance. Other tracts such as thalamo-frontal projections, superior longitudinal fasciculus, and dorsal cingulum bundle demonstrated strong negative associations as well. The results suggest that white matter injury to multiple pathways, including connections of frontal and temporal cortex and frontal-subcortical white matter tracts, plays a critical role in memory differences seen in older individuals