Abrasive wear can have a detrimental effect on machinery, especially in the mining and construction industries. To prolong machinery lifetime and cut down energy consumption, a thorough understanding of abrasive wear is essential: surface topography measurement and interpretation (including form, waviness, and roughness) are vitally important. However, the potentially crucial influence of surface topography intricacies on tribological behavior has been obscured since roughness and waviness are considered simple scalar quantities in most cases (e.g., roughness Ra and waviness Wt). In this work, the complete waviness profile of the sliding track was used to shed light on the influence of surface topography on abrasive wear. Bearing steel (100Cr6, AISI 52100) pins and disks were tribologically tested in a flat-on-flat contact with Al2O3-based slurries as interfacial medium. Using slurries with two different particle sizes, 5 and 13 μm, we found that friction fluctuates only with small abrasive particles (5-µm slurry) and relatively low waviness disks. It was found that even small surface deviations (albeit minimized and controlled for) can significantly increase the friction coefficient—up to 91%. Remarkably, not only are frictional fluctuations strongly correlated with the disk’s initial waviness profile, but these small fluctuations correlate with unevenly distributed high wear. These findings enhance our understanding of the friction wear structure and provide the basis for exploring how surfaces can be optimized for better tribological performance