Spatial working memory (WM) and spatial attention are closely related, but the relationship between non-spatial WM and spatial attention still remains unclear. The present study aimed to investigate the interaction between color WM and smooth pursuit eye movements. A modified delayed-match-to-sample paradigm (DMS) was applied with 2 or 4 items presented in each visual field. Subjects memorized the colors of items in the cued visual field and smoothly moved eyes towards or away from memorized items during retention interval despite that the colored items were no longer visible. The WM performance decreased with higher load in general. More importantly, the WM performance was better when subjects pursued towards rather than away from the cued visual field. Meanwhile, the pursuit gain decreased with higher load and demonstrated a higher result when pursuing away from the cued visual field. These results indicated that spatial attention, guiding attention to the memorized items, benefits color WM. Therefore, we propose that a competition for attention resources exists between color WM and smooth pursuit eye movements