Introduction Surgical trainees at all stages are mandated to use workplace-based assessments (WBAs) to gain feedback from their trainers. Direct observation of procedural skills (DOPS) is a tool to assess hands-on surgical skills. This review of the literature seeks to ascertain how valid DOPS are as an assessment tool for the procedural skills of surgical trainees according to the American Psychology Association (APA) validity framework. Methods Relevant literature was identified through a structured search of Medline, CINAHL and Web of Science databases, with further papers included on citation review. Following this, papers meeting the inclusion and exclusion criteria were included in the final review specifying those investigating WBAs including DOPS, assessments in practice rather than simulation and specifically for postgraduate surgical trainees. Findings Sixteen papers were included in the final analysis. Extracted data from the returned papers were assessed for evidence of validity in each of the five domains on the APA framework: validity based on consequence, response process, reliability, content and relationship to other variables. There are studies that show good parameters for reliability and validity for specific DOPS used in endoscopy and otolaryngology. However, there is confusion over the purpose of DOPS among trainers and trainees as to whether they should be used formatively or summatively. Recent changes to the surgical curriculum have sought to address this, and further work into the impact of this needs to be done. </jats:sec
