Background: Nutrition and physical activity interventions have been effective in creating environmental changes in afterschool programs. However, accurate assessment can be time-consuming and expensive as initiatives are scaled up for optimal population impact. This study aims to determine the criterion validity of a simple, low-cost, practitioner-administered observational measure of afterschool physical activity, nutrition, and screen time practices and child behaviors. Methods: Directors from 35 programs in three cities completed the Out-of-School Nutrition and Physical Activity Observational Practice Assessment Tool (OSNAP-OPAT) on five days. Trained observers recorded snacks served and obtained accelerometer data each day during the same week. Observations of physical activity participation and snack consumption were conducted on two days. Correlations were calculated to validate weekly average estimates from OSNAP-OPAT compared to criterion measures. Weekly criterion averages are based on 175 meals served, snack consumption of 528 children, and physical activity levels of 356 children. Results: OSNAP-OPAT validly assessed serving water (r = 0.73), fruits and vegetables (r = 0.84), juice >4oz (r = 0.56), and grains (r = 0.60) at snack; sugary drinks (r = 0.70) and foods (r = 0.68) from outside the program; and children’s water consumption (r = 0.56) (all p <0.05). Reports of physical activity time offered were correlated with accelerometer estimates (minutes of moderate and vigorous physical activity r = 0.59, p = 0.02; vigorous physical activity r = 0.63, p = 0.01). The reported proportion of children participating in moderate and vigorous physical activity was correlated with observations (r = 0.48, p = 0.03), as were reports of computer (r = 0.85) and TV/movie (r = 0.68) time compared to direct observations (both p < 0.01). Conclusions: OSNAP-OPAT can assist researchers and practitioners in validly assessing nutrition and physical activity environments and behaviors in afterschool settings. Trial registration Phase 1 of this measure validation was conducted during a study registered at clinicaltrials.gov NCT01396473. Electronic supplementary material The online version of this article (doi:10.1186/s12966-014-0145-5) contains supplementary material, which is available to authorized users
