In this study, we argue that conventional methods of measuring agricultural productivity which only uses information about marketed inputs and outputs does not give a true representation of how sustainable the activities of the sector are. Motivated by the Solow-type growth accounting framework, we use the Tornqvist index formula to construct input, output and TFP indices for Nigerian agriculture between 1980 and 2010. We account for environmental externalities by incorporating off-farm damage costs of soil erosion based on different assumptions about possible scenarios of the extent and trajectory of damage costs. The results show that when externalities are not accounted for, productivity in the Nigerian agricultural sector is overestimated. This conclusion is robust to the different assumptions about damage cost scenarios made. The implication is that reducing off-farm erosion damages through improved soil conservation practices will significantly improve productivity and sustainability in the Nigerian agriculture sector