We use N-body/photometric galaxy simulations to examine the impact of sample variance of spectroscopic redshift samples on the accuracy of photometric redshift (photo-z) determination and calibration of photo-z errors. We estimate the biases in the cosmological parameter constraints from weak lensing and derive requirements on the spectroscopic follow-up for three different photo-z algorithms chosen to broadly span the range of algorithms available. We find that sample variance is much more relevant for the photo-z error calibration than for photo-z training, implying that follow-up requirements are similar for different algorithms. We demonstrate that the spectroscopic sample can be used for training of photo-zs and error calibration without incurring additional bias in the cosmological parameters. We provide a guide for observing proposals for the spectroscopic follow-up to ensure that redshift calibration biases do not dominate the cosmological parameter error budget. For example, assuming optimistically (pessimistically) that the weak lensing shear measurements from the Dark Energy Survey could obtain 1σ constraints on the dark energy equation of state w of 0.035 (0.055), implies a follow-up requirement of 150 (40) patches of sky with a telescope such as Magellan, assuming a 1/8 deg2 effective field of view and 400 galaxies per patch. Assuming (optimistically) a VIMOS-VLT Deep Survey-like spectroscopic completeness with purely random failures, this could be accomplished with about 75 (20) nights of observation. For more realistic assumptions regarding spectroscopic completeness, or with the presence of other sources of systematics not considered here, further degradations to dark energy constraints are possible. We test several approaches for making the requirements less stringent. For example, if the redshift distribution of the overall sample can be estimated by some other technique, e.g. cross-correlation, then follow-up requirements could be reduced by an order of magnitud
