We derive Virasoro constraints for the zero momentum part of the QCD-like partition functions in the sector of topological charge ν. The constraints depend on the topological charge only through the combination Nf+βν/2 where the value of the Dyson index β is determined by the reality type of the fermions. This duality between flavor and topology is inherited by the small-mass expansion of the partition function and {\em all} spectral sum-rules of inverse powers of the eigenvalues of the Dirac operator. For the special case β=2 but arbitrary topological charge the Virasoro constraints are solved uniquely by a Generalized Kontsevich Model with potential V(X)=1/X