We study a new generalized version of the square-lattice frustrated XY model
where unequal ferromagnetic and antiferromagnetic couplings are arranged in a
zig-zag pattern. The ratio between the couplings ρ can be used to tune the
system, continuously, from the isotropic square-lattice to the
triangular-lattice frustrated XY model. The model can be physically realized as
a Josephson-junction array with two different couplings, in a magnetic field
corresponding to half-flux quanta per plaquette. Mean-field approximation,
Ginzburg-Landau expansion and finite-size scaling of Monte Carlo simulations
are used to study the phase diagram and critical behavior. Depending on the
value of ρ, two separate transitions or a transition line in the
universality class of the XY-Ising model, with combined Z2 and U(1)
symmetries, takes place. In particular, the phase transitions of the standard
square-lattice and triangular-lattice frustrated XY models correspond to two
different cuts through the same transition line. Estimates of the chiral
(Z2) critical exponents on this transition line deviate significantly from
the pure Ising values, consistent with that along the critical line of the
XY-Ising model. This suggests that a frustrated XY model or Josephson-junction
array with a zig-zag coupling modulation can provide a physical realization of
the XY-Ising model critical line.Comment: 11 pages, 9 figures, RevTex, to appear in Phys. Rev.