The aim of the paper is to provide the mathematical foundation of effective numerical algorithms for the optimal design of periodic binary gratings. Special attention is paid to fast and reliable methods for the computation of diffraction efficiencies and of the gradients of certain functionals with respect to the parameters of the non-smooth grating profile. The methods are based on a generalized finite element discretization of strongly elliptic variational formulations of quasi periodic transmission problems for the Helmholtz equation in a bounded domain coupled with boundary integral representations in the exterior. We prove uniqueness and existence results for quite general situations and analyse the convergence of the numerical solutions. Furthermore, explicit formulas for the partial derivatives of the reflection and transmission coefficients with respect to the parameters of a binary grating profile are derived. Finally, we briefly discuss the implementation of a gradient type algorithm for solving optimal design problems and present some numerical results
