A counterfactual impact evaluation of a bilingual program on students’ grade point average at a spanish university

Abstract

This observational study intends to estimate the causal effects of an English as a Medium of Instruction (EMI) program (as predictor) on students Grade Point Average (GPA) (as outcome) at a particular University in Spain by using a Counterfactual Impact Evaluation (CIE). The need to address the crucial question of causal inferences in EMI programs to produce credible evidences of successful interventions contrasts, however, with the absence of experimental or quasi-experimental research and evaluation designs in the field. CIE approach is emerging as a methodologically viable solution to bridge that gap. The program evaluated here consisted in delivering an EMI program in a Primary Education Teacher Training Degree group. After achieving balance on the observed covariates and recreating a situation that would have been expected in a randomized experiment, three matching approaches such as genetic matching, nearest neighbor matching and Coarsened Exact Matching were used to analyze observational data from a total of 1288 undergraduate students, including both treatment and control group. Results show unfavorable effects of the bilingual group treatment condition. Potential interpretations and recommendations are provided in order to strengthen future causal evidences of bilingual education programs’ effectiveness in Higher Education.This work was supported by the Junta de Andalucía-funded Proyecto de Excelencia: “Análisis y Garantía de Calidad de la Educación Superior Plurilingüe en la Educación Superior de Andalucía [Junta de Andalusia-funded Project of Excelence: Analysis and Warrantee of the Quality of Plurilingual Higher Education in Andalucia] (AGCEPESA; Grant Agreement No. P12-SEJ − 1588)