RACE: Large-scale ReAding Comprehension Dataset From Examinations

We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task. Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students' ability in understanding and reasoning. In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%). We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension. The dataset is freely available at http://www.cs.cmu.edu/~glai1/data/race/ and the code is available at https://github.com/qizhex/RACE_AR_baselines.Comment: EMNLP 201

Overview of CLEF QA Entrance Exams Task 2015

Abstract. This paper describes the Entrance Exams task at the CLEF QA Track 2015. Following the last two editions, the data set has been extracted from actual university entrance examinations including a variety of topics and question types. Systems receive a set of Multiple-Choice Reading Comprehension tests where the task is to select the correct answer among a finite set of candidates, according to the given text. Questions are designed originally for testing human examinees, rather than evaluating computer systems. Therefore, the data set challenges human ability to show their understanding of texts. Thus, questions and answers are lexically distant from their supporting excerpts in text, requiring not only a high degree of textual inference, but also the development of strategies for selecting the correct answer

Matriculation exam vs. entrance exam – First-year English students’ experiences on university admission

As of 2020, approximately more than half of the new university students are chosen via certificate-based admission with points gained from the matriculation examination grades while the rest of the student population is admitted with a programme and subject-specific entrance exam. The aim of this thesis is to evaluate how the Finnish higher education admission reform of 2020 affected the student selection process to the English study track of the Bachelor’s Programme in Languages offered in the University of Helsinki. This is done by studying how the English matriculation exam and the entrance exam into the English study track compare to one another, how the new English students themselves view the university admission process and the two exams, and finally, what types of skills are tested in the exams. Both qualitative and quantitative data are used in the study. The English matriculation exam and the English entrance exam of spring 2021 are analysed systematically with qualitative content analysis. First-year English students’ experiences surrounding the student selection and the two exams are explored with the help of a survey which collects both quantitative and qualitative information. The skills tested in the two exams are studied with close reading of the exams and with the student survey. Limitations such as the COVID-19 outbreak, and the student sample size are considered as well. The results of this study show that the English matriculation exam and the English entrance exam share similarities as they are both language exams testing skills in English, but the main differences emerge from purposes for which the exams were created and their difficulty level. The results also show that the first-year English students consider the English matriculation exam to be easier than the English entrance exam, but the entrance exam prepares students better for university level English studies compared to the matriculation exam. In addition, both the exams test for different English skills both explicitly and implicitly. Finally, it is also discussed how the English matriculation exam and the English entrance exam fit to the criteria of a functional entrance exam, how comparable the two exams truly are as entrance exams to the English study track, and how first-year student expectations of studying in the English study track can be met in the future

Entrenamiento Croslingüe para Búsqueda de Respuestas de Opción Múltiple

In this work we explore to what extent multilingual models can be trained for one language and applied to a different one for the task of Multiple Choice Question Answering. We employ the RACE dataset to fine-tune both a monolingual and a multilingual models and apply these models to another different collections in different languages. The results show that both monolingual and multilingual models can be zero-shot transferred to a different dataset in the same language maintaining its performance. Besides, the multilingual model still performs good when it is applied to a different target language. Additionally, we find that exams that are more difficult to humans are harder for machines too. Finally, we advance the state-of-the-art for the QA4MRE Entrance Exams dataset in several languages.En este trabajo exploramos en qué medida los modelos multilingües pueden ser entrenados para un solo idioma y aplicados a otro diferente para la tarea de respuesta a preguntas de opción múltiple. Empleamos el conjunto de datos RACE para ajustar tanto un modelo monolingüe como multilingüe y aplicamos estos modelos a otras colecciones en idiomas diferentes. Los resultados muestran que tanto los modelos monolingües como los multilingües pueden transferirse a un conjunto de datos diferente en el mismo idioma manteniendo su rendimiento. Además, el modelo multilingüe todavía funciona bien cuando se aplica a un idioma de destino diferente. Asimismo, hemos comprobado que los exámenes que son más difíciles para los humanos también son más difíciles para las máquinas. Finalmente, avanzamos el estado del arte para el conjunto de datos QA4MRE Entrance Exams en varios idiomas.This work has been funded by the Spanish Research Agency under CHIST-ERA LIHLITH project (PCIN-2017-085/AEI) and deepReading (RTI2018-096846-B-C21 /MCIU/AEI/FEDER,UE)

University entrance language tests : a matter of justice

University entrance language tests are often administered under the assumption that even if language proficiency does not determine academic success, a certain proficiency level is still required. Nevertheless, little research has focused on how well L2 students cope with the linguistic demands of their studies in the first months after passing an entrance test. Even fewer studies have taken a longitudinal perspective. Set in Flanders, Belgium, this study examines the opinions and experiences of 24 university staff members and 31 international L2 students, of whom 20 were tracked longitudinally. Attention is also given to test/retest results, academic score sheets, and class recordings. To investigate the validity of inferences made on the basis of L2 students' scores, Kane's (2013) Interpretation/Use Argument approach is adopted, and principles from political philosophy are applied to investigate whether a policy that discriminates among students based on language test results can be considered just. It is concluded that the receptive language requirements of university studies exceed the expected B2 level and that the Flemish entrance tests include language tasks that are of little importance for first-year students. Furthermore, some of the students who failed the entrance test actually managed quite well in their studies - a result that entails broad implications concerning validation and justice even outside the study's localized setting

A Study on the Ideal Amount of Extensive Reading for High Schools in Japan

Abstract     When introducing extensive reading, I always had a difficult time in getting everyone to be contented. Starting with the Oxford Bookworms, I prepared over 500 books in the school library in both fiction and non-fiction books. Vocabulary level tests were also conducted to make sure that the books were at the I+1 level, and the author talked to each student to see which genre they may like. While this is a difficult task, I tried to find the ideal amount of weekly reading for science course students in a senior high school in Japan. A quantitative questionnaire was created using the price sensitivity meter method to measure the learners’ ideal weekly reading amount. Results showed that the ideal amount of weekly extensive reading was 1200 words per week at the 500-word level. When asked directly about the amount of ideal reading according to their level, 44% of the students said that reading 2000 words per week was “little or no problem”