Introducing a framework to assess newly created questions with Natural Language Processing

Statistical models such as those derived from Item Response Theory (IRT) enable the assessment of students on a specific subject, which can be useful for several purposes (e.g., learning path customization, drop-out prediction). However, the questions have to be assessed as well and, although it is possible to estimate with IRT the characteristics of questions that have already been answered by several students, this technique cannot be used on newly generated questions. In this paper, we propose a framework to train and evaluate models for estimating the difficulty and discrimination of newly created Multiple Choice Questions by extracting meaningful features from the text of the question and of the possible choices. We implement one model using this framework and test it on a real-world dataset provided by CloudAcademy, showing that it outperforms previously proposed models, reducing by 6.7% the RMSE for difficulty estimation and by 10.8% the RMSE for discrimination estimation. We also present the results of an ablation study performed to support our features choice and to show the effects of different characteristics of the questions' text on difficulty and discrimination.Comment: Accepted at the International Conference of Artificial Intelligence in Educatio

The differential relations between verbal, numerical and spatial working memory abilities and children’s reading comprehension

Working memory predicts children's reading comprehension but it is not clear whether this relation is due to a modality-specific or general working memory. This study, which investigated the relations between children's reading skills and working memory (WM) abilities in 3 modalities, extends previous work by including measures of both reading comprehension and reading accuracy. Tests of word reading accuracy and reading comprehension, and working memory tests in three different modalities (verbal, numerical and spatial), were given to 197 6- to 11-year old children. The results support the view that working memory tasks that require the processing and recall of symbolic information (words and numbers) are better predictors of reading comprehension than tasks that require visuo-spatial storage and processing. The different measures of verbal and numerical working memory were not equally good predictors of reading comprehension, but their predictive power depended on neither the word vs. numerical contrast nor the complexity of the processing component. In general, performance on the verbal and numerical working memory tasks predicted reading comprehension, but not reading accuracy, and spatial WM did not predict either. The patterns of relations between the measures of working memory and reading comprehension ability were relatively constant across the age group tested

Associated reading skills in children with a history of Specific Language Impairment (SLI)

A large cohort of 200 eleven-year-old children with Specific Language Impairment (SLI) were assessed on basic reading accuracy and on reading comprehension as well as language tasks. Reading skills were examined descriptively and in relation to early language and literacy factors. Using stepwise regression analyses in which age and nonverbal IQ were controlled for, it was found that a single word reading measure taken at 7 years was unsurprisingly a strong predictor of the two different types of reading ability. However, even with this measure included, a receptive syntax task (TROG) entered when reading accuracy score was the DV. Furthermore, a test of expressive syntax/narrative and a receptive syntax task completed at 7 years entered into the model for word reading accuracy. When early reading accuracy was excluded from the analyses, early phonological skills also entered as a predictor of both reading accuracy and comprehension at 11 years. The group of children with a history of SLI were then divided into those with no literacy difficulties at 11 and those with some persisting literacy impairment. Using stepwise logistic regression, and again controlling for IQ and age, 7 years receptive syntax score (but not tests of phonology, expressive vocabulary or expressive syntax/narrative) entered as a positive predictor of membership of the ‘no literacy problems’ group regardless of whether early reading accuracy was controlled for in step one. The findings are discussed in relation to the overlap of SLI and dyslexia and the long term sequelae of language impairment

The Effect of Text Authenticity on the Performance of Iranian EFL Students in a C-Test

As part of growing efforts to understand factors affecting c-test this study aims to investigate the effect of text authenticity on the performance of Iranian EFL students in a C-Test. The C-Test is an integrative testing instrument that measures overall language competence, very much like the cloze test. In this study the rule of two has been applied: "the second half of every second word has been deleted, beginning with the second word of the second sentence; the first and last sentences are left intact" (Katona and Dornyei 1993: 35). The research involves 60 college students in their third year, majoring in English Literature at Ershad-Damavand College. This group were randomly selected applying multi-stage sampling. Since the present study intended to investigate the role of two different formats, i.e. authentic and inauthentic texts (text translated from Persian into English), two different tailored C-Tests were made to measure and compare the performances of the participants. Two C-Tests, one with Authentic Text and the other, with Inauthentic Text were administered to this homogenized group comprising 30 subjects. The findings of this study suggest that authenticity has an effect on the performance of learners in c-tests and we should control this variable while devising a c-test

Predictors of developmental dyslexia in European orthographies with varying complexity

Background: The relationship between phoneme awareness, rapid automatized naming (RAN), verbal short-term/working memory (ST/WM) and diagnostic category is investigated in control and dyslexic children, and the extent to which this depends on orthographic complexity. Methods: General cognitive, phonological and literacy skills were tested in 1138 control and 1114 dyslexic children speaking 6 different languages spanning a large range of orthographic complexity (Finnish, Hungarian, German, Dutch, French, English). Results: Phoneme deletion and RAN were strong concurrent predictors of developmental dyslexia, while verbal ST/WM and general verbal abilities played a comparatively minor role. In logistic regression models, more participants were classified correctly when orthography was more complex. The impact of phoneme deletion and RAN-digits was stronger in complex than in less complex orthographies. Conclusions: Findings are largely consistent with the literature on predictors of dyslexia and literacy skills, while uniquely demonstrating how orthographic complexity exacerbates some symptoms of dyslexia

Reading comprehension: nature, assessment and teaching.

The goal of reading is understanding. In order to understand print, a child must be able to decode the words on the page and to extract meaning. A large body of research focuses on how children learn to decode text and how best to foster children’s decoding skills. In contrast, we know much less about the process of reading comprehension in children. In this booklet we first consider what is required in order to ‘read for meaning’. We then move on to discuss children who have difficulties with reading comprehension. Our aim is to enable teachers to assess individual differences in reading and to foster the comprehension strategies that characterize fluent reading

A summary of research relating to first grade reading, 1932-1963

Thesis (Ed.M.)--Boston UniversityProblem: An attempt to compare the rate of learning of colorful and abstract words of children in grade one. Materials: 1. Thirty words were selected to be used in three groups of ten words each. Each list consisted of five colorful and five abstract words. Colorful words are those which have much meaning for children and abstract words are those with little meaning. The words were printed on 3 x 11 inch flash cards in lower case letters. 2. Check sheets were made with spaces for checking the retention of words at three different times and pictures were made to enrich the colorful words. 3. Directions were prepared and given to each teacher who participated in the study. 4. Kuhlmann-Anderson Intelligence Test: Battery A. 5. Detroit Word Recognition Test: Form A [TRUNCATED