Introducing a framework to assess newly created questions with Natural Language Processing

Statistical models such as those derived from Item Response Theory (IRT) enable the assessment of students on a specific subject, which can be useful for several purposes (e.g., learning path customization, drop-out prediction). However, the questions have to be assessed as well and, although it is possible to estimate with IRT the characteristics of questions that have already been answered by several students, this technique cannot be used on newly generated questions. In this paper, we propose a framework to train and evaluate models for estimating the difficulty and discrimination of newly created Multiple Choice Questions by extracting meaningful features from the text of the question and of the possible choices. We implement one model using this framework and test it on a real-world dataset provided by CloudAcademy, showing that it outperforms previously proposed models, reducing by 6.7% the RMSE for difficulty estimation and by 10.8% the RMSE for discrimination estimation. We also present the results of an ablation study performed to support our features choice and to show the effects of different characteristics of the questions' text on difficulty and discrimination.Comment: Accepted at the International Conference of Artificial Intelligence in Educatio

Using item response theory to explore the psychometric properties of extended matching questions examination in undergraduate medical education

BACKGROUND: As assessment has been shown to direct learning, it is critical that the examinations developed to test clinical competence in medical undergraduates are valid and reliable. The use of extended matching questions (EMQ) has been advocated to overcome some of the criticisms of using multiple-choice questions to test factual and applied knowledge. METHODS: We analysed the results from the Extended Matching Questions Examination taken by 4th year undergraduate medical students in the academic year 2001 to 2002. Rasch analysis was used to examine whether the set of questions used in the examination mapped on to a unidimensional scale, the degree of difficulty of questions within and between the various medical and surgical specialties and the pattern of responses within individual questions to assess the impact of the distractor options. RESULTS: Analysis of a subset of items and of the full examination demonstrated internal construct validity and the absence of bias on the majority of questions. Three main patterns of response selection were identified. CONCLUSION: Modern psychometric methods based upon the work of Rasch provide a useful approach to the calibration and analysis of EMQ undergraduate medical assessments. The approach allows for a formal test of the unidimensionality of the questions and thus the validity of the summed score. Given the metric calibration which follows fit to the model, it also allows for the establishment of items banks to facilitate continuity and equity in exam standards

Exploring differential item functioning in the Western Ontario and McMaster Universities osteoarthritis index (WOMAC)

Background: The Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC) is a widely used patient reported outcome in osteoarthritis. An important, but frequently overlooked, aspect of validating health outcome measures is to establish if items exhibit differential item functioning (DIF). That is, if respondents have the same underlying level of an attribute, does the item give the same score in different subgroups or is it biased towards one subgroup or another. The aim of the study was to explore DIF in the Likert format WOMAC for the first time in a UK osteoarthritis population with respect to demographic, social, clinical and psychological factors. Methods: The sample comprised a community sample of 763 people with osteoarthritis who participated in the Somerset and Avon Survey of Health. The WOMAC was explored for DIF by gender, age, social deprivation, social class, employment status, distress, body mass index and clinical factors. Ordinal regression models were used to identify DIF items. Results: After adjusting for age, two items were identified for the physical functioning subscale as having DIF with age identified as the DIF factor for 2 items, gender for 1 item and body mass index for 1 item. For the WOMAC pain subscale, for people with hip osteoarthritis one item was identified with age-related DIF. The impact of the DIF items rarely had a significant effect on the conclusions of group comparisons. Conclusions: Overall, the WOMAC performed well with only a small number of DIF items identified. However, as DIF items were identified in for the WOMAC physical functioning subscale it would be advisable to analyse data taking into account the possible impact of the DIF items when weight, gender or especially age effects, are the focus of interest in UK-based osteoarthritis studies. Similarly for the WOMAC pain subscale in people with hip osteoarthritis it would be worthwhile to analyse data taking into account the possible impact of the DIF item when age comparisons are of primary interest

A latent trait look at pretest-posttest validation of criterion-referenced test items

Since Cox and Vargas (1966) introduced their pretest-posttest validity index for criterion-referenced test items, a great number of additions and modifications have followed. All are based on the idea of gain scoring; that is, they are computed from the differences between proportions of pretest and posttest item responses. Although the method is simple and generally considered as the prototype of criterion-referenced item analysis, it has many and serious disadvantages. Some of these go back to the fact that it leads to indices based on a dual test administration- and population-dependent item p values. Others have to do with the global information about the discriminating power that these indices provide, the implicit weighting they suppose, and the meaningless maximization of posttest scores they lead to. Analyzing the pretest-posttest method from a latent trait point of view, it is proposed to replace indices like Cox and Vargas’ Dpp by an evaluation of the item information function for the mastery score. An empirical study was conducted to compare the differences in item selection between both methods

Assessing early memories of threat and subordination: Confirmatory factor analyisis of the early life experiences scale for adolescents.

The Early Life Experiences Scale (ELES) is a self-report questionnaire that assesses personal feelings of perceived threat and submissiveness in interactions within family. This paper presents the adaptation and validation of the ELES in Portuguese language for adolescents. The sample was composed of 771 adolescents from community schools with ages between 13 and 18 years old. Along with ELES, participants also answered the Early Memories of Warmth and Safeness Scale and the Positive and Negative Affect Schedule for Children and Adolescents. Confirmatory factor analysis (CFA) was performed to test the factor structure of the ELES and results confirm a three-factor structure, composed by Threat, Submissiveness and Unvalued dimensions. These emotional memories focused on perceived threat, submissiveness and unvalued seem to have a distinct nature. The scale also showed adequate internal consistency, good test-retest reliability and convergent validity with measures of positive emotional memories, positive and negative affect. There were sex differences for threat subscale and age differences for submissiveness subscale. Overall, these findings suggest that the ELES in its Portuguese version for adolescents may be a useful tool for research, educational and clinical contexts with school-aged adolescents

A meeting report: OECD-GESIS Seminar on Translating and Adapting Instruments in Large-Scale Assessments (2018)

This report summarizes the main themes and conclusions from the OECD-GESIS Seminar on Translating and Adapting Instruments in Large-Scale Assessments, which took place at the Organization for Economic Co-operation and Development (OECD), Paris, in June 2018. The five sessions covered the topics (1) etic (universal) vs. emic (culture-specific) measurement instruments, (2) language- and culture-sensitive development of measurement instruments, (3) international guidelines vs. implementation in countries and by translators, (4) tools and technological developments, and (5) quality control of translations. Key players in the field presented on best practice, lessons learned, and innovations and also made suggestions for moving the field forward

Perspectives on utilization of edible coatings and nano-laminate coatings for extension of postharvest storage of fruits and vegetables

It is known that in developing countries, a large quantity of fruit and vegetable losses results at postharvest and processing stages due to poor or scarce storage technology and mishandling during harvest. The use of new and innovative technologies for reducing postharvest losses is a requirement that has not been fully covered. The use of edible coatings (mainly based on biopolymers) as a postharvest technique for agricultural commodities has offered biodegradable alternatives in order to solve problems (e.g., microbiological growth) during produce storage. However, biopolymer-based coatings can present some disadvantages such as: poor mechanical properties (e.g., lipids) or poor water vapor barrier properties (e.g., polysaccharides), thus requiring the development of new alternatives to solve these drawbacks. Recently, nanotechnology has emerged as a promising tool in the food processing industry, providing new insights about postharvest technologies on produce storage. Nanotechnological approaches can contribute through the design of functional packing materials with lower amounts of bioactive ingredients, better gas and mechanical properties and with reduced impact on the sensorial qualities of the fruits and vegetables. This work reviews some of the main factors involved in postharvest losses and new technologies for extension of postharvest storage of fruits and vegetables, focused on perspective uses of edible coatings and nano-laminate coatings.María L. Flores-López thanks Mexican Science and Technology Council (CONACYT, Mexico) for PhD fellowship support (CONACYT Grant Number: 215499/310847). Miguel A. Cerqueira (SFRH/BPD/72753/2010) is recipient of a fellowship from the Fundação para a Ciência e Tecnologia (FCT, POPH-QREN and FSE Portugal). The authors also thank the FCT Strategic Project of UID/ BIO/04469/2013 unit, the project RECI/BBB-EBI/0179/2012 (FCOMP-01-0124-FEDER-027462) and the project ‘‘BioInd Biotechnology and Bioengineering for improved Industrial and AgroFood processes,’’ REF. NORTE-07-0124-FEDER-000028 Co-funded by the Programa Operacional Regional do Norte (ON.2 – O Novo Norte), QREN, FEDER. Fundação Cearense de Apoio ao Desenvolvimento Científico e Tecnológico – FUNCAP, CE Brazil (CI10080-00055.01.00/13)