Conditioning factors of test-taking engagement in PIAAC: an exploratory IRT modelling approach considering person and item characteristics

Background: A potential problem of low-stakes large-scale assessments such as the Programme for the International Assessment of Adult Competencies (PIAAC) is low test-taking engagement. The present study pursued two goals in order to better understand conditioning factors of test-taking disengagement: First, a model-based approach was used to investigate whether item indicators of disengagement constitute a continuous latent person variable by domain. Second, the effects of person and item characteristics were jointly tested using explanatory item response models. Methods: Analyses were based on the Canadian sample of Round 1 of the PIAAC, with N = 26,683 participants completing test items in the domains of literacy, numeracy, and problem solving. Binary item disengagement indicators were created by means of item response time thresholds. Results: The results showed that disengagement indicators define a latent dimension by domain. Disengagement increased with lower educational attainment, lower cognitive skills, and when the test language was not the participant’s native language. Gender did not exert any effect on disengagement, while age had a positive effect for problem solving only. An item’s location in the second of two assessment modules was positively related to disengagement, as was item difficulty. The latter effect was negatively moderated by cognitive skill, suggesting that poor test-takers are especially likely to disengage with more difficult items. Conclusions: The negative effect of cognitive skill, the positive effect of item difficulty, and their negative interaction effect support the assumption that disengagement is the outcome of individual expectations about success (informed disengagement)

Avaliação em ambientes virtuais de aprendizagem Evaluation in virtual learning environments

A demanda por avaliações de projetos de aprendizagem virtual a distância tem requerido o emprego de conceitos e métodos que transcendem o campo exclusivo da Educação, destacando-se a multiplicidade de marcos teóricos e abordagens técnicas empregados nas estratégias avaliativas de aprendizagem online. A despeito da hegemonia dos métodos quantitativos na avaliação das tecnologias de informação e comunicação, o emprego de métodos qualitativos nas avaliações de ambientes virtuais tem crescido ao longo das duas últimas décadas. A combinação de métodos quali-quantitativos possibilita uma melhor compreensão dos fenômenos subjacentes ao uso das tecnologias para a aprendizagem online. Dado que a educação em ambientes virtuais refere-se a experiências de aprendizagem que utilizam recursos hipermidiáticos em ambientes apoiados por uma tecnologia de comunicação online, optou-se neste artigo enfocar alguns tópicos relativos à avaliação de tecnologias de informação e de aprendizagem nesses ambientes, aprofundando a discussão no que tange aos métodos relevantes à avaliação tanto dos ambientes virtuais de aprendizagem quanto da aprendizagem nesse meio. Essa opção se deve tanto ao reconhecimento das especificidades das práticas pedagógicas da EaD que colocam em evidência a relação entre educação e comunicação, viabilizadas por meio das tecnologias de informação e comunicação, quanto aos diversos papéis, negativos e positivos, atribuídos às tecnologias de comunicação e informação na Educação.<br>The demand for evaluations of projects of virtual distance learning has required the use of concepts and methods that go beyond the strictly educational field, with emphasis on the multiplicity of theoretical frameworks and technical approaches employed in the assessment strategies of online learning. Despite the hegemony of quantitative methods in the evaluation of the technologies of information and communication, the use of qualitative methods in the assessment of virtual environments has increased throughout the last decades. The combination of qualitative-quantitative methods allows a better understanding of the phenomena underlying the use of technologies for online learning. Given that education in virtual environments relates to learning experiences that make use of hypermediatic resources within environments supported by an online communication technology, we have opted in the present article for focusing on topics associated to the evaluation of information and learning technologies in these environments, furthering the discussion concerning the methods relevant to the assessment both of the virtual learning environments and of the learning within this medium. This choice is due to the recognition of the specificities of the practices of distance learning, which stress the relationship between education and communication, made possible through the new information and communication technologies. It also recognizes the various roles, positive and negative, attributed to the information and communication technologies in education