VAST: a practical validation framework for e-assessment solutions

The influx of technology in education has made it increasingly difficult to assess the validity of educational assessments. The field of information systems often ignores the social dimension during validation, whereas educational research neglects the technical dimensions of designed instruments. The inseparability of social and technical elements forms the bedrock of socio-technical systems. Therefore, the current lack of validation approaches that address both dimensions is a significant gap. We address this gap by introducing VAST: a validation framework for e-assessment solutions. Examples of such solutions are technology-enhanced learning systems and e-health applications. Using multi-grounded action research as our methodology, we investigate how we can synthesise existing knowledge from information systems and educational measurement to construct our validation framework. We develop an extensive user guideline complementing our framework and find through expert interviews that VAST facilitates a comprehensive, practical approach to validating e-assessment solutions

Embracing Trustworthiness and Authenticity in the Validation of Learning Analytics Systems

Learning analytics sits in the middle space between learning theory and data analytics. The inherent diversity of learning analytics manifests itself in an epistemology that strikes a balance between positivism and interpretivism, and knowledge that is sourced from theory and practice. In this paper, we argue that validation approaches for learning analytics systems should be cognisant of these diverse foundations. Through a systematic review of learning analytics validation research, we find that there is currently an over-reliance on positivistic validity criteria. Researchers tend to ignore interpretivistic criteria such as trustworthiness and authenticity. In the 38 papers we analysed, researchers covered positivistic validity criteria 221 times, whereas interpretivistic criteria were mentioned 37 times. We motivate that learning analytics can only move forward with holistic validation strategies that incorporate “thick descriptions” of educational experiences. We conclude by outlining a planned validation study using argument-based validation, which we believe will yield meaningful insights by considering a diverse spectrum of validity criteria

Aligning the Goals of Learning Analytics with its Research Scholarship: An Open Peer Commentary Approach

To promote cross-community dialogue on matters of significance within the field of learning analytics], we as editors-in- chief of the Journal of Learning Analytics have introduced a section for papers that are open to peer commentary. The first of these papers, “A LAK of Direction: Misalignment Between the Goals of Learning Analytics and its Research Scholarship” by Motz et al. (2023), appeared in the journal’s early access section in March 2023, a few days before the start of the 13th International Learning Analytics and Knowledge Conference (LAK ’23). “A LAK of Direction” takes as its starting point the definition of learning analytics used in the call for papers of the first LAK conference (LAK ’11) and used since then by the Society for Learning Analytics Research (SoLAR): “Learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (Long & Siemens, 2011, p. 24). Following the conference, an invitation to submit proposals for commentaries on the paper was released, and 12 of these proposals were accepted. This paper brings those commentaries togethe

Aligning the Goals of Learning Analytics with its Research Scholarship: An Open Peer Commentary Approach

To promote cross-community dialogue on matters of significance within the field of learning analytics (LA), we as editors-in-chief of the Journal of Learning Analytics (JLA) have introduced a section for papers that are open to peer commentary. An invitation to submit proposals for commentaries on the paper was released, and 12 of these proposals were accepted. The 26 authors of the accepted commentaries are based in Europe, North America, and Australia. They range in experience from PhD students and early-career researchers to some of the longest-standing, most senior members of the learning analytics community. This paper brings those commentaries together, and we recommend reading it as a companion piece to the original paper by Motz et al. (2023), which also appears in this issu

VAST: A Practical Validation Framework

Project relating to the VAST validation framework, including an extensive guideline to assist users of the framework

The Effect of Countermeasure Readability on Security Intentions

Human failure is a primary contributor to successful cyber attacks. For any cybersecurity initiative, it is therefore vital to motivate individuals to implement secure behavior. Research using protection motivation theory (PMT) has given insights into what motivates people to safeguard themselves in cyberspace. Recent PMT results have highlighted the central role of the coping appraisal in the cybersecurity context. In cybersecurity, we cope with threats using countermeasures. Research has shown that countermeasure awareness is a significant antecedent to all coping appraisal elements. Yet, although awareness plays a key role within the PMT framework, it is generally challenging to influence. A factor that is easy to influence is countermeasure readability. Earlier work has shown the impact of readability on understanding and that readability metrics make measuring and improving readability simple. Therefore, our research aims to clarify the relationship between countermeasure readability and security intentions. We propose an extended theoretical framework and investigate its implications using a survey. In line with related studies, results indicate that people are more likely to have favorable security intentions if they are aware of countermeasures and are confident in their ability to implement them. Crucially, the data show that countermeasure readability influences security intentions. Our results imply that cybersecurity professionals can utilize readability metrics to assess and improve the readability of countermeasure texts, providing an actionable avenue towards influencing security intentions

Respite for SMEs: A Systematic Review of Socio-Technical Cybersecurity Metrics

Cybersecurity threats are on the rise, and small- and medium-sized enterprises (SMEs) struggle to cope with these developments. To combat threats, SMEs must first be willing and able to assess their cybersecurity posture. Cybersecurity risk assessment, generally performed with the help of metrics, provides the basis for an adequate defense. Significant challenges remain, however, especially in the complex socio-technical setting of SMEs. Seemingly basic questions, such as how to aggregate metrics and ensure solution adaptability, are still open to debate. Aggregation and adaptability are vital topics to SMEs, as they require the assimilation of metrics into an actionable advice adapted to their situation and needs. To address these issues, we systematically review socio-technical cybersecurity metric research in this paper. We analyse aggregation and adaptability considerations and investigate how current findings apply to the SME situation. To ensure that we provide valuable insights to researchers and practitioners, we integrate our results in a novel socio-technical cybersecurity framework geared towards the needs of SMEs. Our framework allowed us to determine a glaring need for intuitive, threat-based cybersecurity risk assessment approaches for the least digitally mature SMEs. In the future, we hope our framework will help to offer SMEs some deserved respite by guiding the design of suitable cybersecurity assessment solutions

A shared cyber threat intelligence solution for smes

Small-and medium-sized enterprises (SMEs) frequently experience cyberattacks, but often do not have the means to counter these attacks. Therefore, cybersecurity researchers and practitioners need to aid SMEs in their defence against cyber threats. Research has shown that SMEs require solutions that are automated and adapted to their context. In recent years, we have seen a surge in initiatives to share cyber threat intelligence (CTI) to improve collective cybersecurity resilience. Shared CTI has the potential to answer the SME call for automated and adaptable solutions. Sadly, as we demonstrate in this paper, current shared intelligence approaches scarcely address SME needs. We must investigate how shared CTI can be used to improve SME cybersecurity resilience. In this paper, we tackle this challenge using a systematic review to discover current state-of-the-art approaches to using shared CTI. We find that threat intelligence sharing platforms such as MISP have the potential to address SME needs, provided that the shared intelligence is turned into actionable insights. Based on this observation, we developed a prototype application that processes MISP data automatically, prioritises cybersecurity threats for SMEs, and provides SMEs with actionable recommendations tailored to their context. Subsequent evaluations in operational environments will help to improve our application, such that SMEs are enabled to thwart cyberattacks in future