Gamification as a neuroergonomic approach to improving interpersonal situational awareness in cyber defense

In cyber threat situations, the establishment of a shared situational awareness as a basis for cyber defense decision-making results from adequate communication of a Recognized Cyber Picture (RCP). RCPs consist of actively selected information and have the goal of accurately presenting the severity and potential consequences of the situation. RCPs must be communicated between individuals, but also between organizations, and often from technical to non-/less technical personnel. The communication of RCPs is subject to many challenges that may affect the transfer of critical information between individuals. There are currently no common best practices for training communication for shared situational awareness among cyber defense personnel. The Orient, Locate, Bridge (OLB) model is a pedagogic tool to improve communication between individuals during a cyber threat situation. According to the model, an individual must apply meta-cognitive awareness (O), perspective taking (L), and communication skills (B) to successfully communicate the RCP. Gamification (applying game elements to non-game contexts) has shown promise as an approach to learning. We propose a novel OLB-based Gamification design to improve dyadic communication for shared situational awareness among (technical and non-technical) individuals during a cyber threat situation. The design includes the Gamification elements of narrative, scoring, feedback, and judgment of self. The proposed concept contributes to the educational development of cyber operators from both military and civilian organizations responsible for defending and securing digital infrastructure. This is achieved by combining the elements of a novel communication model with Gamification in a context in urgent need for educational input.publishedVersio

Cognisance as a Human Factor in Military Cyber Defence Education

The authors may share the final published article on public non-commercial sites in the terms of the Creative Commons CC-BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/. Additionally, the authors may use the article in a subsequent compilation of their work, or extend it to book length form, or include it in a thesis or dissertation, or otherwise to use or re-use portions or excerpts in other works, for both commercial and non-commercial purposes. All copies of the article should link to the original publication in IFAC PapersOnline via its DOI and have a copyright statement and a reference to the CC-BY-NC-ND license. Except for such uses, IFAC has the exclusive right to make or sub-license commercial useCyber Defence Exercises (CDX) are common training and learning tools. A recently discussed challenge in cyber defence teaching and training is the gap between the fast technological advancement accompanied by rapidly changing demands on future cyber defence operators, and the lack of science-based teaching and training methods. A growing body of evidence suggests a crucial role of human factors as a central predictor for human performance in sociotechnical systems. While this has been acknowledged in a wide range of safety-critical applied fields, there is still a lack of knowledge about the impact of human factors on cyber defence performance. The lack of conventional metrics of performance and learning progress contribute to this deficit. To address this gap, the Norwegian Defence Cyber Academy (NDCA) follows a science-based educational approach that identified in a series of empirical studies cognitive-psychological predictors for learning success of future cyber defence operators. These predictors and elements of a human factors research program are deeply embedded into educational practice and include processes such as metacognition, self-regulation, coping, communication and shared mental modelling. Slow education methods and mentoring are fundamental to enabling the advancement of human factors cognisance among military cyber cadets. As a tool for efficient training, the NDCA developed and implemented a mentoring concept that involves a cyber defence retrospective timeline analysis involving expert and practitioner level mentors. The timeline differentiates between performance relevant hard-and soft-skills and leads progressively towards an alignment of Security Operation Centre (SOC)-and expert judgments of performance. The NDCA argues that this educational concept facilitates educational benefits based on insight, accurate self-perception, motivation and decreased team workloads following more efficient collaboration.publishedVersio

Using metacognitive monitoring feedback to improve student learning in augmented reality environments

This research aims to use metacognitive monitoring feedback to improve student learning performance in an augmented reality environment. In this study, Microsoft HoloLens, a prominent augmented reality device and independent mobile computer, provided a more realistic augmented reality environment to engineering students. The near field electromagnetic ranging system collected students' real-time location data when they experienced the augmented reality learning modules. In Phase 1, the study utilized one of the topics in the Ergonomic class, called manual material handling. The Phase 1 experiment results showed that retrospective confidence judgments in augmented reality modules could significantly influence the way students learn when the contents require a high level of spatial awareness during content learning. Therefore, Phase 2 research considered specific engineering education related to spatial recognition. For Phase 2, the location-based augmented reality system was developed to improve user interaction. The augmented reality learning module was biomechanics: one of the Ergonomic class problematic concepts to engineering students. This new location-based augmented reality system allowed students to immerse themselves in the studying process and improved student engagement of hands-on training in an augmented reality environment. Metacognitive monitoring feedback was another tool applied to improve students' learning performance. Student test scores, confidence level, answering time, and reviewing time were collected as metrics for performance assessment during the experiment. Overall, Phases 1 and 2 study outcomes advanced our understanding of students' interactions and the learning content in an augmented reality learning environment. This study also provided a guideline for how engineers need to develop valuable learning content in augmented reality 'environments. Furthermore, using a metacognitive monitoring feedback tool in an augmented reality learning environment is an effective strategy to improve students' academic performance and calibration.Includes bibliographical references (pages 93-108)

Socio-technical communication: The Hybrid Space and the OLB-Model for science-based cyber education

publishedVersio

Exploring the development of clinical reasoning skills among doctors-in-training

Clinical reasoning is complex, difficult to conceptualise and learn, and important as it is closely linked with medical expertise. Learning clinical reasoning skills is primarily an unguided and subconscious process for doctors-in-training, and there is a need for an evidence based, explicit approach to support the learning of these core skills. The focus of this research is the process by which doctors-in-training learn clinical reasoning skills within the context of General Medicine in north Queensland. The literature to date has been extensive but has struggled to identify a practical framework for doctors-in-training which clearly supports their learning of clinical reasoning skills. This program of research investigated four factors identified in the literature as influencing the development of clinical reasoning skills: the metacognitive awareness levels of doctors-in-training; the learning climate of Intern doctors in their first year of clinical work; the influence of Consultants; and the role of Interns as learners. The first factor was investigated by exploring whether metacognitive awareness correlated with performance in medical undergraduate examinations, and whether there was an increase in metacognitive awareness from the first to the fifth-year of the undergraduate medical course. Volunteer medical students completed the Metacognitive Awareness Inventory (MAI), as well as consenting to give access to their examination scores for this study. For the first-year undergraduate doctors-in-training there were correlations between the Knowledge of Cognition domain of the MAI and their end of year examination results, but not with the Regulation of Cognition domain. For fifth-year students there were correlations between both the Knowledge and Regulation of Cognition domains and their end of year examination results. This study found that the overall MAI scores were not significantly different between first and fifth-year undergraduates in this sample. The Regulation of Cognition domain and its sub-domains, regarded as key factors in clinical reasoning skill development, did not significantly differ between first and fifth-year undergraduate doctors-in-training. The second factor investigated was whether the learning climate of Intern doctors-in-training was conducive to learning. The validated Dutch Resident Educational Climate Test (D-RECT) was used, and written responses invited to the question 'What three aspects of the junior doctor learning environment would you alter?' The Coaching and Assessment and the Relations between Consultants domains were identified as significantly lower in General Medicine than for other units, triangulating the written comments provided by the Interns. The third factor investigated Consultant Physicians as role models for doctors-in-training learning clinical reasoning skills. The focus of the semi-structured interviews explored how the Physicians understood clinical reasoning, their understanding of how they had acquired these skills, and the ways they sought to foster these skills among their doctors-in-training. The seven Consultants described their journey to gaining clinical reasoning expertise as being unguided, generally subconscious and seldom discussed. Most Consultants spoke of being unaware of their own journey to gaining clinical reasoning expertise, and did not regard themselves as role models for doctors-in-training. Most Consultants indicated that acquiring clinical knowledge and learning to think about their decision-making processes (metacognition), were crucial for acquiring expertise, but very few Consultants explained how they could intentionally foster these skills. The final factor was explored by investigating how Intern doctors-in-training understood their own development of clinical reasoning skills. At the start of their General Medicine term, Interns were presented with basic information about clinical reasoning. At the end of that term, participating Interns were interviewed. A paper copy of the presentation given at the start of the term was used to stimulate Intern reflections on their learning during the General Medicine term. The 27 Interns interviewed identified that learning clinical reasoning was a tacit, personal journey influenced by enabling and inhibitory factors. The Interns attributed the differences between their clinical reasoning skills and those of their Consultants as being primarily due to the experience and superior clinical knowledge of the Consultants. A multi-methods research design was used to answer the research questions across the four studies. The first two factors were investigated using quantitative methods, while qualitative methods were employed for the last two. The multi-methods approach enabled findings from the separate studies to be triangulated, supporting confidence in the trustworthiness of the synthesised outcomes and reducing an over-dependence on any individual study. The Synthesis and Proposed Framework chapter initially integrates the findings from the four studies to provide an overall understanding of how clinical reasoning skills are currently fostered in north Queensland. These synthesised results are then used to propose an evidence-based learning model and a method for its implementation at the teaching hospital. The modified Cognitive Apprenticeship Learning Model (mCALM) could help to make expert thinking visible by explicitly supporting constructivist learning practices, metacognitive skills, deliberate practice and a conducive learning climate. The mCALM appears well suited to explicitly fostering the learning of clinical reasoning skills for doctors-in-training in north Queensland

Driver behaviour with adaptive cruise control

This paper reports on the evaluation of adaptive cruise control (ACC) from a psychological perspective. It was anticipated that ACC would have an effect upon the psychology of driving, i.e. make the driver feel like they have less control, reduce the level of trust in the vehicle, make drivers less situationally aware, but workload might be reduced and driving might be less stressful. Drivers were asked to drive in a driving simulator under manual and ACC conditions. Analysis of variance techniques were used to determine the effects of workload (i.e. amount of traffic) and feedback (i.e. degree of information from the ACC system) on the psychological variables measured (i.e. locus of control, trust, workload, stress, mental models and situation awareness). The results showed that: locus of control and trust were unaffected by ACC, whereas situation awareness, workload and stress were reduced by ACC. Ways of improving situation awareness could include cues to help the driver predict vehicle trajectory and identify conflicts

Gamification as a neuroergonomic approach to improving interpersonal situational awareness in cyber defense

In cyber threat situations, the establishment of a shared situational awareness as a basis for cyber defense decision-making results from adequate communication of a Recognized Cyber Picture (RCP). RCPs consist of actively selected information and have the goal of accurately presenting the severity and potential consequences of the situation. RCPs must be communicated between individuals, but also between organizations, and often from technical to non−/less technical personnel. The communication of RCPs is subject to many challenges that may affect the transfer of critical information between individuals. There are currently no common best practices for training communication for shared situational awareness among cyber defense personnel. The Orient, Locate, Bridge (OLB) model is a pedagogic tool to improve communication between individuals during a cyber threat situation. According to the model, an individual must apply meta-cognitive awareness (O), perspective taking (L), and communication skills (B) to successfully communicate the RCP. Gamification (applying game elements to non-game contexts) has shown promise as an approach to learning. We propose a novel OLB-based Gamification design to improve dyadic communication for shared situational awareness among (technical and non-technical) individuals during a cyber threat situation. The design includes the Gamification elements of narrative, scoring, feedback, and judgment of self. The proposed concept contributes to the educational development of cyber operators from both military and civilian organizations responsible for defending and securing digital infrastructure. This is achieved by combining the elements of a novel communication model with Gamification in a context in urgent need for educational input

Aircrew Adaptive Decision Making: A Cross-Case Analysis

Although the accident rate for military aviation has declined significantly from earlier decades, during the 1990\u27s it reached a plateau. Human error in the cockpit still accounts for over 80% of the aircraft mishaps resulting in loss of life or over one million dollars in damage. Decision error has been a contributing factor for approximately 60% of these mishaps. The purpose of this research was to investigate aircrew process performance variables as predictors of decision-making outcomes. This study was modeled on elements of previous research in naturalistic decision making. Data were collected for cross-case analysis of the role experience plays in efficient decision strategy selection and use in an uncertain, dynamic high stakes environment. Multiple raters evaluated eight novice and eight experienced military aircrews at seven decision points in a 20-minute flight scenario conducted in a full motion flight simulator. Other raters independently rank ordered the quality of the final outcome. A comprehensive approach to collecting and analyzing data included: (1) development and use of a behaviorally-anchored assessment instrument, (2) use of a digitally integrated presentation of audio/video and flight data, and (3) development of context-specific analytical frameworks and models of observed behaviors and metacognitive processes. Results included inferential and descriptive statistics of process/outcome scores, instructor comments, excerpts of cockpit recordings, participant interviews, and field notes. The study findings were: (1) high individual and collective crew experience had a significant positive effect on process and outcome scores, (2) there was no statistically reliable difference in process scores between experience levels in the three procedurally-based events, (3) experienced crews performed better than novice crews in the four less structured events, (4) novice crews\u27 process/outcome correlation did not approach significance, (5) a strong positive correlation of process/outcome scores was found for experienced crews in the two most challenging (i.e., unstructured) scenario events, (6) qualitative analysis revealed strong relationships between performance and crew interactions/attributes, and (7) in dynamic, time critical situations, the use of adaptive decision-making strategies led to better performance outcomes

Final Report: Review of the FITS Program: Program Tasks, Goals and Pilot Training Initiatives

In 2002 the FAA, academic and industry partners established the FAA/Industry Training Standards (FITS) program whose purpose is to modernize General Aviation (GA) pilot training. The FAA recognized the need to modernize training standards for pilots who would use new avionics technology that integrate the GPS (Global Positioning Systems) with the autopilot along with multifunction displays capable of depicting flight path, weather, terrain and traffic information. These avionics and displays are touted as improving safety by enhancing pilot Situational Awareness and reducing pilot workload. The new technology has highlighted the need for programs to train and certify pilots to use the avionics suites. The instrumentation places new demands on pilots including changes in the level and distribution of pilot workload during a flight, the need to manage and integrate information from multiple displays, navigate complex menu structures, and program navigation computers. The literature describing the FITS program argues that the current structure and content of GA pilot training programs will not adequately prepare pilots for the challenges of using these technologies (FAA, 2003a; Glista, 2003b; Wright, 2002). The FITS curriculum attempts to address these issues by stressing training of risk management (RM), situational awareness (SA), aeronautical decision making (ADM) and single-pilot resource management (SRM). It also proposes to change pilot instruction to make it more relevant to real world flying by relying on scenario-based training (SBT). FITS proposes to emphasize the use of scenarios as a means to practice the integration of individual skills as they might occur in the real world. For instance, a student pilot might be instructed to plan a flight from Wichita, KS to Kansas City, MO. The student would perform all the tasks necessary to plan the flight including preflight checks, route planning, checking the weather reroute etc. During the flight the student would demonstrate individual flight skills including turns, climbs, navigation, and communication while executing the scenario. The purpose of this project was to review research related to the proposed initiatives and to identify future research needs to support the long-term objectives of FITS. In addition to reviewing pertinent academic and government literature, the objectives of FITS were reviewed with representatives of the FAA, academic and industry partners. At present FITS materials provide few details regarding important components of the training initiative including decision making, the training requirements of advanced avionics technology and its effects on situation awareness. Future work should draw on an extensive academic literature and on lessons learned from prior industry experience when similar avionics technologies were introduced to commercial aviation. Also, clear distinctions should be made between SBT as employed in FITS and SBT used by the military and in commercial aviation. These are very different programs

Development of a Situational Judgement Test and an assessment of its efficacy as a stimulus of metacognitive behaviour in engineering students

Metacognition entails the conscious evaluation and control of one\u27s cognitive processes. This meta-level control of cognitive process is not essential for all activities, but in the domain of problem solving and the development of new expertise, conscious control of mental functioning is essential to success. Previous studies have shown a relationship between metacognitive knowledge & skills and student self-regulated learning, self-efficacy and more generally, with success in academic and non-academic endeavours; they represent critical skills for an aspiring engineer to possess for their future employability. Metacognition can be stimulated by allowing students to engage and reflect on the problem-solving process. Studies in STEM education focus almost entirely on the use of technical problems for the source of this stimulation. The drawback of this approach is that these problems generally require prior knowledge of physics or mathematics for the students to engage in the process. Recent research utilising naturalistic observations of students’ behaviour while they were engaged in technical problem solving found that metacognitive knowledge and skills can be categorised into discrete metacognitive behaviours. Specifically, metacognitive behaviour can be measured through analysis of students’ discourse with one another as they engage in the problem-solving process. This research utilised a sequential mixed methods design, which contained two strands – the first sought to develop a Situational Judgment Test (SJT) while the second strand sought to utilise the SJT as a stimulus of metacognitive behaviour. An SJT was developed, evaluated by fifty-three engineering professionals in eleven expert panels and rolled out to three hundred and third four final year and masters level engineering students at TU Dublin and KU Leuven, who took the SJT as a test. The SJT items were then delivered to a further fifty-five first year engineering students at TU Dublin, this time in groups, for them to choose responses and discuss them with their peers. The items which stimulated metacognitive behaviour amongst these students were identified using the Naturalistic Observations of Metacognition in Engineering students (NOME) protocol. The resulting items were provided to a group of eight first year engineering students and the NOME protocol was re-applied to evaluate the efficacy of the new metacognitive learning resource in stimulating metacognitive behaviour. The development of a means of stimulating metacognitive behaviour that was not conditional on students’ having prior knowledge of physics and mathematics or a reliance on inventory style assessment allowed iii for a better-quality assessment of a students’ metacognitive knowledge and skills. Allowing students to apply their metacognitive knowledge and skills in groups permitted students to construct tools of higher mental functioning though peer dialogue, using an SJT in the stimulation of this dialogue had pedagogical merit, as particular SJT items proved highly effective in eliciting the use of metacognitive skills. This research work aims to add to engineering education scholarship in three ways. Firstly, to provide an engineering specific SJT to enable educators to identify areas of relative strength and weakness in students’ professional judgements in order to better prepare them for their future careers. Secondly, to use the insights and resources generated from the development and evaluation of the SJT to develop a resource for engineering educators to stimulate students’ metacognitive behaviour that does not rely on a students’ prior knowledge of physics and mathematics, in order to provide them with the skills to self-regulate their learning. Thirdly, this research provides fresh insights into how engineering student’s exhibit metacognitive behaviours when working in groups, adding to an existing body of literature about how students exhibit these behaviours during the problem-solving process