Development of a Situational Judgement Test and an assessment of its efficacy as a stimulus of metacognitive behaviour in engineering students

Metacognition entails the conscious evaluation and control of one\u27s cognitive processes. This meta-level control of cognitive process is not essential for all activities, but in the domain of problem solving and the development of new expertise, conscious control of mental functioning is essential to success. Previous studies have shown a relationship between metacognitive knowledge & skills and student self-regulated learning, self-efficacy and more generally, with success in academic and non-academic endeavours; they represent critical skills for an aspiring engineer to possess for their future employability. Metacognition can be stimulated by allowing students to engage and reflect on the problem-solving process. Studies in STEM education focus almost entirely on the use of technical problems for the source of this stimulation. The drawback of this approach is that these problems generally require prior knowledge of physics or mathematics for the students to engage in the process. Recent research utilising naturalistic observations of students’ behaviour while they were engaged in technical problem solving found that metacognitive knowledge and skills can be categorised into discrete metacognitive behaviours. Specifically, metacognitive behaviour can be measured through analysis of students’ discourse with one another as they engage in the problem-solving process. This research utilised a sequential mixed methods design, which contained two strands – the first sought to develop a Situational Judgment Test (SJT) while the second strand sought to utilise the SJT as a stimulus of metacognitive behaviour. An SJT was developed, evaluated by fifty-three engineering professionals in eleven expert panels and rolled out to three hundred and third four final year and masters level engineering students at TU Dublin and KU Leuven, who took the SJT as a test. The SJT items were then delivered to a further fifty-five first year engineering students at TU Dublin, this time in groups, for them to choose responses and discuss them with their peers. The items which stimulated metacognitive behaviour amongst these students were identified using the Naturalistic Observations of Metacognition in Engineering students (NOME) protocol. The resulting items were provided to a group of eight first year engineering students and the NOME protocol was re-applied to evaluate the efficacy of the new metacognitive learning resource in stimulating metacognitive behaviour. The development of a means of stimulating metacognitive behaviour that was not conditional on students’ having prior knowledge of physics and mathematics or a reliance on inventory style assessment allowed iii for a better-quality assessment of a students’ metacognitive knowledge and skills. Allowing students to apply their metacognitive knowledge and skills in groups permitted students to construct tools of higher mental functioning though peer dialogue, using an SJT in the stimulation of this dialogue had pedagogical merit, as particular SJT items proved highly effective in eliciting the use of metacognitive skills. This research work aims to add to engineering education scholarship in three ways. Firstly, to provide an engineering specific SJT to enable educators to identify areas of relative strength and weakness in students’ professional judgements in order to better prepare them for their future careers. Secondly, to use the insights and resources generated from the development and evaluation of the SJT to develop a resource for engineering educators to stimulate students’ metacognitive behaviour that does not rely on a students’ prior knowledge of physics and mathematics, in order to provide them with the skills to self-regulate their learning. Thirdly, this research provides fresh insights into how engineering student’s exhibit metacognitive behaviours when working in groups, adding to an existing body of literature about how students exhibit these behaviours during the problem-solving process

From design to narrative: the development of inquiry-based learning models

The University of Nottingham and the Open University are partners in a ca. £1.2m project to help school students learn the skills of modern science. The three-year project, Personal Inquiry (PI) (funded by the UK ESRC and EPSRC research councils), is developing a new approach of 'scripted inquiry learning', where children investigate a science topic with classmates by carrying out explorations between their classroom, homes and discovery centres, guided by a personal computer. This paper describes our progress to date on the development of four models for inquiry-based learning, as part of the PI project. These are being used as the basis for the development of educational scenarios and associated scripts to explore the use of mobile technologies in supporting an inquiry-based approach to teaching Scientific thinking across formal and informal learning

D3.2 Instructional Designs for learning analytics and reflection support

In the context of the METALOGUE project this deliverable discusses the role and scope of Learner Analytics in METALOGUE, what type of data is available and should be used, the use of Learning Dashboards and visualisations to enable the learner (and tutor) to access the outcomes of the learner analytics, a set of initial example visualisations to be used in METALOGUE, and, finally, it concludes with an instructional design blueprint giving a global outline of a set of tasks with stepwise increasing complexity and the feedback proposed.METALOGUE10000-01-0

Investigating Developmental Trends in Metacognitive Knowledge with School-aged Children using Pupil Views Templates

This thesis explores developmental trends in metacognitive knowledge with school-aged children using Pupil Views Templates (PVTs). PVTs were developed in the mid 2000s to explore pupil views of learning. Informed by the findings of previous research, the empirical data collection used a more systematic and stratified sampling technique. A systematic review of tools and methods to measure or assess metacognition was included as a way of codifying PVTs. The systematic review makes an original contribution to both this study and the field; in a field as vast as metacognition it provides a valuable summary. The exploration of metacognitive knowledge is based on, but does not completely replicate, the pre-existing approach to coding PVTs. A rigorous examination of relevant literature rationalised and grounded the focus on metacognitive knowledge. This underscored ambiguity around defining metacognition, sub-divisions of it and crossover between these. Thus, the clarity of defining metacognition for and within this study was key. The mixed method approach to PVT analysis was distinctive in its application of traditional statistical analysis and emergent interpretivist methods including word clouds. Analysis confirmed the utility of PVTs as a means to explore metacognition in school-aged children. It supported the assertion that PVTs are a tool that can be used with a wide range of ages to explore metacognitive knowledge, including children as young as four years old. There was evidence of developmental trends in metacognitive knowledge and indications to support inextricable links between underlying cognitive skills and metacognition. This study also showed the importance of considering how metacognition is explored; including the definition of metacognition applied, how it is operationalised and then analysed. If a study does not have clear links between the concept, its measurement and outcomes; it becomes difficult to determine validity and subsequent value both within and for the field

D3.1 Instructional Designs for Real-time Feedback

The main objective of METALOGUE is to produce a multimodal dialogue system that is able to implement an interactive behaviour that seems natural to users and is flexible enough to exploit the full potential of multimodal interaction. The METALOGUE system will be arranged in the context of educational use-case scenarios, i.e. for training active citizens (Youth Parliament) and call centre employees. This deliverable describes the intended real-time feedback and reflection in-action support to support the training. Real-time feedback informs learners how they perform key skills and enables them to monitor their progress and thus reflect in-action. This deliverable examines the theoretical considerations of reflection in-action, what type of data is available and should be used, the timing and type of real-time feedback and, finally, concludes with an instructional design blueprint giving a global outline of a set of tasks with stepwise increasing complexity and the feedback proposed.The underlying research project is partly funded by the METALOGUE project. METALOGUE is a Seventh Framework Programme collaborative project funded by the European Commission, grant agreement number: 611073 (http://www.metalogue.eu)

THE EFFECTS OF DEBRIEFING ON DIAGNOSTIC REASONING DEVELOPMENT IN FAMILY NURSING PRACTITIONER STUDENTS

This dissertation explores the effect of simulation and the debriefing method of Debriefing for Meaningful Learning (DML) on diagnostic reasoning development in family nurse practitioner students (FNP-s) as measured by the diagnostic thinking inventory (DTI). A total of 13 FNP-s participated in this exploratory descriptive pilot study. All students completed both the pre-DTI survey prior to the start of the study, and the post DTI survey at the study conclusion followed by the Debriefing Assessment for Simulation in Healthcare-Student Version (DASH-SV) survey. Students participated in three urgent care simulations followed by the debriefing method of DML. The results of this study used mean comparisons in a repeated measure analysis given the small sample size. Dependent groups t tests revealed significant gains on the knowledge subscale but not on the flexibility items of the DTI, suggesting that the improvement in diagnostic thinking skills evidenced in this sample was due to the increase in knowledge gained from participation in the simulations and associated DML debriefings but not to any significant changes in the flexibility subscales. The effect of the simulations followed by DML method was also evaluated on reaction time (RT) indices. Although the total DTI scores did not show evidence of a significant improvement in time related to the RT to the diagnostic questions, the knowledge subscale of the DTI showed evidence of a significant improvement in RT. The observation that these FNP-s responded to the knowledge subscale of the DTI significantly faster after the intervention than before, provides additional evidence that suggests that the diagnostic knowledge of these FNP-s was improved by this intervention. Knowledge (non-analytic reasoning) was improved by participation in the simulations followed by DML as evidenced by improvement in knowledge decision efficiency (shorter RTs) in this subscale, however, there was not a similar improvement in the RTs in the overall total DTI scores or in the flexibility subscale. Overall scores on this debriefing method using the DASH-SV were positive. Simulation with the debriefing method of DML was found to significantly increase knowledge structure in this small sample of FNP-s

Exploring Female Perceptions of Metacognitive Development in Online Learning

With increased access to higher education through online delivery mediums, it is necessary to evaluate the impact of the learning environment on disadvantaged populations such as female students. As the online learning classroom challenges through distance, isolation, and communication, these factors can influence a positive perception of the learning environment and interfere with deep learning. This qualitative study explored female perceptions of metacognitive development within the online learning environment, as metacognition is a core element of academic success in higher education. Through the design of the conceptual framework and with the support of the literature review, a methodology was selected to holistically explore the female experience in light of deep learning achievement and their use of metacognitive practices. Participants were recruited according to selective criteria and engaged in the study through semistructured interviews, personal journal entries, and the presentation of an artifact. A meticulous coding process was used to analyze the data, which revealed four primary themes and nine subthemes. The analysis supports the importance of metacognitive development as influential in course completion, yet offered insight into factors affecting a positive perception of the learning environment. Key themes of identity, community, self-efficacy, and surface learning prompted a critical look at implications for future practice and policy within the online learning context. A response to these implications that will generate a more targeted metacognitive focus should include a stronger teacher presence within the online classroom, diversified instructional methods, and an increased endorsement of the value of the online classroom community

Creative Solutions and their Evaluation: Comparing the Effects of Explanation and Argumentation Tasks on Student Reflections

Creative problem solving which results in novel and effective ideas or products is most advanced when learners can analyze, evaluate, and refine their ideas to improve creative solutions.  The purpose of this investigation was to examine creative problem solving performance in undergraduate students and determine the tasks that support critical self-evaluations of creative solutions by comparing alternative types of reflective tasks. Participants (n = 103) first provided demographic information and responded to individual difference measures (i.e., divergent thinking, need for cognition, and beliefs about creative outcomes) and then read a problem scenario in which they assumed the role of a high school teacher who was asked to design a creative college preparatory course.  Following, participants completed either an explanation reflective task or an argument based reflective task.  Finally, participants evaluated their proposed course by rating it on characteristics that describe the originality and effectiveness of creative solutions. Findings confirmed the role of divergent thinking as a positive predictor of the originality of a creative solution, whereas, need for cognition, and academic major were positive predictors of the effectiveness of a creative solution.  Participants rated their creative solutions differentially depending on their beliefs and the type of reflective task.  Those whose beliefs aligned better with conceptualizations of creative outcomes assessed more positively the originality and effectiveness of their solution.  The findings indicate that the argumentation task could potentially promote reflective and critical thinking about a creative solution as participants who completed the argumentation task evaluated their solution more conservatively