Development of a Situational Judgement Test and an assessment of its efficacy as a stimulus of metacognitive behaviour in engineering students

Metacognition entails the conscious evaluation and control of one\u27s cognitive processes. This meta-level control of cognitive process is not essential for all activities, but in the domain of problem solving and the development of new expertise, conscious control of mental functioning is essential to success. Previous studies have shown a relationship between metacognitive knowledge & skills and student self-regulated learning, self-efficacy and more generally, with success in academic and non-academic endeavours; they represent critical skills for an aspiring engineer to possess for their future employability. Metacognition can be stimulated by allowing students to engage and reflect on the problem-solving process. Studies in STEM education focus almost entirely on the use of technical problems for the source of this stimulation. The drawback of this approach is that these problems generally require prior knowledge of physics or mathematics for the students to engage in the process. Recent research utilising naturalistic observations of students’ behaviour while they were engaged in technical problem solving found that metacognitive knowledge and skills can be categorised into discrete metacognitive behaviours. Specifically, metacognitive behaviour can be measured through analysis of students’ discourse with one another as they engage in the problem-solving process. This research utilised a sequential mixed methods design, which contained two strands – the first sought to develop a Situational Judgment Test (SJT) while the second strand sought to utilise the SJT as a stimulus of metacognitive behaviour. An SJT was developed, evaluated by fifty-three engineering professionals in eleven expert panels and rolled out to three hundred and third four final year and masters level engineering students at TU Dublin and KU Leuven, who took the SJT as a test. The SJT items were then delivered to a further fifty-five first year engineering students at TU Dublin, this time in groups, for them to choose responses and discuss them with their peers. The items which stimulated metacognitive behaviour amongst these students were identified using the Naturalistic Observations of Metacognition in Engineering students (NOME) protocol. The resulting items were provided to a group of eight first year engineering students and the NOME protocol was re-applied to evaluate the efficacy of the new metacognitive learning resource in stimulating metacognitive behaviour. The development of a means of stimulating metacognitive behaviour that was not conditional on students’ having prior knowledge of physics and mathematics or a reliance on inventory style assessment allowed iii for a better-quality assessment of a students’ metacognitive knowledge and skills. Allowing students to apply their metacognitive knowledge and skills in groups permitted students to construct tools of higher mental functioning though peer dialogue, using an SJT in the stimulation of this dialogue had pedagogical merit, as particular SJT items proved highly effective in eliciting the use of metacognitive skills. This research work aims to add to engineering education scholarship in three ways. Firstly, to provide an engineering specific SJT to enable educators to identify areas of relative strength and weakness in students’ professional judgements in order to better prepare them for their future careers. Secondly, to use the insights and resources generated from the development and evaluation of the SJT to develop a resource for engineering educators to stimulate students’ metacognitive behaviour that does not rely on a students’ prior knowledge of physics and mathematics, in order to provide them with the skills to self-regulate their learning. Thirdly, this research provides fresh insights into how engineering student’s exhibit metacognitive behaviours when working in groups, adding to an existing body of literature about how students exhibit these behaviours during the problem-solving process

Graduate Engineering Skills A Literature Review & Call for Rigorous Methodological Approaches

What sort of working world do our engineering graduates face? Engineering has become a global profession, where professional skills are as important as the intellectual prowess gained by obtaining the qualification itself. First, we must consider what skills are needed for engineers to meet the challenges of Industry 4.0. Academics who wish to engage in activities to enhance engineering education might therefore initially seek to identify which skills are most important and there is a wealth of literature addressing different viewpoints which adds further complexity to evaluating such studies. This paper reports on two independent systematic reviews of literature to identify the most commonly discussed skills that engineering graduates require from the focus of different stakeholders. The first study audited 129 papers and identified the list of most commonly discussed skills, which was then condensed into a list of 17 professional skills. Independently, a review of the lists of skills used in 16 quantitative studies was carried out in relation to engineering skills requirements for graduate engineers specifically. The results of both studies are compared to highlight the similarities and differences between the results of each method. The work also aims to highlight concerns over providing lists of skills in survey questionnaires without a rigorous research methodology. It is hoped that this paper will generate discussion and aims to raise additional research questions to initiate more in-depth research, into the differing views and contextual relationships of skills’ listing

The development of a psychometric test aimed at aligning students to a range of professional roles

Dublin Institute of Technology (DIT), working as part of the PREFER project, a European commission funded project, have developed a psychometric test in order to better align engineering students to three distinct professional roles within industry. This paper reports on the development process of the test, which took place between February and August of 2018

Embedding Sustainability In The Engineering Curriculum: Meeting The Requirements Of Professional Accreditation

Professional accreditation agencies are increasing requirements on sustainability in engineering education as a response to ethical obligations, industry needs and emerging academic best practice. In 2021, Engineers Ireland increased sustainability requirements in new accreditation criteria. This paper reports on a thematic analysis carried out by Engineers Ireland on the self-assessment and achievement of these new accreditation criteria on sustainability. The analysis was conducted on the self-assessment reports from a large Irish University, referred to as University A hereafter. The results indicate that, for the purpose of meeting accreditation requirements, University A has interpreted sustainability in their programmes as either meeting the UN Sustainability Goals (SDG’s) by mapping modules to the SDG\u27s, or by aligning Programme Area (PA) 7 Sustainability of the Engineers Ireland accreditation criteria with the Engineers Ireland Programme Outcomes (PO\u27s). The paper outlines the main themes and approaches identified across 17 engineering programmes and presents 2 case studies of how sustainability is embedded in engineering curricula in Ireland

Undergraduate Engineers\u27 Preference for a Range of Professional Roles

This paper reports on a personal preference test which aligns students to a range of professional roles based on their attitudes towards performing particular tasks. The 10-item test was administered to 109 first-year engineering students at TU Dublin, Ireland and 159 third-year engineering students at KU Leuven, Belgium in September of the 2018/19 academic year. The test had two purposes: • to align students to three professional engineering roles based on their preference for performing certain tasks; • to allow students to reflect on an initially tacit model of professional roles. In this paper only the first purpose is considered, followed by an evaluation of the reliability of the test. Preliminary results indicate that the majority of students at TU Dublin and at KU Leuven wish to work in roles which involve the development of radically new products and services, while a much smaller proportion of students wish to work with product and process optimisation. The data also indicates that, in general, students have less favourable attitudes towards working in client-centred roles. These findings present a unique challenge for engineering educators and employers alike in Ireland and Belgium, as industries in these nations shift towards services and away from manufacture. So too do the skills requirements to work effectively in the modern engineering sector

A situational judgement test for engineers to evaluate their professional strengths & weaknesses

This paper reports on the development and evaluation of a 23 item Situational Judgement Test (SJT) with scenarios tailored to the engineering profession. The SJT was developed around the PREFER model, with the support of professional engineers and academics in 11 panel discussions. In total 53 engineering professionals and academics were consulted during the development of both the item stems and the item responses of the SJT. Subsequently, the SJT was rolled out to 334 final year and masters students enrolled in engineering programmes at TU Dublin and KU Leuven respectively. After taking part in the test, students were sent automated reports on their performance and the test which highlighted how their response compared to a response gathered from a professional engineer with feedback on how they might improve their competence in a particular area, while also commending their performance in other areas. The results of this study highlight that 8 SJT items had significantly lower mean scores when compared with the test-mean. These items, which were related to perseverance, client focus, vision, planning and organising, solution orientation, team player, work organisation, clear communication and networking all represent potential competence deficits in the population of final year and master students that were tested. This work adds to engineering education scholarship by providing an engineering-specific SJT that enables educators to identify areas of relative strength and weakness in students’ professional judgements in order to better prepare them for their future careers

Engineering students\u27 preferred roles: Are they stable, are there gender differences?

Being able to situate oneself in an engineering role is a developmental process. Students may initially have idealized perceptions of a professional role and over time, they make this role more congruent with their own values and goals [1]. In light of this, Higher Education Institutions are being challenged to offer learning experiences and career exploration activities to enable students to clarify their interests, values and competencies in relation to a professional role [2]. This study compared the professional role preferences of more than 700 engineering students at TU Dublin (Ireland) and KU Leuven (Belgium). Professional role preference was measured with PREFER Explore, a personal preference test for engineers. The test aligns students to three professional roles for early career engineers: Product leadership (focus on radical innovation), Operational excellence (focus on process optimization) and Customer intimacy (focus on tailored solutions and customer satisfaction). A comparison was drawn between the role preference of first year students at TU Dublin and KU Leuven to establish if there were significant differences in preference across both universities. The results suggest that the role preference of engineering students does not shift from first to third year. There is also evidence that the PREFER Explore is sensitive to gender differences, with female students showing a greater preference for customer intimacy than males and males showing a greater preference for operational excellence than females at TU Dublin. The data have a number of implications for the labor market in Ireland and Belgium

Students’ Experiences Of Reflecting On Their Development Of Professional Skills In An Engineering Programme.

Engineers play a central role in addressing the challenges which face society, and recent literature highlights the need for emphasis on the development of professional skills in engineering programmes. This paper describes the outcomes of a study which investigated students’ experiences of reflecting on the development of their professional skills using an ePortfolio in a pilot project. A focus group was used to capture students’ experiences of the reflection process and the use of the ePortfolio. Transcripts were analysed thematically to draw out the key experiences and to provide feed-forward advice for the next iteration of the project. The findings show that students need support in the reflection process, and clearer signposting between each skill and the modules relevant to their development. Students also found it difficult to ascertain their competency levels and felt that industry experience was needed to help score themselves accurately. Feed-forward advice included incorporating an ePortfolio throughout all years of the programme which would track their improvement in a range of skills, and providing a rubric to help assess their competency. The outcome of this study can be used by educators who wish to incorporate a professional skills ePortfolio in their engineering programmes

THE IDENTIFICATION OF FUTURE PROFESSIONAL SKILLS FOR THE GRADUATE STRUCTURAL ENGINEER AND THE CO-CREATION OF THEIR DEFINITIONS.

Employers recognise that the future is changing and as such the structural engineer’s role is changing along with the skill set required. The skills gap has been acknowledged yet there is no consensus on which skills are most important for these engineers. This research presents the outcome of a project which proposes future professional skills needs for the structural engineer and the co-creation of their definitions. A review of the most recent relevant literature alongside chartership requirements of the Institution of Structural Engineers (IStructE) and Engineers Ireland (EI), as well as consideration of three seminal consultation and analysis reports on the future skills in the sector, led to the identification of 7 skills. These are the traditional, though evolving skills related to communication, technical ability, management and engineering practice as well as emerging skills related to sustainability, technology and digitisation and society. It is accepted, however, that there may be different conceptions of each term, therefore, the presented research describes the co-creation of definitions for each of these skills with undergraduate structural engineering students. The work describes how focus groups were used to engage students in a conversation around the meaning and importance of each skill resulting in specific action orientated definitions for each skill. These definitions will then be used in the next phases of the project which engage the same students in a reflective e-portfolio exercise and structural engineering educators in a review of the programme outcomes in relation to such skills