Dublin Institute of Technology (DIT), working as part of the PREFER project, a European commission funded project, have developed a psychometric test in order to better align engineering students to three distinct professional roles within industry. This paper reports on the development process of the test, which took place between February and August of 2018

Bowe, Brian

Carthy, Darren Brian

Gaughan, Kevin

English

Arrow@TUDublin

Technological University Dublin ARROW@TU Dublin Conference papers School of Multidisciplinary Technologies 2018-11-01 The development of a psychometric test aimed at aligning students to a range of professional roles Darren Brian Carthy Technological University Dublin, c11403568@mydit.ie Brian Bowe Technological University Dublin, Brian.Bowe@TUDublin.ie Kevin Gaughan Technological University Dublin, Kevin.Gaughan@tudublin.ie Follow this and additional works at: https://arrow.tudublin.ie/schmuldistcon  Part of the Education Commons, and the Other Engineering Commons Recommended Citation Carthy, Bowe & Gaughan (2018) The development of a psychometric test aimed at aligning students to a range of professional roles, Proceedings of the 6th UK & IE Engineering Education Network Symposium, 1-2 November 2018, Portsmouth, UK This Conference Paper is brought to you for free and open access by the School of Multidisciplinary Technologies at ARROW@TU Dublin. It has been accepted for inclusion in Conference papers by an authorized administrator of ARROW@TU Dublin. For more information, please contact yvonne.desmond@tudublin.ie, arrow.admin@tudublin.ie, brian.widdis@tudublin.ie. This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 License The development of a psychometric test aimed at aligning students to a range of professional roles  Darren Carthy, * Kevin Gaughan, * Brian Bowe * *CREATE Research Group, Dublin Institute of Technology, Bolton Street, Dublin    Corresponding Author’s Email: darren.carthy@dit.ie    KEY WORDS: Graduate, competence, skills, engineering    ABSTRACT Dublin Institute of Technology (DIT), working as part of the PREFER project, a European commission funded project, have developed a psychometric test in order to better align engineering students to three distinct professional roles within industry. This paper reports on the development process of the test, which took place between February and August of 2018.   BACKGROUND / CONTEXT  The PREFER psychometric test consists of two parts: a motivation questionnaire and a competence test. In the motivation questionnaire, students were asked to indicate their preferred course of action on a particular item from a list of 3 responses. Each course of action was aligned to one of the three professional roles within the PREFER professional roles framework as illustrated in figure 1.   Figure 1. The model of professional roles utilised in the PREFER project   The main objective of the questionnaire was to familiarize participants with the model and raise awareness about the variety of engineering functions an engineer fulfils. It was operationalised as a questionnaire to assess a participant’s motivation to complete certain tasks and one major advantage of this format was that the keying process was quite straightforward, making it much easier to map a particular choice to a role. Another advantage of this approach was that the questionnaire required only a small amount of cognitive effort to complete. As students were subsequently asked to complete the competence test (a cognitively loaded test), it was desirable to keep cognitive load at a minimum in the motivation test.   The competence test was developed as a Situational Judgement Test (SJT) to assess how a participant distinguishes appropriate responses from inappropriate responses to a particular dialogue between 2 actors. These dialogues represented realistic engineering scenarios drawn from multiple interviews with practicing engineers, in the next phase of the research the test will undergo a rigorous face validation drawing on the methods proposed by Nevo (Nevo, 1985). The objective of the competence test was to evaluate a students’ fit to a particular professional role based on the ratings they provide to each of the responses on a particular item. This was achieved by developing the test items around the competences collected from the expert panels. A measurement of role fit rather than competence fit was chosen because measurement of competence fit would have required a large number of observations per competence for a reliable measurement. Looking for a role fit allowed 23 competences to be OperationalexcellenceProductleadershipCustomerintimacymeasured and used as indicators of the role fit such that the construct under examination was the role, not the individual competences.   LITERATURE REVIEW / RATIONALE   In 2014, a study into the labour markets of Europe revealed that a large number of European countries encounter difficulties when recruiting STEM skilled labour. Looking at past, current and future trends, the study identified a number of occupations in Ireland as mismatch priority occupations. An occupational mismatch occurs when there are more job vacancies than individuals who meet the criteria to fulfil the role. Following an assessment of Irish labour markets, engineering professionals and technicians were identified as a sector with a high degree of occupational mismatch (EU Skills Panorama, 2014). The report suggests that at a European level:  "There are emerging skills shortages, skills gaps and recruitment difficulties . . . These include the need to combine the STEM skills of graduates with the ‘soft’ employability skills as communication skills, team working and creative thinking which help apply STEM skills in the business world and which are important to innovation "  pg. 4  There is an ongoing shift in the labour market as industries moves away from traditional manufacturing and begin to take a larger stake in the services sector (Schettkat and Yocarini, 2003).This shift has resulted in companies and firms seeking out graduates who possess the transversal or professional skills required for success in these new roles. To this end, the PREFER project was established to identify the competences associated with the 3 professional roles as outlined by the PREFER model, to allow students to reflect on these roles and identify their relative strengths and weaknesses within a particular role and finally to develop unique curriculum elements to help train these competences.    AIM AND OBJECTIVES / RESEARCH QUESTION(S)  The PREFER psychometric test aims to evaluate participants’ motivations to work in a particular professional role and to align each participant to one of the three roles based on 23 competence indicators. Moreover, the test will provide tailored feedback to the participant about their motivations and their fit to a particular professional role.  Once the test has been placed through a full validation, it will be transformed into a pedagogical tool to give students a better sense of who they are as engineers and to allow them to reflect on the competences they require for success in the labour market.    METHODOLOGICAL APPROACH   Adopting the view that a unique set of competences can be attached to each of the three professional roles, a modified DELPHI study was carried out to gain industry’s perspectives on the competences they deemed most important for each role. These competences and their definitions were drawn from Binder Dijker Otte’s (BDO’s) list of competences. The DELPHI method has proven a popular method in identifying core competences which are deemed important in a number of fields, including food safety undergraduates (Johnston et al., 2014) audio engineering technology undergraduates (Tough, 2009) and management information systems graduates (Strnad, 2013). First, drawing data from the industry partners of the PREFER project in 3 expert panels with Electricity Supply Board International (ESBI) in Ireland, Siemens in the Netherlands and ENGIE in Belgium, whom all engage in similar economic activities, the study was expanded to include 9 additional expert panels which were conducted with an automation firm, a construction consultancy firm and a telecommunications firm. Several other companies and consultancies were also included, with variation in both firm size and economic activities to capture a wide cross section of perspectives from engineers across Ireland, Belgium and The Netherlands. (Craps et al., 2018). In the final stage of the development process, the psychometric test items were brought back to ESBI, ENGIE & Siemens for their HR departments for further consideration and feedback.    DISCUSSION & FUTURE WORK  The next stage of the research involves establishing the face validity of the test items on the SJT. This will be achieved by triangulating the feedback provided by the HR representatives in ESBI, ENGIE & Siemens with the feedback from 3 expert panels of engineering academics. This research constitutes the face validation phase. Following on from this, the same items will be provided to practicing engineers to establish an expert scoring key to appropriately weight the responses to each of the items. After the face validation phase a pilot version of the SJT will be combined with the motivation test and piloted on a cohort of engineering students the data from which will form the basis of a concurrent validation study.    CONCLUSIONS & RECOMMENDATIONS  In conclusion, a vast amount of literature has explored the transversal skills required for an engineer to succeed in the labour market (Carthy, Gaughan and Bowe, 2018) but there is a scarcity of research regarding the types of roles that an engineer can fulfil after graduation (Craps et al., 2017). It is therefore the aim of the PREFER project to fill this void in the literature and explore not only the roles available to new graduates but also to help students to develop the competences required to succeed in these roles.    REFERENCES  Carthy, D., Gaughan, K. and Bowe, B. (2018) ‘What are the engineering professional competences ?’, in 46th SEFI Conference, 18-21 September 2018, Copenhagen, Denmark.  Craps, S. et al. (2017) ‘Professional roles and employability of future engineers’, Proceedings of the 45th SEFI Annual Conference 2017 , Azores, Portugal, (September), pp. 18–21.  Craps, S. et al. (2018) ‘Wanted : Super ( wo ) man . A study to define professional roles for future engineers , defined by professional competences Wanted : super ( wo ) man A study to define professional roles for future engineers by distinctive professional competences .’, in 46th SEFI Conference, 18-21 September 2018, Copenhagen, Denmark.  EU Skills Panorama (2014) EU Skills Panorama 2014. Available at: http://skillspanorama.cedefop.europa.eu/sites/default/files/EUSP_AH_STEM_0.pdf.  Johnston, L. M. et al. (2014) ‘Identification of Core Competencies for an Undergraduate Food Safety Curriculum Using a Modified Delphi Approach’, Journal of Food Science Education, 13(1), pp. 12–21. doi: 10.1111/1541-4329.12024.  Nevo, B. (1985) ‘Face Validity Revisited’, Journal of Educational Measurement, 20(4), pp. 287–293. doi: 1529-8604.  Schettkat, R. and Yocarini, L. (2003) ‘The Shift to Services: A Review of the Literature’, IZA Discussion Paper, 964(964), p. 44.  Strnad, M. (2013) ‘Recommended skill requirements of recent management information systems graduates for employment: A modified DELPHI study’, ProQuest LLC, (April).  Tough, D. T. (2009) ‘Developing a Consensus-Driven, Core Competency Model to Shape Future Audio Engineering Technology Curriculum: A Web-based Modified Delphi Study’, ProQuest LLC, (December), p. 374. Available at: http://www.eric.ed.gov/ERICWebPortal/recordDetail?accno=ED513721.  

The development of a psychometric test aimed at aligning students to a range of professional roles

https://arrow.tudublin.ie/cgi/viewcontent.cgi?article=1022&context=schmuldistcon

The development of a psychometric test aimed at aligning students to a range of professional roles

Abstract

Similar works

Full text

Available Versions

Arrow@TUDublin