Fischer, Andreas

Martin, Alexander

Martin Alexander

Fischer Andreas

University of Szeged

76 D1 – TEMATIKUS SZEKCIÓ/PAPER SESSION ÁPRILIS 21. (CSÜTÖRTÖK) 17.30–19.00 Assessing thinking skills Díszterem/Ceremonial Hall DEVELOPING SITUATIONAL JUDGMENT TASKS BASED ON COMPLEX PROBLEMS OF MULTIPLE DOMAINS Alexander Martin, Andreas Fischer University of Heildeberg Keywords: Complex Problem Solving; Situational Judgment Task; Domain-specificity Assessment of Complex Problem Solving (CPS) is often based on dynamic computer simulations of complex problems, although incremental validity over traditional measures of cognitive ability tends to be small. In the current study we introduce a different approach for assessing CPS: Sternberg (2000) assessed practical intelligence (i.e., the ability to solve real problems) using Situational Judgment Tests (SJTs). In general, situational judgment tests show good criterion-related validity above intelligence and personality measures. However, all traditional SJTs are based on short situational stems and do not include dynamic features. The current test development expands on traditional SJTs by incorporating highly complex situational descriptions and dynamic elements. Another issue that is insufficiently adressed by current research on CPS is the domain specifity of complex problem solving: domains refer to a concrete context, and differ from each other in that they require (and trigger) distinct thoughts and actions. According to dual space theory, domain specific prior knowledge is crucial to the hypothesis being tested and thereby, for action planning. Lastly, cognitive load theory claims that prior learned domain-specific schemes reduce cognitive load on working memory when solving complex problems. By using complex problems from multiple domains, the present study allows for gaining further insights in the domain specifity of complex problem solving. Weekley, Ployhart & Holtz (2006) describe an empirical and a theory-based approach on developing situational judgment tests. In the current study we applied a hybrid approach, because theoretical criteria of complex problem solving had to be taken into account along with the goal of generating realistic, domain-specific situational descriptions and respective solving approaches. In the first phase situational content was gathered using a modified critical incidents technique. Partially structured interviews were conducted with experts from three professional domains – physicists, psychotherapists and business consultants. Afterwards, situational descriptions were standardized and sent out to experts from the respective domains asking them to generate problem-solving approaches. Lastly, prototypical solutions were selected, standardized and presented to a larger sample of experts. As during the second and third phase experts were required to generate and respectively rate solutions to all of the three problem scenarios, main effects as well as within-between interactions will be calculated using a repeated-measures analysis of variance. Results revealed strong qualitative differences between experts of different domains, both with regard to the problems they described (phase 1) and with regard to the solutions they suggested (phase 2). Results of phase 3 will also be presented at the conference and impliations for assessment of CPS will be discussed.      

Developing situational judgment tasks based on complex problems of multiple domains

http://acta.bibl.u-szeged.hu/61340/1/pek_2016_076.pdf

Developing situational judgment tasks based on complex problems of multiple domains

Abstract

Similar works

Full text

Available Versions

University of Szeged