Greiff, Samuel

Iliescu, Dragos

Stadler, Matthias

Open Repository and Bibliography - Luxembourg

EditorialValidly AuthenticSome Recommendations to Researchers Using Simulationsin Psychological AssessmentMatthias Stadler1, Dragos Iliescu2, and Samuel Greiff31 Department of Psychology, Ludwig-Maximilians-University of Munich, Germany2 Faculty of Psychology and Educational Sciences, University of Bucharest, Romania3 Department of Behavioural and Cognitive Sciences, University of Luxembourg, LuxembourgTechnology has become an indispensable tool for educa-tional and psychological assessment in today’s world(Goldhammer et al., 2019). Digital assessments facilitatethe creation of new types of stimuli and response formatsthat were out of reach for assessments using paper; forinstance, interactive simulations may include multimediaelements, as well as virtual or augmented realities (Cipressoet al., 2018; de-Juan-Ripoll et al., 2018). These types ofassessments also allow for widening the construct coveragein an assessment, for instance, through stimulating andmaking visible specific problem-solving strategies thatrepresent new forms of problem-solving (Han et al., 2019).Correspondingly, simulation-based assessment tools thatauthentically represent certain aspects of reality but remainsubject to the control of users (researchers and practitioners)have long been proposed to bridge the gap between labora-tory and field assessment (Brehmer & Dörner, 1993).Indeed, simulations have proven to be potent tools inassessment and training across many different domains(Chernikova et al., 2020), with a primary contribution tothe assessment’s realism and ecological validity. Unlikemore traditional assessment instruments, simulations oftenemulate real-world problem-solving scenarios attempting tocapture highly complex abilities such as medical reasoning(Schuwirth & van der Vleuten, 2003) or diagnostic compe-tencies (Chernikova et al., 2020). Striving for authentic sim-ulations harbors the danger of losing focus of the intendedconstruct by introducing additional challenges that requiredifferent competencies to solve the problems posed bythe simulations, such as handling the interface or, moregenerally, competencies that are not construct-relevantbut rather relate to, for instance, human-computer interac-tion. In other words, the authenticity of a simulation mayadd systematic variance to the measure that is not neces-sarily due to the intended construct.In this Editorial, we encourage researchers to developcomplex simulations to assess novel and exciting con-structs without losing track of their targeted constructswhen designing authentic simulations (Greiff, 2017). Wewill also focus on the pervasive difficulty of adequate con-sideration of participants’ previous knowledge in measuresof ability (such as general or domain-specific problemsolving).The Interplay Between Knowledgeand Ability in SimulationsTo illustrate the interplay between knowledge and ability insimulations, take this example. A highly domain-generalsimulation might ask a participant to determine the relationbetween a set of fictional drugs and some fictional diseasesby interacting with the simulation (see, e.g., Greiff et al.,2015). In this case, medical knowledge does not help suc-ceed in the task, as all features are fictional. However, crit-ics could affirm that such a simulation is not authenticenough – and it would be effortlessly possible to shift thissimulation toward an authentic medical scenario in whichthe names of drugs and diseases and their relations presentrealistic cases (see, e.g., Drews et al., 2006). In this case,the construct assessed would shift from domain-generalreasoning to domain-specific (medical) reasoning.Suppose the task becomesmaximally realistic and requiresexpert knowledge to be solved. In that case, it will also assessspecific domain knowledge rather than a latent problem-sol-ving ability (e.g., it will implicitly contain a measure of med-ical knowledge on a specific topic). This interaction betweenknowledge, ability, and problem-solving performancehas been described in the Elshout-Raaheim-Hypothesis2021 Hogrefe Publishing European Journal of Psychological Assessment (2021), 37(6), 419–422https://doi.org/10.1027/1015-5759/a000686 https://econtent.hogrefe.com/doi/pdf/10.1027/1015-5759/a000686 - Matthias Stadler <matthias.stadler@lmu.de> - Thursday, December 09, 2021 7:35:13 AM - IP Address:138.246.3.172 (Elshout, 1987; Raaheim, 1988; see also Leutner, 2002;Weise et al., 2020). The hypothesis describes a non-linearmoderation of the relation between ability and performanceby applicable knowledge. If a participant knows how to solvethe problem in a simulation, the problem becomes a meretask and does not require any ability.On the other hand, if the participant does not have suffi-cient knowledge to interact with the problem, they have noopportunity to use their ability (i.e., mere guessing). In theexample introduced above, the same simulation could betrivial to a medical expert but virtually unsolvable to anovice. Therefore, a simulation intended to assess anylatent ability should require just enough domain knowledgeso the target population will interact systematically with thesimulation without knowing the correct solution at the sim-ulation’s onset.What Is Measured?Making sure the simulation assesses the intended constructrequires careful item construction. The principles we wantto stress in this Editorial are not unique to simulation-basedassessment (see Ziegler, 2014) but become particularlyapparent in these cases where validity and authenticityneed to be balanced. In particular, we want to focus on fouraspects of assessment development: (1) clear definitions ofthe targeted construct and (2) the targeted population aswell as (3) theoretical considerations of adequate difficulty,and (4) required number of simulations.First, there should be a clear definition of the constructthe assessment intends to measure. This call for the defini-tion of constructs may seem trivial, but many simulationsprioritize authentic representation of real-life problem sce-narios over psychometric quality and psychological theory(Codreanu et al., 2020; Schuwirth & van der Vleuten,2003). Next to a theoretical description of the constructitself should entail a description of the construct’s nomolog-ical network (i.e., the construct’s relation to observables anddifferent theoretical constructs; Cronabch & Meehl, 1955).For research on simulations submitted to EJPA, we expecta concise definition of the construct, as with all researchsubmitted to EJPA, but will specifically focus on thetrade-off between authenticity and validity.Second, because of the non-linear interaction betweendomain knowledge, ability, and problem-solving perfor-mance described above, the target population needs to beclearly defined regarding their expected level of domainknowledge. The intended target population is, thus, closelylinked to the intended use of the simulation (Ziegler, 2014).A simulation that provides a good assessment for novicesmay not apply to experts or vice versa. When submittingresearch to EJPA, please state the intended use of your sim-ulation, the target population, and potential issues withusing the simulation outside of the target population. Itshould also be evident that the samples used to constructthe test must be derived from the targeted population.Third, the concept of adequacy for the target populationleads directly to the question of item difficulty. Ideally,researchers have a theoretical model of what features oftheir items influence difficulty (Gierl & Haladyna, 2012).This model should include domain-specific aspects in a sim-ulation emulating real-world problem-solving scenariossuch as domain knowledge and domain-general aspects(Schuwirth & van der Vleuten, 2003). These features of sim-ulations are theoretically orthogonal as it is possible to makea problem more difficult by increasing domain-specific fea-tures independently of domain-general features. Applied tothe example of the medical simulation introduced above,the difficulty could be increased by using rare or specific dis-eases (Norman et al., 2006) or by adding complexity byincreasing the number of variables to consider or thedynamics of the simulation (e.g., Stadler et al., 2016). Thesefeatures are generally independent of each other, andresearchers should theoretically describe their contributionto task difficulty before constructing the simulations. A suf-ficiently elaborated theoretical model can be tested empiri-cally using item-response approaches such as the linearlogistic test model (Kubinger, 2009). Not every submissionof simulation-based assessment to EJPAwill need to provideempirical tests of the underlying item model. However,there should be theoretical considerations as to what fea-tures of the simulation contribute to item difficulty andsome reasoning as to why the exact set of simulations waschosen beyond authentic content. Finally, testing a theoret-ical model of the impact had by the features of a simulationon item difficulty requires employing a whole set of system-atically constructed simulations that assess the same ability,which leads to the final aspect that we want to emphasize:the required amount of different (and at least somewhatindependent) simulations in an assessment. This is anothertrade-off situation in which validity can easily be overshad-owed by convenience. Developing authentic simulations isoften very laborious and costly for the test developer, andcomplex simulations may also take test-takers more timeto complete. For these reasons, researchers are often limitedin the number of simulations they can include in theirassessment. It is essential to consider that each simulationrepresents a single item in a test providing just a single indi-cator of the latent ability of interest. Even if multiple behav-ior indicators are extracted from a simulation, these arehighly interdependent. The dependency should beaccounted for statistically by applying hierarchical modelsEuropean Journal of Psychological Assessment (2021), 37(6), 419–422 2021 Hogrefe Publishing420 Editorial https://econtent.hogrefe.com/doi/pdf/10.1027/1015-5759/a000686 - Matthias Stadler <matthias.stadler@lmu.de> - Thursday, December 09, 2021 7:35:13 AM - IP Address:138.246.3.172 (e.g., von Davier et al., 2017) or aggregating all indicators toa single performance score.Using a small number of simulations (or even just a sin-gle simulation) to assess a latent construct may lead tounreliable results. Especially highly context-rich simulationsare likely to have considerable measurement error. Thecontexts of the various simulations will trigger differentexpectations in participants resulting in different answeringbehaviors and differential item functioning (Holland &Wainer, 2012). Examples for this are plentiful in the earlydays of assessing complex problem solving using simula-tions (Stadler et al., 2015). These first simulations emulatedvery complex problems (such as running a whole town)consisting of hundreds of variables and requiring multipledays of testing (Doerner, 1980). Correspondingly, theyemployed only one unique simulation with questionablepsychometric quality (Süß et al., 1993).In deciding how many simulations to use for an assess-ment, we encourage researchers to apply the same criteriaas any other psychometric measure. In line with the previ-ous statements on simulation difficulty, this also includesan adequate distribution of difficulty across the assessmentto allow for discrimination between groups of low, average,and high performers (Greiff et al., 2015). For every submis-sion to EJPA, we will critically evaluate whether the authorsprovide convincing arguments for the length of their assess-ment instrument.ConclusionSimulations allow researchers to assess essential abilitiesunder authentic conditions yielding the potential toincrease, among other, construct coverage. Moreover, thecomplexity of some abilities (such as complex or collabora-tive problem solving) cannot be realized at all without sim-ulations. In this Editorial, we want to encourage authors tocontinue exploring the use of simulations as assessmenttools but also make them aware of potential pitfalls thatcome with them. Most importantly, researchers need toadhere to the standards of psychometric quality whenusing simulations in assessment. Suppose the simulationrequires too much time or is too costly to employ sufficientdifferent simulations for an acceptably reliable measure-ment, it may not be the adequate approach to measurethe intended construct. However, if researchers use simula-tions appropriately, they represent potent tools that allowfor deep insights into psychological processes that are notaccessible to psychometric assessment. So please submityour research on simulations in assessment to EJPA, includ-ing adequate sections on all four issues raised in thisEditorial.ReferencesBrehmer, B., & Dörner, D. (1993). Experiments with computer-simulated microworlds: Escaping both the narrow straits of thelaboratory and the deep blue sea of the field study. Computersin Human Behavior, 9(2–3), 171–184. https://doi.org/10.1016/0747-5632(93)90005-dChernikova, O., Heitzmann, N., Stadler, M., Holzberger, D., Seidel,T., & Fischer, F. (2020). Simulation-based learning in highereducation: A meta-analysis. Review of Educational Research,90(4), 499–541. https://doi.org/10.3102/0034654320933544Cipresso, P., Pedroli, E., Serino, S., Semonella, M., Tuena, C.,Colombo, D., Pallavicini, F., & Riva, G. (2018). Assessment ofunilateral spatial neglect using a free mobile application forItalian clinicians. Frontiers in Psychology, 9, Article 2241.https://doi.org/10.3389/fpsyg.2018.02241Codreanu, E., Sommerhoff, D., Huber, S., Ufer, S., & Seidel, T. (2020).Between authenticity and cognitive demand: Finding a balance indesigning a video-based simulation in the context of mathemat-ics teacher education. Teaching and Teacher Education, 95,Article 103146. https://doi.org/10.1016/j.tate.2020.103146Cronabch, L. J., & Meehl, P. E. (1955). Construct validity inpsychological tests. Psychological Bulletin, 52(4), 281–302.https://doi.org/10.1037/h0040957de-Juan-Ripoll, C., Soler-Domínguez, J. L., Guixeres, J., Contero, M.,Álvarez Gutiérrez, N., & Alcañiz, M. (2018). Virtual reality as anew approach for risk taking assessment. Frontiers in Psychol-ogy, 9, Article 2532. https://doi.org/10.3389/fpsyg.2018.02532Doerner, D. (1980). On the difficulties people have in dealing withcomplexity. Simulation & Games, 11(1), 87–106. https://doi.org/10.1177/104687818001100108Drews, F. A., Syroid, N., Agutter, J., Strayer, D. L., & Westenskow,D. R. (2006). Drug delivery as control task: Improving perfor-mance in a common anesthetic task. Human Factors, 48(1),85–94. https://doi.org/10.1518/001872006776412216Elshout, J. J. (1987). Problem solving and education. In E. deCorte, H. Lodewijks, & R. Parmentier (Eds.), Learning andinstruction (pp. 259–273). Pergamon.Gierl, M. J., & Haladyna, T. M. (2012). Automatic item generation.Routledge. https://doi.org/10.4324/9780203803912Goldhammer, F., Scherer, R., & Greiff, S. (2019). Editorial:Advancements in technology-based assessment: Emergingitem formats, test designs, and data sources. Frontiers inPsychology, 10, Article 3047. https://doi.org/10.3389/fpsyg.2019.03047Greiff, S. (2017). The field of psychological assessment: Where itstands and where it’s going – A personal analysis of foci, gaps,and implications for EJPA. European Journal of PsychologicalAssessment, 33(1), 1–4. https://doi.org/10.1027/1015-5759/a000412Greiff, S., Fischer, A., Stadler, M., & Wüstenberg, S. (2015).Assessing complex problem-solving skills with multiple com-plex systems. Thinking & Reasoning, 21(3), 356–382.https://doi.org/10.1080/13546783.2014.989263Han, Z., He, Q., & von Davier, M. (2019). Predictive featuregeneration and selection using process data from PISA inter-active problem-solving items: An application of random forests.Frontiers in Psychology, 10, Article 2461. https://doi.org/10.3389/fpsyg.2019.02461Holland, P. W., & Wainer, H. (2012). Differential item functioning.Routledge. https://doi.org/10.4324/9780203357811Kubinger, K. D. (2009). Applications of the linear logistic testmodel in psychometric research. Educational and PsychologicalMeasurement, 69(2), 232–244. https://doi.org/10.1177/00131644083220212021 Hogrefe Publishing European Journal of Psychological Assessment (2021), 37(6), 419–422Editorial 421 https://econtent.hogrefe.com/doi/pdf/10.1027/1015-5759/a000686 - Matthias Stadler <matthias.stadler@lmu.de> - Thursday, December 09, 2021 7:35:13 AM - IP Address:138.246.3.172 Leutner, D. (2002). The fuzzy relationship of intelligence andproblem solving in computer simulations. Computers in HumanBehavior, 18(6), 685–697. https://doi.org/10.1016/s0747-5632(02)00024-9Norman, G., Bordage, G., Page, G., & Keane, D. (2006). Howspecific is case specificity? Medical Education, 40(7), 618–623.https://doi.org/10.1111/j.1365-2929.2006.02511.xRaaheim, K. (1988). Intelligence and task novelty. In R. J.Sternberg (Ed.), Advances in the psychology of human intelli-gence (Vol. 4, pp. 73–97). Erlbaum.Schuwirth, L. W. T., & van der Vleuten, C. P. M. (2003). The useof clinical simulations in assessment. Medical Education,37(Suppl 1), 65–71. https://doi.org/10.1046/j.1365-2923.37.s1.8.xStadler, M., Becker, N., Gödker, M., Leutner, D., & Greiff, S. (2015).Complex problem solving and intelligence: A meta-analysis.Intelligence, 53, 92–101. https://doi.org/10.1016/j.intell.2015.09.005Stadler, M., Niepel, C., & Greiff, S. (2016). Easily too difficult:Estimating item difficulty in computer simulated microworlds.Computers in Human Behavior, 65, 100–106. https://doi.org/10.1016/j.chb.2016.08.025Süß, H.-M., Kersting, M., & Oberauer, K. (1993). Zur Vorhersagevon Steuerungsleistungen an Computersimulierten Systemendurch Wissen und Intelligenz [On the predictability of controlperformance on computer-simulated systems by knowledgeand intelligence]. Zeitschrift für Differentielle und DiagnostischePsychologie, 14(3), 189–203.von Davier, A. A., Zhu, M., & Kyllonen, P. C. (2017). Innovativeassessment of collaboration: Methodology of educational mea-surement and assessment. Springer.Weise, J. J., Greiff, S., & Sparfeldt, J. R. (2020). The moderatingeffect of prior knowledge on the relationship between intelli-gence and complex problem solving – testing the Elshout-Raaheim hypothesis. Intelligence, 83, Article 101502.https://doi.org/10.1016/j.intell.2020.101502Ziegler, M. (2014). Stop and state your intentions. EuropeanJournal of Psychological Assessment, 30(4), 239–242.https://doi.org/10.1027/1015-5759/a000228Published online December 6, 2021FundingThis research was supported by a grant from the DeutscheForschungsgemeinschaft DFG (COSIMA; DFG-Forschungsgruppe2385).Matthias StadlerLudwig-Maximilians-University of MunichLeopoldstr. 1380802 MunichGermanymatthias.stadler@lmu.deEuropean Journal of Psychological Assessment (2021), 37(6), 419–422 2021 Hogrefe Publishing422 Editorial https://econtent.hogrefe.com/doi/pdf/10.1027/1015-5759/a000686 - Matthias Stadler <matthias.stadler@lmu.de> - Thursday, December 09, 2021 7:35:13 AM - IP Address:138.246.3.172 

Validly authentic. Some recommendations to researchers using simulations in psychological assessment. European

https://orbilu.uni.lu/bitstream/10993/50950/1/Stadlerteal2021_EJPA.pdf

Validly authentic. Some recommendations to researchers using simulations in psychological assessment. European

Abstract

Similar works

Full text

Available Versions

Open Repository and Bibliography - Luxembourg