König, Cornelius J.

Melchers, Klaus G.

RERO DOC Digital Library

It Is Not Yet Time to Dismiss Dimensionsin Assessment CentersKLAUS G. MELCHERS AND CORNELIUS J. KO¨NIGUniversita¨t Zu¨richPast research has well documented thatassessors’ postexercise dimension ratings(PEDRs) in an assessment center (AC) reflectthe different exercises much more than thedifferent dimensions. This is often under-stood as a problem for the construct validityof ACs. However, Lance (2008) does notconsider the construct validity of ACs to betroubling. On the basis of his description ofthe situation (which we consider to be veryaccurate), Lance instead either explicitly orimplicitly makes the following suggestions:(a) regard ACs as a collection of work sampletests, (b) treat only exercise variance in asses-sors’ PEDRs as relevant variance, and (c) giveup the search for design fixes that mightimprove construct validity. We would liketo challenge these three suggestions.Does Considering ACs as aCollection of Work Sample TestsChange the Problem?Lance’s suggestion to change the design ofACs toward task- or role-based ACs meanstreating them as collections of work sampletests. However, it has already been acknowl-edged in the past that it remains undeter-mined what work sample tests measure(e.g., Roth, Bobko, & McFarland, 2005). Intheir meta-analysis on work sample tests,Roth et al. even said that they ‘‘believe thatwork sample tests could benefit from a con-tinued focus on the constructs they assess’’(p. 1031). Thus, turning away from the seem-ingly troubling construct validity evidence inthe AC domain and rethinking ACs as a col-lection of work samples neither alters theproblem nor makes it go away. Instead,important questions remain: What is it thatsimulation exercises, such as those in ACs orwork sample tests, measure? To what degreedo these simulations measure the targeteddimensions identified in a previous job anal-ysis? And what leads to successful or lesssuccessful completion of these simulations?Lance ignores the fact that there isalready one answer to the question of whatleads to successful completion of simulationexercises: Candidates who are able to cor-rectly identify the criteria for success inthe different AC exercises (i.e., the targeteddimensions) manage to handle these simu-lations more successfully and perform bet-ter in the AC (Kleinmann, 1993; Preckel &Schu¨pbach, 2005). Furthermore, the degreeto which candidates have correctly identi-fied the targeted dimensions in an AC is pre-dictive (even after controlling for generalmental ability) of their performance in a dif-ferent situation (a selection interview; Ko¨nig,Melchers, Richter, Kleinmann, & Klehe, 2007).Thus, identifying the aspects (i.e., the behav-ioral dimensions) that are essential in a givensituation and directing one’s behavior towardCorrespondence concerning this article should beaddressed to Klaus G. Melchers. E-mail: k.melchers@psychologie.uzh.chAddress: Psychologisches Institut, Universita¨t Zu¨rich,Binzmu¨hlestrasse 14/12, CH-8050 Zu¨rich, SwitzerlandKlaus G. Melchers and Cornelius J. Ko¨nig, Psycho-logisches Institut, Universita¨t Zu¨richIndustrial and Organizational Psychology, 1 (2008), 125–127.Copyright ª 2008 Society for Industrial and Organizational Psychology. 1754-9426/08125https:/www.cambridge.org/core/terms. https://doi.org/10.1111/j.1754-9434.2007.00023.xDownloaded from https:/www.cambridge.org/core. University of Basel Library, on 30 May 2017 at 19:20:22, subject to the Cambridge Core terms of use, available atthese aspects help a candidate to successfullyhandle that situation.Taken together, considering ACs as col-lections of work samples constitutes onlya change of name but does not change theconstruct validity problem. Furthermore, tofocus on tasks and to abandon dimensionswould mean taking out the explicit acknowl-edgment of the criteria that are essential forsuccessful performance in an AC.Is Variance Related to DimensionFactors Irrelevant?Lance strongly argues in favor of acknowl-edging that exercise factors contain true var-iance. We do agree with this, but we arehesitant to follow the reverse conclusion thatdimension variance should be consideredirrelevant. This would mean throwing outthe baby with the bathwater.Lance and his collaborators have pro-vided convincing evidence that exercisefactors represent true variance and showsubstantial correlations with external mea-sures (e.g., Lance, Foster, Gentry, & Thoresen,2004). Similarly, it has been found thatassessors’ ratings mirror variations in candi-dates’ performance but that construct valid-ity evidence was only established whena candidate’s performance profile was con-sistent across situations but differentiatedbetween dimensions (e.g., Lievens, 2002).These findings are important because theyshow that assessor ratings are much moreaccurate than previously thought and cap-ture meaningful variations in candidates’performance and not just method bias.However, substantial amounts of vari-ance in Lievens’ (2001, 2002) studies, forexample, were not related to true perfor-mance variations. Thus, although assessors’ratings were more veridical than previouslythought, they were far from perfect, meaningthat it remains important to identify factorsthat influence the accuracy of such ratings.Lance does not mention either thatLievens’ (2001, 2002) studies only consid-ered very prototypical AC candidates. Morespecifically, Lievens did not assess whetherassessors’ PEDRs would capture existing vari-ations due to exercises as well as variationsdue to dimensions when both sources of vari-ation are present for a candidate but whenthe latter are smaller than the former. There-fore, Lievens’ studies cannot be taken as evi-dence that dimension variance is trivial intypical ACs and that dimensions can gener-ally be ignored.Finally, at leastone recent large-scaleevalu-ation of AC construct validity has found evi-dence that PEDRs reflect dimension factorsin addition to exercise factors (Bowler &Woehr, 2006). Even though PEDRs showedhigher loadings on exercise factors, dimen-sion factors also accounted for substantialvariance. Accordingly, Bowler and Woehrclearlyarguedagainst dismissing dimensions.In summary, although a large amount ofrelevant variance in PEDRs can be attributedto exercise factors, definitive evidence thatPEDRs do not contain any meaningfuldimension variance has not yet been pre-sented. Thus, for the time being, it seems pre-mature to suggest that dimension factors canbe dismissed.Can Possible Improvements inConstruct Measurement in ACsReally Be Ignored?Lance concludes that none of the attemptsfrom the past quarter century to improve ACdesign (i.e., the ‘‘design fixes’’) have come upwith strong results. Implicitly, he suggestsabandoning this line of research. We dis-agree because an important question regard-ing the effects of the suggested design fixeshas not yet been answered.According to the unitarian conceptuali-zation of validity, improvements in constructvalidity should lead to improvements incriterion-related validity. Even if the effectsof design fixes on construct validity are low,these fixes might still result in meaningfulimprovements in criterion-related validity.Unfortunately, only a single primary studyhas investigated this assumption for ACs sofar: A study by Schleicher, Day, Mayes, andRiggio (2002) found that so-called frame-of-reference rater training for assessors led toimprovements in construct validity and to126 K.G. Melchers and C.J. Ko¨nighttps:/www.cambridge.org/core/terms. https://doi.org/10.1111/j.1754-9434.2007.00023.xDownloaded from https:/www.cambridge.org/core. University of Basel Library, on 30 May 2017 at 19:20:22, subject to the Cambridge Core terms of use, available atimprovements in criterion-related validity.Further evidence in line with the unitarianconceptualization of validity can be indi-rectly inferred from two meta-analyses: Psy-chologists who serve as assessors providemore construct-valid ratings than managersaccording to Woehrand Arthur’s (2003)meta-analysis and more criterion-valid ratingsaccording to Gaugler, Rosenthal, Thornton,and Bentson’s (1987) meta-analysis.Thus, some promising results for designfixes can be found in the literature, and moreprimary research regarding the joint effectsof design fixes on construct and criterion val-idity is clearly needed—not less—to ensurethat we do not miss an important chance toimprove ACs.ReferencesBowler, M. C., & Woehr, D. J. (2006). A meta-analyticevaluation of the impact of dimension and exercisefactors on assessment center ratings. Journal ofApplied Psychology, 91, 1114–1124.Gaugler, B. B., Rosenthal, D. B., Thornton, G. C., &Bentson, C. (1987). Meta-analysis of assessmentcenter validity. Journal of Applied Psychology, 72,493–511.Kleinmann, M. (1993). Are rating dimensions in assess-ment centers transparent for participants? Conse-quences for criterion and construct validity. Journalof Applied Psychology, 78, 988–993.Ko¨nig, C. J., Melchers, K. G., Richter, G. M., Kleinmann,M., & Klehe, U.-C. (2007). The ability to identifycriteria in nontransparent selection procedures: Evi-dence from an assessment center and a structuredinterview. International Journal of Selection andAssessment, 15, 283–292.Lance, C. E., Foster, M. R., Gentry, W. A., & Thoresen, J.D. (2004). Assessor cognitive processes in an oper-ational assessment center. Journal of Applied Psy-chology, 89, 22–35.Lance, C. E. (2008). Why assessment centers do notwork the way they are supposed to. Industrial andOrganizational Psychology: Perspectives on Scienceand Practice, 1, 84–97.Lievens, F. (2001). Assessors and use of assessmentcentre dimensions: A fresh look at a troubling issue.Journal of Organizational Behavior, 22, 203–221.Lievens, F. (2002). Trying to understand the differentpieces of the construct validity puzzle of assessmentcenters: An examination of assessor and assesseeeffects. Journal of Applied Psychology, 87, 675–686.Preckel, D., & Schu¨pbach, H. (2005). Zusammenha¨ngezwischen rezeptiver Selbstdarstellungskompetenzund Leistung im Assessment Center [Correlationsbetween receptive self-presentation competenceand performance in an assessment center]. Zeit-schrift fu¨r Personalpsychologie, 4, 151–158.Roth, P. L., Bobko, P., & McFarland, L. A. (2005). A meta-analysis of work sample test validity: Updating andintegrating some classic literature. Personnel Psy-chology, 58, 1009–1037.Schleicher, D. J., Day, D. V., Mayes, B. T., & Riggio, R. E.(2002). A new frame for frame-of-reference training:Enhancing the construct validity of assessment cen-ters. Journal of Applied Psychology, 87, 735–746.Woehr, D. J., & Arthur, W. (2003). The construct-relatedvalidity of assessment center ratings: A review andmeta-analysis of the role of methodological factors.Journal of Management, 29, 231–258.Not yet time 127https:/www.cambridge.org/core/terms. https://doi.org/10.1111/j.1754-9434.2007.00023.xDownloaded from https:/www.cambridge.org/core. University of Basel Library, on 30 May 2017 at 19:20:22, subject to the Cambridge Core terms of use, available at

It Is Not Yet Time to Dismiss Dimensions in Assessment Centers

Abstract

Similar works

Full text

Available Versions

RERO DOC Digital Library