Pushing the Limits for Judgmental Consistency: Comparing Random Weighting Schemes with Expert Judgments

Consistent use of information has been identified as a critical issue that can undermine expert predictions. Using three personnel assessment datasets, we conduct Monte Carlo simulations to compare the accuracy of expert judgements for predicting the job performance of managers against four different weighting schemes: consistent random weights, completely random weights, unit weights, and optimal weights. Expert accuracy fell within the completely random weight distribution in two samples and at the low end of the consistent random weight distribution in one sample. In other words, consistent random weights reliably outperformed expert judgment for hiring decisions across three datasets with a total sample size of 847. We see this as a call to develop decision making systems that help control consistency or to manage consistency by aggregating multiple expert judgments

Decoy Effects Improve Diversity Hiring

A growing literature demonstrates that when making choices among multiple options, decision makers are strongly influenced by the mere presence of additional options, even when those options are largely undesirable and are never actually selected. The effects of irrelevant options on decisions, often called decoy effects, have been observed in hiring and admissions decisions where the nature of a third candidate can radically shift preferences. In this study, we examine the influence of decoy effects on diversity hiring and extend research by examining choices with more than two organizational goals. Results indicate that the presence of a second candidate who meets diversity goals markedly increases how frequently decision makers indicate that they would make an offer to diverse candidates. This effect occurs even when decision makers must sacrifice some credential quality to obtain the diverse candidate. Overall, diverse candidates are more likely to receive offers when more than one diversity candidate is included in the finalist pool. The practical implications are clear: When a major organizational goal is to increase diversity, a policy that includes evaluating multiple diverse candidates in a final applicant slate should be considered

Faking and the Validity of Personality Tests: An Experimental Investigation Using Modern Forced Choice Measures

Despite the established validity of personality measures for personnel selection, their susceptibility to faking has been a persistent concern. However, the lack of studies that combine generalizability with experimental control makes it difficult to determine the effects of applicant faking. This study addressed this deficit in two ways. First, we compared a subtle incentive to fake with the explicit “fake-good” instructions used in most faking experiments. Second, we compared standard Likert scales to multidimensional forced choice (MFC) scales designed to resist deception, including more and less fakable versions of the same MFC inventory. MFC scales substantially reduced motivated score elevation but also appeared to elicit selective faking on work-relevant dimensions. Despite reducing the effectiveness of impression management attempts, MFC scales did not retain more validity than Likert scales when participants faked. However, results suggested that faking artificially bolstered the criterion-related validity of Likert scales while diminishing their construct validity

Individual differences and their measurement: A review of 100 years of research

This article reviews 100 years of research on individual differences and their measurement, with a focus on research published in the Journal of Applied Psychology. We focus on 3 major individual differences domains: (a) knowledge, skill, and ability, including both the cognitive and physical domains; (b) personality, including integrity, emotional intelligence, stable motivational attributes (e.g., achievement motivation, core self-evaluations), and creativity; and (c) vocational interests. For each domain, we describe the evolution of the domain across the years and highlight major theoretical, empirical, and methodological developments, including relationships between individual differences and variables such as job performance, job satisfaction, and career development. We conclude by discussing future directions for individual differences research. Trends in the literature include a growing focus on substantive issues rather than on the measurement of individual differences, a differentiation between constructs and measurement methods, and the use of innovative ways of assessing individual differences, such as simulations, other-reports, and implicit measures

Cognitive Skills and Critical Thinking

This presentation, by Nathan Kuncel of the University of Minnesota, was given during the second Workshop of Assessment of 21st Century Skills was held in May, 2011. In it, Kuncel discusses traditional cognitive skills, definitions of critical thinking, traditional examples of critical thinking skills, the validity of measures of those skills, and their implications

Measurement and Meaning of Critical Thinking

This presentation and accompanying paper by Nathan Kuncel was given to the Workshop on Assessment of 21st Century Skills, held in January 2011. Kuncel discusses definitions and examples of critical thinking skills and these questions: 1) What are 21st century cognitive skills? To what extent do they differ from each other and from general cognitive ability? What are the conceptual differences that are proposed to exist between these constructs? 2) What are the existing measures of these constructs, and to what extent do these existing measures match their conceptual specifications? 3) What are the relationships between the existing measures of these constructs

Academic performance, career potential, creativity and job performance; can one construct predict them all/ Kuncel

terdiri 2 artike

The process of attrition in pre-medical studies: A large-scale analysis across 102 schools.

The important but difficult choice of vocational trajectory often takes place in college, beginning with majoring in a subject and taking relevant coursework. Of all possible disciplines, pre-medical studies are often not a formally defined major but pursued by a substantial proportion of the college population. Understanding students' experiences with pre-med coursework is valuable and understudied, as most research on medical education focuses on the later medical school and residency. We examined the pattern and predictors of attrition at various milestones along the pre-med coursework track during college. Using a College Board dataset, we analyzed a sample of 15,442 students spanning 102 institutions who began their post-secondary education in years between 2006 and 2009. We examined whether students fulfilled the required coursework to remain eligible for medical schools at several milestones: 1) one semester of general chemistry, biology, physics, 2) two semesters of general chemistry, biology, physics, 3) one semester of organic chemistry, and 4) either the second semester of organic chemistry or one semester of biochemistry, and predictors of persistence at each milestone. Only 16.5% of students who intended to major in pre-med graduate college with the required coursework for medical schools. Attrition rates are highest initially but drop as students take more advanced courses. Predictors of persistence include academic preparedness before college (e.g., SAT scores, high school GPA) and college performance (e.g., grades in pre-med courses). Students who perform better academically both in high school and in college courses are more likely to remain eligible for medical school