Using personal statements in college admissions::An investigation of gender bias and the effects of increased structure

Personal statements are among the most commonly used instruments in college admissions procedures. Yet, little research on their reliability, validity, and fairness exists. The first aim of this paper was to investigate hypotheses about adverse impact and underprediction for female applicants, which could result from lower tendencies to use agentic language compared to male applicants. Second, we examined if rating personal statements in a more structured manner would increase reliability and validity. Using personal statements (250 words) from a large cohort of applicants to an undergraduate psychology program at a Dutch University, we found no evidence for adverse impact for female applicants or more agentic language use by male applicants, and no relationship between agentic language use and personal statement ratings. In contrast, we found that personal statements of female applicants were rated slightly more positively than those of males. Exploratory analyses suggest that female applicants’ better writing skills might explain this difference. A more structured approach to rating personal statements yielded higher, but still only ‘moderate’ inter-rater reliability, and virtually identical, negligible predictive validity for first year GPA and dropout

Psychological and educational testing and decision making:The lack of knowledge dissemination in textbooks and test guidelines

When it comes to decision-making based on psychological and educational assessments, there is compelling evidence that statistical judgment is superior to holistic judgment. Yet, implementing this finding in practice has proven to be difficult for both academic and professional psychologists. Knowledge transfer from research findings to practitioners and other stakeholders in psychological assessment is a necessary condition to close this gap. To obtain insight into how academic specialists in psychological testing disseminate knowledge about research findings in this area, we investigated how textbooks on testing and guidelines on test use report on, or do not to report on, decision-making in psychological and educational assessment. Second, we discuss some commonly encountered misunderstandings, and third we argue for a broader and more in-depth dissemination of research findings on this topic in textbooks and test standards; to this end we provide some suggestions

A trial studying approach to predict college achievement

We argue that using trial studying is a reliable and valid way to select students for higher education. This method is based on a work sample approach often used in personnel selection contexts. We discuss that this method has predictive validity for study success, has high acceptance by stakeholders, and measures self-regulation in a high-stakes testing context that cannot be measured through self-report questionnaires. We suggest further research to implement this method to select students

Using narratives and numbers in performance prediction:Attitudes, confidence and validity

In a preregistered prediction-task experiment, we investigated the effect of narrative versus quantified information on decision-maker perceptions, confidence, predictor weighting, and predictive accuracy when making performance predictions. We also investigated the effect of who quantifies information (the decision maker or someone else). As expected, we found higher perceived informativeness and use intentions for narrative than quantified information. Information presented narratively was also weighted somewhat more heavily than quantified information. Using quantitative information quantified by decision makers themselves yielded higher perceived autonomy and use intentions than quantitative information quantified by someone else. However, no differences in prediction confidence were found and self- and other-produced quantifications received identical weight. Moreover, unexpectedly, differences in weighting did not translate to differences in predictive accuracy

Implementing evidence-based assessment and selection in organizations:A review and an agenda for future research

In personnel- and educational selection, a substantial gap exists between research and practice, since evidence-based assessment instruments and decision-making procedures are underutilized. We provide an overview of studies that investigated interventions to encourage the use of evidence-based assessment methods, or factors related to their use. The most promising studies were grounded in self-determination theory. Training and autonomy in the design of evidence-based assessment methods were positively related to their use, while negative stakeholder perceptions decreased practitioners’ intentions to use evidence-based assessment methods. Use of evidence-based decision-making procedures was positively related to access to such procedures, information to use it, and autonomy over the procedure, but negatively related to receiving outcome feedback. A review of the professional selection literature showed that the implementation of evidence-based assessment was hardly discussed. We conclude with an agenda for future research on encouraging evidence-based assessment practice