Scoring method of a Situational Judgment Test:influence on internal consistency reliability, adverse impact and correlation with personality?

textabstractSituational Judgment Tests (SJTs) are increasingly used for medical school selection. Scoring an SJT is more complicated than scoring a knowledge test, because there are no objectively correct answers. The scoring method of an SJT may influence the construct and concurrent validity and the adverse impact with respect to non-traditional students. Previous research has compared only a small number of scoring methods and has not studied the effect of scoring method on internal consistency reliability. This study compared 28 different scoring methods for a rating SJT on internal consistency reliability, adverse impact and correlation with personality. The scoring methods varied on four aspects: the way of controlling for systematic error, and the type of reference group, distance and central tendency statistic. All scoring methods were applied to a previously validated integrity-based SJT, administered to 931 medical school applicants. Internal consistency reliability varied between .33 and .73, which is likely explained by the dependence of coefficient alpha on the total score variance. All scoring methods led to significantly higher scores for the ethnic majority than for the non-Western minorities, with effect sizes ranging from 0.48 to 0.66. Eighteen scoring methods showed a significant small positive correlation with agreeableness. Four scoring methods showed a significant small positive correlation with conscientiousness. The way of controlling for systematic error was the most influential scoring method aspect. These results suggest that the increased use of SJTs for selection into medical school must be accompanied by a thorough examination of the scoring method to be used

Why Do Situational Interviews Predict Performance? Is it Saying How You Would Behave or Knowing How You Should Behave?

Purpose: The present study examined two theoretical explanations for why situational interviews predict work-related performance, namely (a) that they are measures of interviewees’ behavioral intentions or (b) that they are measures of interviewees’ ability to correctly decipher situational demands. Design/Methodology/Approach: We tested these explanations with 101 students, who participated in a 2-day selection simulation. Findings: In line with the first explanation, there was considerable similarity between what participants said they would do and their actual behavior in corresponding work-related situations. However, the underlying postulated mechanism was not supported by the data. In line with the second explanation, participants’ ability to correctly decipher situational demands was related to performance in both the interview and work-related situations. Furthermore, the relationship between the interview and performance in the work-related situations was partially explained by this ability to decipher situational demands. Implications: Assessing interviewees’ ability to identify criteria might be of additional value for making selection decisions, particularly for jobs where it is essential to assess situational demands. Originality/Value: The present study made an effort to open the ‘black box’ of situational interview validity by examining two explanations for their validity. The results provided only moderate support for the first explanation. However, the second explanation was fully supported by these results

The role of gender in a smoking cessation intervention: a cluster randomized clinical trial

<p>Abstract</p> <p>Background</p> <p>The prevalence of smoking in Spain is high in both men and women. The aim of our study was to evaluate the role of gender in the effectiveness of a specific smoking cessation intervention conducted in Spain.</p> <p>Methods</p> <p>This study was a secondary analysis of a cluster randomized clinical trial in which the randomization unit was the Basic Care Unit (family physician and nurse who care for the same group of patients). The intervention consisted of a six-month period of implementing the recommendations of a Clinical Practice Guideline. A total of 2,937 current smokers at 82 Primary Care Centers in 13 different regions of Spain were included (2003-2005). The success rate was measured by a six-month continued abstinence rate at the one-year follow-up. A logistic mixed-effects regression model, taking Basic Care Units as random-effect parameter, was performed in order to analyze gender as a predictor of smoking cessation.</p> <p>Results</p> <p>At the one-year follow-up, the six-month continuous abstinence quit rate was 9.4% in men and 8.5% in women (p = 0.400). The logistic mixed-effects regression model showed that women did not have a higher odds of being an ex-smoker than men after the analysis was adjusted for confounders (OR adjusted = 0.9, 95% CI = 0.7-1.2).</p> <p>Conclusions</p> <p>Gender does not appear to be a predictor of smoking cessation at the one-year follow-up in individuals presenting at Primary Care Centers.</p> <p>ClinicalTrials.gov Identifier</p> <p><a href="http://www.clinicaltrials.gov/ct2/show/NCT00125905">NCT00125905</a>.</p

New approaches to selection system design in healthcare: The practical and theoretical relevance of a modular approach

This chapter presents a modular approach to healthcare selection system design. Contrary to the traditional holistic view on selection procedures, a modular approach highlights the components underlying selection procedures. Our framework identifies seven key design components of selection procedures (The stimulus format, contextualization, stimulus presentation consistency, the response format, response evaluation consistency, information source, and instructions) and reviews studies in the healthcare selection literature that compared the effect of these components on key selection outcomes. A modular approach allows (1) gaining insights into how the different components underlying selection procedures affect selection outcomes and (2) drawing conceptual similarities between components of different selection procedures. At a practical level, a modular approach permits developing a myriad of new selection procedures by "mixing and matching" different building blocks. We present two case studies and future research avenues to further illustrate these merits of a modular approach