Faking on a situational judgment test in a medical school selection setting: Effect of different scoring methods?

We examined the occurrence of faking on a rating situational judgment test (SJT) by comparing SJT scores and response styles of the same individuals across two natu‐ rally occurring situations. An SJT for medical school selection was administered twice to the same group of applicants (N = 317) under low‐stakes (T1) and high‐stakes (T2) circumstances. The SJT was scored using three different methods that were differ‐ entially affected by response tendencies. Applicants used significantly more extreme responding on T2 than T1. Faking (higher SJT score on T2) was only observed for scoring methods that controlled for response tendencies. Scoring methods that do not control for response tendencies introduce systematic error into the SJT score, which may lead to inaccurate conclusions about the existence of faking

Subgroup differences in situational judgment test scores: Evidence from large applicant samples

To promote diversity in organizations it is important to have accurate knowledge about subgroup differences associated with selection procedures. However, current estimates of subgroup differences in situational judgment tests (SJTs) are overwhelmingly based on range‐restricted incumbent samples that are downwardly biased. This study provides much‐needed applicant level estimates of SJT subgroup differences (N = 37,530). As a key finding, Black‐White differences (d = 0.66) were higher than in incumbent samples (d = 0.38)

Greek community needs assessment: Reducing the negative impact of alcohol and drugs

Participants representing Clemson\u27s Greek community have designed a research project that aims to define the problems associated with alcohol and drug misuse in the Greek community at Clemson University and to implement action steps based on sound evidence to mitigate the negative consequences associated with that misuse. The team is made up of Greek student leaders who are passionate about making a difference in their community and ultimately creating a plan to reduce alcohol and drug abuse among members. The team has conducted IRB and National PanHellenic approved focus groups and is in the process of coding the data. This poster reflects a preliminary analysis of that data

Situational judgement tests in medical education and training: Research, theory and practice: AMEE Guide No. 100

Why use SJTs? Traditionally, selection into medical education professions has focused primarily upon academic ability alone. This approach has been questioned more recently, as although academic attainment predicts performance early in training, research shows it has less predictive power for demonstrating competence in postgraduate clinical practice. Such evidence, coupled with an increasing focus on individuals working in healthcare roles displaying the core values of compassionate care, benevolence and respect, illustrates that individuals should be selected on attributes other than academic ability alone. Moreover, there are mounting calls to widen access to medicine, to ensure that selection methods do not unfairly disadvantage individuals from specific groups (e.g. regarding ethnicity or socio-economic status), so that the future workforce adequately represents society as a whole. These drivers necessitate a method of assessment that allows individuals to be selected on important non-academic attributes that are desirable in healthcare professionals, in a fair, reliable and valid way. [Table: see text] What are SJTs? Situational judgement tests (SJTs) are tests used to assess individuals' reactions to a number of hypothetical role-relevant scenarios, which reflect situations candidates are likely to encounter in the target role. These scenarios are based on a detailed analysis of the role and should be developed in collaboration with subject matter experts, in order to accurately assess the key attributes that are associated with competent performance. From a theoretical perspective, SJTs are believed to measure prosocial Implicit Trait Policies (ITPs), which are shaped by socialisation processes that teach the utility of expressing certain traits in different settings such as agreeable expressions (e.g. helping others in need), or disagreeable actions (e.g. advancing ones own interest at others, expense). Are SJTs reliable, valid and fair? Several studies, including good quality meta-analytic and longitudinal research, consistently show that SJTs used in many different occupational groups are reliable and valid. Although there is over 40 years of research evidence available on SJTs, it is only within the past 10 years that SJTs have been used for recruitment into medicine. Specifically, evidence consistently shows that SJTs used in medical selection have good reliability, and predict performance across a range of medical professions, including performance in general practice, in early years (foundation training as a junior doctor) and for medical school admissions. In addition, SJTs have been found to have significant added value (incremental validity) over and above other selection methods such as knowledge tests, measures of cognitive ability, personality tests and application forms. Regarding differential attainment, generally SJTs have been found to have lower adverse impact compared to other selection methods, such as cognitive ability tests. SJTs have the benefit of being appropriate both for use in selection where candidates are novices (i.e. have no prior role experience or knowledge such as in medical school admissions) as well as settings where candidates have substantial job knowledge and specific experience (as in postgraduate recruitment for more senior roles). An SJT specification (e.g. scenario content, response instructions and format) may differ depending on the level of job knowledge required. Research consistently shows that SJTs are usually found to be positively received by candidates compared to other selection tests such as cognitive ability and personality tests. Practically, SJTs are difficult to design effectively, and significant expertise is required to build a reliable and valid SJT. Once designed however, SJTs are cost efficient to administer to large numbers of candidates compared to other tests of non-academic attributes (e.g. personal statements, structured interviews), as they are standardised and can be computer-delivered and machine-marked

Situational judgement tests for selection

When situational judgement tests (SJTs) began to regain popularity among the scientific community in the 1990s, there was an implicit notion that they captured contextdependent knowledge. In fact, the term ‘situational judgement’ carries the connotation of test‐takers’ responses being more effective when they consider the specifics of the situation. In recent years another perspective has emerged, which views SJTs as capturing relatively context‐independent knowledge (or general domain knowledge; Motowidlo, Crook, Kell & Naemi, 2009; Motowidlo, Hooper & Jackson, 2006a). Although SJTs and their items will often fall somewhere between these two perspectives, we posit in this chapter that it might be useful to distinguish between them. So far, there has been no review of the SJT literature in terms of these two approaches. This is understandable, as over the years the two perspectives have emerged alongside each other. Therefore, the aim of this chapter is to review SJT research according to these two approaches

Scoring method of a Situational Judgment Test:influence on internal consistency reliability, adverse impact and correlation with personality?

textabstractSituational Judgment Tests (SJTs) are increasingly used for medical school selection. Scoring an SJT is more complicated than scoring a knowledge test, because there are no objectively correct answers. The scoring method of an SJT may influence the construct and concurrent validity and the adverse impact with respect to non-traditional students. Previous research has compared only a small number of scoring methods and has not studied the effect of scoring method on internal consistency reliability. This study compared 28 different scoring methods for a rating SJT on internal consistency reliability, adverse impact and correlation with personality. The scoring methods varied on four aspects: the way of controlling for systematic error, and the type of reference group, distance and central tendency statistic. All scoring methods were applied to a previously validated integrity-based SJT, administered to 931 medical school applicants. Internal consistency reliability varied between .33 and .73, which is likely explained by the dependence of coefficient alpha on the total score variance. All scoring methods led to significantly higher scores for the ethnic majority than for the non-Western minorities, with effect sizes ranging from 0.48 to 0.66. Eighteen scoring methods showed a significant small positive correlation with agreeableness. Four scoring methods showed a significant small positive correlation with conscientiousness. The way of controlling for systematic error was the most influential scoring method aspect. These results suggest that the increased use of SJTs for selection into medical school must be accompanied by a thorough examination of the scoring method to be used

Challenge clusters facing LCA in environmental decision-making—what we can learn from biofuels

Purpose Bioenergy is increasingly used to help meet greenhouse gas (GHG) and renewable energy targets. However, bioenergy’s sustainability has been questioned, resulting in increasing use of life cycle assessment (LCA). Bioenergy systems are global and complex, and market forces can result in significant changes, relevant to LCA and policy. The goal of this paper is to illustrate the complexities associated with LCA, with particular focus on bioenergy and associated policy development, so that its use can more effectively inform policymakers. Methods The review is based on the results from a series of workshops focused on bioenergy life cycle assessment. Expert submissions were compiled and categorized within the first two workshops. Over 100 issues emerged. Accounting for redundancies and close similarities in the list, this reduced to around 60 challenges, many of which are deeply interrelated. Some of these issues were then explored further at a policyfacing workshop in London, UK. The authors applied a rigorous approach to categorize the challenges identified to be at the intersection of biofuels/bioenergy LCA and policy. Results and discussion The credibility of LCA is core to its use in policy. Even LCAs that comply with ISO standards and policy and regulatory instruments leave a great deal of scope for interpretation and flexibility. Within the bioenergy sector, this has led to frustration and at times a lack of obvious direction. This paper identifies the main challenge clusters: overarching issues, application and practice and value and ethical judgments. Many of these are reflective of the transition from application of LCA to assess individual products or systems to the wider approach that is becoming more common. Uncertainty in impact assessment strongly influences planning and compliance due to challenges in assigning accountability, and communicating the inherent complexity and uncertainty within bioenergy is becoming of greater importance. Conclusions The emergence of LCA in bioenergy governance is particularly significant because other sectors are likely to transition to similar governance models. LCA is being stretched to accommodate complex and broad policy-relevant questions, seeking to incorporate externalities that have major implications for long-term sustainability. As policy increasingly relies on LCA, the strains placed on the methodology are becoming both clearer and impedimentary. The implications for energy policy, and in particular bioenergy, are large

Removing situation descriptions from situational judgment test items: Does the impact differ for video-based versus text-based formats?

Recent research has shown that many text-based situational judgment test (SJT) items can be solved even when the situational descriptions in the item stems are not presented to test takers. This finding challenges the traditional view of SJTs as low-fidelity simulations that rely on 'situational' (context-dependent) judgment. However, media richness theory and construal level theory suggest that situation descriptions presented in a richer and more concrete format (video format) will reduce uncertainty about inherent requirements and facilitate the perception that the situation is taking place in the here and now. Therefore, we hypothesized that situational judgment would be more important in video situation descriptions than in text situation descriptions. We adapted a leadership SJT to realize a 3 (situation description in the item stem: video vs. text vs. none) x 2 (response format: video response options vs. text response options) between-subjects design (N = 279). Participants were randomly assigned to one of the six conditions. The removal of video-based situation descriptions in item stems led to an equivalent decrease in SJT scores as the removal of text-based situation descriptions in item stems (video-based version: Cohen's d = 0.535 vs. text-based version: Cohen's d = 0.531). SJT scores were also contingent on the presentation format of both situation descriptions and response options: The highest scores were observed when situation descriptions and response options were presented in the same format. Implications for SJT theory and research are discussed. Practitioner points The presentation format did not moderate the effect of omitting situation descriptions in SJTs - that is, the context dependency of SJT performance did not increase when the SJT was administered in a video-based rather than a text-based format. The elimination of situation descriptions in item stems had a medium effect on overall test scores: SJT scores were significant lower without situation descriptions in comparison to SJT scores with situation descriptions (video-based version: Cohen's d = 0.535 vs. text-based version: Cohen's d = 0.531). It is important to match the stimulus and response formats in SJTs

Why Do Situational Interviews Predict Performance? Is it Saying How You Would Behave or Knowing How You Should Behave?

Purpose: The present study examined two theoretical explanations for why situational interviews predict work-related performance, namely (a) that they are measures of interviewees’ behavioral intentions or (b) that they are measures of interviewees’ ability to correctly decipher situational demands. Design/Methodology/Approach: We tested these explanations with 101 students, who participated in a 2-day selection simulation. Findings: In line with the first explanation, there was considerable similarity between what participants said they would do and their actual behavior in corresponding work-related situations. However, the underlying postulated mechanism was not supported by the data. In line with the second explanation, participants’ ability to correctly decipher situational demands was related to performance in both the interview and work-related situations. Furthermore, the relationship between the interview and performance in the work-related situations was partially explained by this ability to decipher situational demands. Implications: Assessing interviewees’ ability to identify criteria might be of additional value for making selection decisions, particularly for jobs where it is essential to assess situational demands. Originality/Value: The present study made an effort to open the ‘black box’ of situational interview validity by examining two explanations for their validity. The results provided only moderate support for the first explanation. However, the second explanation was fully supported by these results