Underlying Psychological Processes of Situational Judgment Tests: Towards a Theory-Driven Integration of Person-Situation Interactions

In recent years, more and more psychological assessments aimed at capturing interactions between the person and situations. Situational Judgment Tests (SJTs) are built on a similar premise, as they were designed as low-fidelity simulations of situations. These tests incorporate short situation descriptions with several behavioral response options. However, the validity and underlying psychological processes of SJTs generally remained subject to debate as a growing body of research suggested that SJTs may reflect context-independent measures. Within this debate, other scholars argued in favor of the relevance of person-situation processes for SJT responses. So far, sufficient evidence that unravels the true underlying processes of SJTs is missing. This dissertation aims at closing this gap and at contributing to a deeper understanding of SJTs as psychological as-sessment methods. Four empirical research papers provide theory-driven insights on context-independent and person-situation processes as potential determinants of SJT responses. First, the construct-related validity of Implicit Trait Policies is examined and therefore the notion of SJTs as context-independent measures. Next, situation construal (i.e., the perception of situations), and processes postulated by Trait Activation Theory are considered as relevant theoretical underpinnings for SJTs. Results overall supported the relevance of person-situation interactions as underlying processes and particularly challenged SJTs as measures of Implicit Trait Policies. Especially situation construal explained SJT responses consistently across three studies. However, the results also showed that not situation descriptions but response options were often crucial for relevant person-situation processes as captured in SJT response. This lack of impact of situation descriptions also potentially limited the explanatory power of Trait Activation Theory in the context of SJT items. The results are discussed in regard to the debate about underlying processes of SJT responses. All in all, these studies raise the question whether key design features of common SJTs (i.e. situation descriptions and response options) are optimally developed for the assessment of person-situation interactions. The final paper of this dissertation introduces Standardized State Assessment as narrower and theory-driven methodological framework for the assessment of psychological states in hypothetical situations. Limitations of this dissertation, as well as implications for research and practice of psychological assessments based on situation descriptions are discussed.Die Berücksichtigung psychologischer Prozesse, die die Interaktion zwischen Personeneigenschaften und Situationen widerspiegeln, hat in den letzten Jahren für die psychologische Diagnostik an Bedeutung gewonnen. Dieser Prozess zeigt sich auch in Situational Judgment Tests (SJTs), die ursprünglich als simulationsbasiertes Verfahren entwickelt wurden. Diese Tests enthalten kurze Situationsbeschreibungen und mehrere verhaltensbasierte Antwortoptionen. Die Validität und die zugrundeliegenden psychologischen Prozesse von SJTs sind bislang allerdings nicht abschließend geklärt. Insbesondere neure Studien legen nahe, dass SJTs kontextunabhängige Messungen repräsentieren. Gleichzeitig existieren mehrere Argumente, die für situationsabhängige Prozesse in SJTs sprechen. Bislang fehlen jedoch ausführliche und abschließende Untersuchungen dieser Prozesse. Diese Dissertation möchte diese Lücke schließen und zu einem tieferen Verständnis von SJTs als Methode der psychologischen Diagnostik beitragen. Anhand von vier empirischen Artikeln werden theoriegeleitete Annahmen über kontext- und situationsabhängige Prozesse, die SJTs zugrunde liegen könnten, untersucht. Zunächst steht die Konstruktvalidität von Implicit Trait Policies im Vordergrund, die als erklärendes Konstrukt für SJTs als kontextunabhängige Messungen vorgebracht wurden. Weiterhin werden die Situationswahrnehmung und zentrale Aspekte der Trait Activation Theory als relevantes theoretisches Gerüst für SJTs untersucht. Die Ergebnisse unterstützen insgesamt die Relevanz situationsabhängiger Prozesse für SJTs und Zweifeln insbesondere an der Validität von Implicit Trait Policies. Vor allem die Situationswahrnehmung von SJT Items konnte das Antwortverhalten konsistent über drei Studien hinweg vorhersagen. Allerdings zeigte sich auch, dass hauptsächlich Antwortoptionen und nicht Situationsbeschreibungen entscheidend für situationsbasierte Prozesse in SJTs sind. Dies könnte auch die fehlende Relevanz der Trait Activation Theory für SJTs erklären. Die Ergebnisse werden im Kontext der Debatte über zugrundeliegende Prozesse von SJTs betrachtet. Insgesamt werfen die Ergebnisse die Frage auf, ob bisherige Konstruktionsweisen von SJTs (d.h. Situationsbeschreibungen und Antwortoptionen) eine optimale Erfassung von Interaktionen zwischen Personeneigenschaften und Situationen ermöglicht. Der letzte Artikel dieser Dissertation schlägt Standardized State Assessment als enger gefasstes und theoriegeleitetes, methodisches Modell für die Messung psychologischer Momentanzustände vor. Einschränkungen dieser Dissertation, sowie auch Konsequenzen für die Anwendung von und Forschung über psychologische Diagnostik mittels Situationsbeschreibungen werden diskutiert

Is it all in the eye of the beholder? The importance of situation construal for situational judgment test performance

German Research Foundatio

The importance of situation construal for situational judgment test performance

Recent research challenges the importance of situation descriptions for situational judgment test (SJT) performance. This study contributes to resolving the ongoing debate on whether SJTs are situational measures, by incorporating findings on person × situation interactions into SJT research. Specifically, across three studies (NTotal = 1,239), we first tested whether situation construal (i.e., the individual perception of situations in SJTs) predicts responses to SJT items. Second, we assessed whether the relevance of situation construal for SJT performance depends on test elements (i.e., situation descriptions and response options) and item features (i.e., description‐dependent vs. description‐independent SJT items). Lastly, we determined whether situation construal has incremental validity for job‐related criteria over and above SJT performance. The results showed that, for most SJT items, situation construal significantly contributed to SJT performance, even if only response options were available. This was also true for SJT items that are significantly more difficult to solve when situation descriptions are omitted (i.e., description‐dependent SJT items). Finally, situation construal explained variance in relevant criteria over and above SJT performance. Despite recent efforts to reconceptualize SJTs, our results suggest that they can still be viewed as situational measures. However, situation descriptions may be less crucial for these underlying situational processes. Theoretical and practical implications are discussed

Effects of situation descriptions on the construct-related validity of construct-driven situational judgment tests

All data and code are available on the Open Science Framework https://osf.io/py6cm/</p

The role of situations in situational judgment tests: Effects on construct saturation, predictive validity, and applicant perceptions

Recent theorizing and empirical evidence suggesting that Situational Judgment Tests (SDI's) are more context-independent than previously thought has sparked a debate about the role of situation descriptions in S.M. To contribute to this debate and add to our understanding of how SJTs work, this article conceptually embeds SJT performance in a situation construal model and examines the effects of situation descriptions on the construct saturation and predictive validity of SJT scores, as well as on applicant perceptions. Across two studies (N = 1,092 and 578) and different Sri's, personality and cognitive ability were equally important determinants of SJT performance regardless of whether situation descriptions were presented or omitted. The effects of removing situation descriptions on the criterion-related validity of SJT scores differed depending on the breadth of the criteria. For predicting global job performance criteria (in-role performance and organizational citizenship behavior), SJT validity was not significantly affected, whereas it decreased for predicting more specific criteria (interpersonal adaptability, efficacy for teamwork). Finally, the effects of omitting situation descriptions in SJTs on applicant perceptions were either negligible or small. Implications for SJT theory, research, and design are discussed

Developing a short-form situational judgment test to assess implicit trait policies for agreeableness

Implicit Trait Policies (ITPs) are defined as implicit beliefs about the effectiveness of behaviors that express a certain trait. They are typically assessed with Situational Judgment Tests (SJTs). However, such tests often lack sufficient psychometric properties. In this study (N = 133), we aimed at developing a short-form of an SJT to assess ITPs for agreeableness. Results showed, that the six-item short-version had superior model fit when compared to the original test while maintaining the same correlation to self-reported personality. Overall, the short-form is suitable for future application. Limitations and future research perspectives are discussed

Does understanding what a test measures make a difference? On the relevance of the ability to identify criteria for situational judgment test performance

Situational judgment tests (SJTs) are low‐fidelity simulations that are often used in personnel selection. Previous research has provided evidence that the ability to identify criteria (ATIC)—individuals' capability to detect underlying constructs in nontransparent personnel selection procedures—is relevant in simulations in personnel selection, such as assessment centers and situational interviews. Building on recent theorizing about response processes in SJTs as well as on previous empirical results, we posit that ATIC predicts SJT performance. We tested this hypothesis across two preregistered studies. In Study 1, a between‐subjects planned‐missingness design (N = 391 panelists) was employed and 55 selected items from five different SJTs were administered. Mixed‐effects‐modeling revealed a small effect for ATIC in predicting SJT responses. Results were replicated in Study 2 (N = 491 panelists), in which a complete teamwork SJT was administered with a high‐ or a low‐stakes instruction and showed either no or a small correlation with ATIC, respectively. We compare these findings with other studies, discuss implications for our understanding of response processes in SJTs, and derive avenues for future research

Hidden framings and hidden asymmetries in the measurement of personality: A combined lens-model and frame-of-reference perspective

Objective The symmetry principle and the frame-of-reference perspective have each made contributions to improving the measurement of personality. Although each perspective is valuable in its own right, we argue that even greater improvement can be achieved through the combination of both. Therefore, the goal of the current article was to show the value of a combined lens-model and frame-of-reference perspective. Method We conducted a literature review to summarize relevant research findings that shed light on the interplay of both perspectives and developed an integrative model. Results Based on the literature review and on theoretical grounds, we argue that a basic premise of the frame-of-reference literature--that personality items are open to interpretation and allow individuals to impose their own contextual framings--should be considered from a symmetry perspective. Unintended context-specificity in items may "spread" to personality facets and domains, and thus, impact the symmetry of personality measures with other criteria. As the individuals ' frames-of-reference and (a)symmetric relationships are not always apparent, we term them as "hidden." Conclusions The proposed combination of lens-model and frame-of-reference perspectives provides further insights into current issues in personality research and uncovers important avenues for future research