Psychometric modeling as a tool to investigate heterogeneous response scale use

Respondents use different ways to respond to rating scale items. Hence, item responses do not only capture the trait to be measured, but also the way respondents react to rating scales. So-called response styles have been incorporated in a variety of psychometric modeling approaches and investigated in applied fields. In my dissertation, I address psychometric and substantive research questions with regards to response styles in four research articles. In the first article, we structure the variety of psychometric approaches accounting for response styles. We propose a superordinate, unifying framework for such models by introducing one common parameterization. This parameterization then guides our analysis of commonalities and differences, assumptions and identification constraints in the psychometric approaches (Henninger & Meiser, 2019a). We build on the proposed framework in our second article. Herein, we highlight application scenarios and demonstrate how assumptions on response styles can be tested through psychometric approaches. We furthermore develop two novel modeling extensions that lift constraints on model parameters or explain the influence of response styles on items through item attributes (Henninger & Meiser, 2019b). In the third article (Henninger, 2019), I develop a psychometric modeling approach using a theoretically motivated restriction to achieve statistical identification. The model incorporates little a priori assumptions on response styles and retains the flexibility to account for various kinds of response tendencies in the data. Therefore, it is particularly useful in research environments where response styles differ between subgroups of respondents. The new model is tested in a simulation study and illustrated in a multi-country analysis using data measuring the Big Five personality factors. The fourth article (Henninger & Plieninger, 2019) deals with processes underlying rating scale responses by examining response times. We find that extreme responding follows a different process than agree and mid responding, and that responses that are in line with the response style trait are given faster. Our analyses suggest that every respondent employs some type of response tendencies that facilitate certain category choices in terms of response speed. In summary, I integrate existing and propose novel psychometric approaches for response style modeling, and provide new insights into the processes impacting rating scale responses. The two perspective on response styles are mutually reinforcing: psychometric models allow us to test assumptions on response styles. In turn, knowledge about the response process guides psychometricians in refining assumptions that are incorporated in modeling approaches

A New Stopping Criterion for Rasch Trees Based on the Mantel–Haenszel Effect Size Measure for Differential Item Functioning

To detect differential item functioning (DIF), Rasch trees search for optimal splitpoints in covariates and identify subgroups of respondents in a data-driven way. To determine whether and in which covariate a split should be performed, Rasch trees use statistical significance tests. Consequently, Rasch trees are more likely to label small DIF effects as significant in larger samples. This leads to larger trees, which split the sample into more subgroups. What would be more desirable is an approach that is driven more by effect size rather than sample size. In order to achieve this, we suggest to implement an additional stopping criterion: the popular Educational Testing Service (ETS) classification scheme based on the Mantel–Haenszel odds ratio. This criterion helps us to evaluate whether a split in a Rasch tree is based on a substantial or an ignorable difference in item parameters, and it allows the Rasch tree to stop growing when DIF between the identified subgroups is small. Furthermore, it supports identifying DIF items and quantifying DIF effect sizes in each split. Based on simulation results, we conclude that the Mantel–Haenszel effect size further reduces unnecessary splits in Rasch trees under the null hypothesis, or when the sample size is large but DIF effects are negligible. To make the stopping criterion easy-to-use for applied researchers, we have implemented the procedure in the statistical software R. Finally, we discuss how DIF effects between different nodes in a Rasch tree can be interpreted and emphasize the importance of purification strategies for the Mantel–Haenszel procedure on tree stopping and DIF item classification

The effect of response formats on response style strength: An experimental comparison

Many researchers use self-report data to examine abilities, personality, or attitudes. At the same time, there is a widespread concern that response styles, such as the tendency to give extreme, midscale, or acquiescent responses, may threaten data quality. As an alternative to post hoc control of response styles using psychometric models, a priori control using specific response formats may be a means to reduce biasing response style effects in self-report data in day-to-day research practice. Previous research has suggested that response styles were less influential in a Drag-and-Drop format compared to the traditional Likert-type format. In this article, we further examine the advantage of the Drag-and-Drop format, test its generalizability, and investigate its underlying mechanisms. In two between-subject experiments, we tested different versions of the Drag-and-Drop format against the Likert format. We found no evidence for reduced response style influence in any of the Drag-and-Drop conditions, nor did we find any difference between the conditions in terms of the validity of the measures to external criteria. We conclude that adaptations of response formats, such as the Drag-and-Drop format, may be promising, but require more thorough examination before recommending them as a means to reduce response style influence in psychological measurement

Interpretable machine learning for psychological research: Opportunities and pitfalls

In recent years, machine learning methods have become increasingly popular prediction methods in psychology. At the same time, psychological researchers are typically not only interested in making predictions about the dependent variable, but also in learning which predictor variables are relevant, how they influence the dependent variable, and which predictors interact with each other. However, most machine learning methods are not directly interpretable. Interpretation techniques that support researchers in describing how the machine learning technique came to its prediction may be a means to this end. We present a variety of interpretation techniques and illustrate the opportunities they provide for interpreting the results of two widely used black box machine learning methods that serve as our examples: random forests and neural networks. At the same time, we illustrate potential pitfalls and risks of misinterpretation that may occur in certain data settings. We show in which way correlated predictors impact interpretations with regard to the relevance or shape of predictor effects and in which situations interaction effects may or may not be detected. We use simulated didactic examples throughout the article, as well as an empirical data set for illustrating an approach to objectify the interpretation of visualizations. We conclude that, when critically reflected, interpretable machine learning techniques may provide useful tools when describing complex psychological relationships

Changes in attitudes towards smoking during smoking cessation courses for Turkish- and Albanian-speaking migrants in Switzerland and its association with smoking behavior: A latent change score approach

Introduction Migrant populations usually report higher smoking rates than locals. At the same time, people with a migration background have little or no access to regular smoking cessation treatment. In the last two decades, regular smoking cessation courses were adapted to reach out to Turkish- and Albanian-speaking migrants living in Switzerland. The main aims of the current study were (1) to analyze the effects of an adapted smoking cessation course for Turkish- and Albanian-speaking migrants in Switzerland on attitudes toward smoking and smoking behavior; and (2) to elucidate whether changes in attitudes toward smoking were associated to changes in smoking behavior in the short- and in the long-term. Methods A total of 59 smoking cessation courses (Turkish: 37; Albanian: 22) with 436 participants (T: 268; A: 168) held between 2014 and 2019 were evaluated. Attitudes toward smoking and cigarettes smoked per day were assessed at baseline and 3-months follow-up. One-year follow-up calls included assessment of cigarettes smoked per day. Data were analyzed by means of structural equation modeling with latent change scores. Results Participation in an adapted smoking cessation course led to a decrease of positive attitudes toward smoking (T: β = −0.65, p < 0.001; A: β = −0.68, p < 0.001) and a decrease of cigarettes smoked per day in the short-term (T: β = −0.58, p < 0.001; A: β = −0.43, p < 0.001) with only Turkish-speaking migrants further reducing their smoking in the long-term (T: β = −0.59, p < 0.001; A: β = −0.14, p = 0.57). Greater decreases in positive attitudes were associated with greater reductions of smoking in the short-term (T: r = 0.39, p < 0.001; A: r = 0.32, p = 0.03), but not in the long-term (T: r = −0.01, p = 0.88; A: r = −0.001, p = 0.99). Conclusion The adapted smoking cessation courses fostered changes in positive attitudes toward smoking that were associated with intended behavior change in the short-term. The importance of socio-cognitive characteristics related to behavior change maintenance to further increase treatment effectiveness in the long-term is discussed

Investigating Relationships Among Self-Efficacy, Mood, and Anxiety Using Digital Technologies: Randomized Controlled Trial.

BACKGROUND Digital tools assessing momentary parameters and offering interventions in people's daily lives play an increasingly important role in mental health research and treatment. Ecological momentary assessment (EMA) makes it possible to assess transient mental health states and their parameters. Ecological momentary interventions (EMIs) offer mental health interventions that fit well into individuals' daily lives and routines. Self-efficacy is a transdiagnostic construct that is commonly associated with positive mental health outcomes. OBJECTIVE The aim of our study assessing mood, specific self-efficacy, and other parameters using EMA was 2-fold. First, we wanted to determine the effects of daily assessed moods and dissatisfaction with social contacts as well as the effects of baseline variables, such as depression, on specific self-efficacy in the training group (TG). Second, we aimed to explore which variables influenced both groups' positive and negative moods during the 7-day study period. METHODS In this randomized controlled trial, we applied digital self-efficacy training (EMI) to 93 university students with elevated self-reported stress levels and daily collected different parameters, such as mood, dissatisfaction with social contacts, and specific self-efficacy, using EMA. Participants were randomized to either the TG, where they completed the self-efficacy training combined with EMA, or the control group, where they completed EMA only. RESULTS In total, 93 university students participated in the trial. Positive momentary mood was associated with higher specific self-efficacy in the evening of the same day (b=0.15, SE 0.05, P=.005). Higher self-efficacy at baseline was associated with reduced negative mood during study participation (b=-0.61, SE 0.30, P=.04), while we could not determine an effect on positive mood. Baseline depression severity was significantly associated with lower specific self-efficacy over the week of the training (b=-0.92, SE 0.35, P=.004). Associations between higher baseline anxiety with higher mean negative mood (state anxiety: b=0.78, SE 0.38, P=.04; trait anxiety: b=0.73, SE 0.33, P=.03) and lower mean positive mood (b=-0.64, SE 0.28, P=.02) during study participation were found. Emotional flexibility was significantly enhanced in the TG. Additionally, dissatisfaction with social contacts was associated with both a decreased positive mood (b=-0.56, SE 0.15, P<.001) and an increased negative mood (b=0.45, SE 0.12, P<.001). CONCLUSIONS This study showed several significant associations between mood and self-efficacy as well as those between mood and anxiety in students with elevated stress levels, for example, suggesting that improving mood in people with low mood could enhance the effects of digital self-efficacy training. In addition, engaging in 1-week self-efficacy training was associated with increased emotional flexibility. Future work is needed to replicate and investigate the training's effects in other groups and settings. TRIAL REGISTRATION ClinicalTrials.gov NCT05617248; https://clinicaltrials.gov/study/NCT05617248

Investigating Relationships Among Self-Efficacy, Mood, and Anxiety Using Digital Technologies: Randomized Controlled Trial

Background Digital tools assessing momentary parameters and offering interventions in people’s daily lives play an increasingly important role in mental health research and treatment. Ecological momentary assessment (EMA) makes it possible to assess transient mental health states and their parameters. Ecological momentary interventions (EMIs) offer mental health interventions that fit well into individuals’ daily lives and routines. Self-efficacy is a transdiagnostic construct that is commonly associated with positive mental health outcomes. Objective The aim of our study assessing mood, specific self-efficacy, and other parameters using EMA was 2-fold. First, we wanted to determine the effects of daily assessed moods and dissatisfaction with social contacts as well as the effects of baseline variables, such as depression, on specific self-efficacy in the training group (TG). Second, we aimed to explore which variables influenced both groups’ positive and negative moods during the 7-day study period. Methods In this randomized controlled trial, we applied digital self-efficacy training (EMI) to 93 university students with elevated self-reported stress levels and daily collected different parameters, such as mood, dissatisfaction with social contacts, and specific self-efficacy, using EMA. Participants were randomized to either the TG, where they completed the self-efficacy training combined with EMA, or the control group, where they completed EMA only. Results In total, 93 university students participated in the trial. Positive momentary mood was associated with higher specific self-efficacy in the evening of the same day (b=0.15, SE 0.05, P=.005). Higher self-efficacy at baseline was associated with reduced negative mood during study participation (b=–0.61, SE 0.30, P=.04), while we could not determine an effect on positive mood. Baseline depression severity was significantly associated with lower specific self-efficacy over the week of the training (b=–0.92, SE 0.35, P=.004). Associations between higher baseline anxiety with higher mean negative mood (state anxiety: b=0.78, SE 0.38, P=.04; trait anxiety: b=0.73, SE 0.33, P=.03) and lower mean positive mood (b=–0.64, SE 0.28, P=.02) during study participation were found. Emotional flexibility was significantly enhanced in the TG. Additionally, dissatisfaction with social contacts was associated with both a decreased positive mood (b=–0.56, SE 0.15, P<.001) and an increased negative mood (b=0.45, SE 0.12, P<.001). Conclusions This study showed several significant associations between mood and self-efficacy as well as those between mood and anxiety in students with elevated stress levels, for example, suggesting that improving mood in people with low mood could enhance the effects of digital self-efficacy training. In addition, engaging in 1-week self-efficacy training was associated with increased emotional flexibility. Future work is needed to replicate and investigate the training’s effects in other groups and settings. Trial Registration ClinicalTrials.gov NCT05617248; https://clinicaltrials.gov/study/NCT0561724

A novel Partial Credit extension using varying thresholds to account for response tendencies

Item Response Theory models with varying thresholds are essential tools to account for unknown types of response tendencies in rating data. However, in order to separate constructs to be measured and response tendencies, specific constraints have to be imposed on varying thresholds and their interrelations. In this article, a multidimensional extension of a Partial Credit Model using a sum-to-zero constraint for varying thresholds is proposed. The new model allows us to flexibly account for response tendencies and to model covariations between varying thresholds that are commonly found in empirical data. The model’s ability to estimate different types of response tendencies under various data structures is shown in a simulation study. An illustrative multi-country analysis demonstrates that differences between respondents in terms of their response tendencies exist and can be captured by the new model. Furthermore, it is well suited to account for extreme and mid response styles, but also to accommodate unknown, previously unmodeled, response tendencies. Therewith, the sum-to-zero model can be considered a suitable candidate to examine the types response tendencies in rating data and to account for biases in construct measures due to response tendencies