Detection, avoidance, and compensation - three studies on extreme response style

Extreme Response Style (ERS) describes individual differences in selecting extreme response options in Likert scale items, which are stable over time (Weijters et al., 2010b; Wetzel, Lüdtke, et al., 2016) and across different psychological constructs (Wetzel, Carstensen, & Böhnke, 2013). This thesis contains three empirical studies on the detection, avoidance, and compensation of ERS: In the first study, we introduce a new method to detect ERS which uses an ERS index from heterogeneous items as covariate in partial credit trees (PC trees; Komboz et al., 2016). This approach combines the objectivity of ERS indices from heterogeneous items (Greenleaf, 1992) with the threshold interpretation of ERS known from analyses with the ordinal mixed-Rasch model (Rost, 1991). We analyzed personality facets of 11714 subjects from the German nonclinical normative sample of the Revised NEO Personality Inventory (NEO-PI-R; Ostendorf & Angleitner, 2004), and 3835 participants of the longitudinal panel of the GESIS - Leibniz-Institute for the Social Sciences (GESIS, 2015), who filled out the Positive and Negative Affect Schedule (Krohne et al., 1996), and the Questionnaire of Spatial Strategies (Münzer & Hölscher, 2011). ERS was detected in all analyzed scales. The resulting pattern suggests that ERS reflects a stable trait with a continuous structure. In the second study, we investigate whether data from items with dichotomous response formats are unaffected by ERS, as has been assumed in the literature (Wetzel, Carstensen, & Böhnke, 2013). In a paper and pencil questionnaire, 429 German psychology students completed the Shyness scale from the Revised Minnesota Multiphasic Personality Inventory (Butcher, Dahlstrom, Graham, Tellegen, & Kaemmer, 1989) and the Achievement Orientation scale from the Revised Freiburger Persöhnlichkeitsinventar (Fahrenberg et al., 2001). ERS was assessed by an ERS index from heterogeneous items, a binary ERS measure based on the classification of an ordinal mixed-Rasch model, and a binary self-report measure of ERS. ERS measures were used as covariates in Rasch trees (Strobl et al., 2013) and DIF Lasso models (Tutz & Schauberger, 2015) of the dichotomous scales. We did not find any effect of ERS on dichotomous item responses. Adopting dichotomous response formats seems to be a reasonable strategy to avoid ERS. In the third study, we test whether instructions to give more or less extreme responses depending on participants’ individual response tendencies, can counterbalance the impact of ERS. In an online questionnaire, 788 German subjects completed the Impulsivity and Order facets of the NEO-PI-R three times under different ERS instructions. In the first round, a standard instruction was used. Participants in the experimental group received instructions for more or less extreme responses in the second and third round, while subjects in the control group responded under neutral instructions. ERS was measured by an ERS index from heterogeneous items and a self-report measure of ERS. Binary ERS classifications were used to create artificial datasets in which participants received an instruction which should either compensate or aggravate their individual response tendencies. Predictive performance of Random Forest models (Breiman, 2001), in which self-reported impulsive and orderly behaviors were predicted by the item responses, was compared between the compensation, aggravation, and control settings. No differences in predictive performance were observed between the settings. Likewise, PC tree analyses suggest that ERS was still present in the compensation setting. Including ERS measures as predictors did not increase predictive performance when items were answered under standard instructions. Our findings are in line with simulation studies suggesting that ERS has a small impact on applied psychological measurement (Plieninger, 2016; Wetzel, Böhnke, & Rose, 2016). Future research on ERS could improve psychological measurements by considering continuous models of ERS (Jin & Wang, 2014; Tutz et al., 2016). In light of recent calls to turn psychology into a more predictive science (Yarkoni & Westfall, 2017; Chapman et al., 2016), investigating the impact of ERS on criterion validity should also have high priority

Detection, avoidance, and compensation - three studies on extreme response style

Extreme Response Style (ERS) describes individual differences in selecting extreme response options in Likert scale items, which are stable over time (Weijters et al., 2010b; Wetzel, Lüdtke, et al., 2016) and across different psychological constructs (Wetzel, Carstensen, & Böhnke, 2013). This thesis contains three empirical studies on the detection, avoidance, and compensation of ERS: In the first study, we introduce a new method to detect ERS which uses an ERS index from heterogeneous items as covariate in partial credit trees (PC trees; Komboz et al., 2016). This approach combines the objectivity of ERS indices from heterogeneous items (Greenleaf, 1992) with the threshold interpretation of ERS known from analyses with the ordinal mixed-Rasch model (Rost, 1991). We analyzed personality facets of 11714 subjects from the German nonclinical normative sample of the Revised NEO Personality Inventory (NEO-PI-R; Ostendorf & Angleitner, 2004), and 3835 participants of the longitudinal panel of the GESIS - Leibniz-Institute for the Social Sciences (GESIS, 2015), who filled out the Positive and Negative Affect Schedule (Krohne et al., 1996), and the Questionnaire of Spatial Strategies (Münzer & Hölscher, 2011). ERS was detected in all analyzed scales. The resulting pattern suggests that ERS reflects a stable trait with a continuous structure. In the second study, we investigate whether data from items with dichotomous response formats are unaffected by ERS, as has been assumed in the literature (Wetzel, Carstensen, & Böhnke, 2013). In a paper and pencil questionnaire, 429 German psychology students completed the Shyness scale from the Revised Minnesota Multiphasic Personality Inventory (Butcher, Dahlstrom, Graham, Tellegen, & Kaemmer, 1989) and the Achievement Orientation scale from the Revised Freiburger Persöhnlichkeitsinventar (Fahrenberg et al., 2001). ERS was assessed by an ERS index from heterogeneous items, a binary ERS measure based on the classification of an ordinal mixed-Rasch model, and a binary self-report measure of ERS. ERS measures were used as covariates in Rasch trees (Strobl et al., 2013) and DIF Lasso models (Tutz & Schauberger, 2015) of the dichotomous scales. We did not find any effect of ERS on dichotomous item responses. Adopting dichotomous response formats seems to be a reasonable strategy to avoid ERS. In the third study, we test whether instructions to give more or less extreme responses depending on participants’ individual response tendencies, can counterbalance the impact of ERS. In an online questionnaire, 788 German subjects completed the Impulsivity and Order facets of the NEO-PI-R three times under different ERS instructions. In the first round, a standard instruction was used. Participants in the experimental group received instructions for more or less extreme responses in the second and third round, while subjects in the control group responded under neutral instructions. ERS was measured by an ERS index from heterogeneous items and a self-report measure of ERS. Binary ERS classifications were used to create artificial datasets in which participants received an instruction which should either compensate or aggravate their individual response tendencies. Predictive performance of Random Forest models (Breiman, 2001), in which self-reported impulsive and orderly behaviors were predicted by the item responses, was compared between the compensation, aggravation, and control settings. No differences in predictive performance were observed between the settings. Likewise, PC tree analyses suggest that ERS was still present in the compensation setting. Including ERS measures as predictors did not increase predictive performance when items were answered under standard instructions. Our findings are in line with simulation studies suggesting that ERS has a small impact on applied psychological measurement (Plieninger, 2016; Wetzel, Böhnke, & Rose, 2016). Future research on ERS could improve psychological measurements by considering continuous models of ERS (Jin & Wang, 2014; Tutz et al., 2016). In light of recent calls to turn psychology into a more predictive science (Yarkoni & Westfall, 2017; Chapman et al., 2016), investigating the impact of ERS on criterion validity should also have high priority

Best Practices in Supervised Machine Learning: A Tutorial for Psychologists

Supervised machine learning (ML) is becoming an influential analytical method in psychology and other social sciences. However, theoretical ML concepts and predictive-modeling techniques are not yet widely taught in psychology programs. This tutorial is intended to provide an intuitive but thorough primer and introduction to supervised ML for psychologists in four consecutive modules. After introducing the basic terminology and mindset of supervised ML, in Module 1, we cover how to use resampling methods to evaluate the performance of ML models (bias-variance trade-off, performance measures, k-fold cross-validation). In Module 2, we introduce the nonlinear random forest, a type of ML model that is particularly user-friendly and well suited to predicting psychological outcomes. Module 3 is about performing empirical benchmark experiments (comparing the performance of several ML models on multiple data sets). Finally, in Module 4, we discuss the interpretation of ML models, including permutation variable importance measures, effect plots (partial-dependence plots, individual conditional-expectation profiles), and the concept of model fairness. Throughout the tutorial, intuitive descriptions of theoretical concepts are provided, with as few mathematical formulas as possible, and followed by code examples using the mlr3 and companion packages in R. Key practical-analysis steps are demonstrated on the publicly available PhoneStudy data set (N = 624), which includes more than 1,800 variables from smartphone sensing to predict Big Five personality trait scores. The article contains a checklist to be used as a reminder of important elements when performing, reporting, or reviewing ML analyses in psychology. Additional examples and more advanced concepts are demonstrated in online materials (https://osf.io/9273g/)

Regularized target encoding outperforms traditional methods in supervised machine learning with high cardinality features

Since most machine learning (ML) algorithms are designed for numerical inputs, efficiently encoding categorical variables is a crucial aspect in data analysis. A common problem are high cardinality features, i.e. unordered categorical predictor variables with a high number of levels. We study techniques that yield numeric representations of categorical variables which can then be used in subsequent ML applications. We focus on the impact of these techniques on a subsequent algorithm's predictive performance, and-if possible-derive best practices on when to use which technique. We conducted a large-scale benchmark experiment, where we compared different encoding strategies together with five ML algorithms (lasso, random forest, gradient boosting, k-nearest neighbors, support vector machine) using datasets from regression, binary- and multiclass-classification settings. In our study, regularized versions of target encoding (i.e. using target predictions based on the feature levels in the training set as a new numerical feature) consistently provided the best results. Traditionally widely used encodings that make unreasonable assumptions to map levels to integers (e.g. integer encoding) or to reduce the number of levels (possibly based on target information, e.g. leaf encoding) before creating binary indicator variables (one-hot or dummy encoding) were not as effective in comparison

Aging and the rehabilitation of homonymous hemianopia: The efficacy of compensatory eye-movement training techniques and a five-year follow up

The specificity and effectiveness of eye-movement training to remedy impaired visual exploration and reading with particular consideration of age and co-morbidity was tested in a group of 97 patients with unilateral homonymous hemianopia using a single subject /n-of-1 design. Two groups received either scanning training followed by reading training, or vice versa. The third group acted as a control group and received non-specific detailed advice, followed by training of scanning and reading. Scanning and reading performance was assessed before and after the waiting period, before and after scanning and reading training, and at short-term (11 weeks on average) and long-term follow-up (5 years on average). Improvements after training were practice-dependent and task-specific. Scanning performance improved by ∼40%, reading by ∼45%, and was paralleled by a reduction of subjective complaints. The advice (=control) condition was without effect. All improvements occurred selectively in the training period, not in treatment-free intervals, and persisted in the short- and long-term follow-up over several years. Age had only a minor, although significant effect on improvement in reading after training; co-morbidity had no significant impact on the outcome of training. In conclusion, visual impairments associated with homonymous hemianopia can be successfully and durably reduced by systematic and specific training of compensatory eye-movement strategies. The improvements in compensation strategies were independent of subjects’ age and of co-morbidity

Cognitive Reserve in Young and Old Healthy Subjects: Differences and Similarities in a Testing-the-Limits Paradigm with DSST

Cognitive reserve (CR) is understood as capacity to cope with challenging conditions, e. g. after brain injury or in states of brain dysfunction, or age-related cognitive decline. CR in elderly subjects has attracted much research interest, but differences between healthy older and younger subjects have not been addressed in detail hitherto. Usually, one-time standard individual assessments are used to characterise CR. Here we observe CR as individual improvement in cognitive performance (gain) in a complex testing-the-limits paradigm, the digit symbol substitution test (DSST),with 10 repeated measurements, in 140 younger (20-30 yrs) and 140 older (57-74 yrs) healthy subjects. In addition, we assessed attention, memory and executive function, and mood and personality traits as potential influence factors for CR. We found that both, younger and older subjects showed significant gains, which were significantly correlated with speed of information processing, verbal short-term memory and visual problem solving in the older group only. Gender, personality traits and mood did not significantly influence gains in either group. Surprisingly about half of the older subjects performed at the level of the younger group, suggesting that interindividual differences in CR are possibly age-independent. We propose that these findings may also be understood as indication that one-time standard individual measurements do not allow assessment of CR, and that the use of DSST in a testing-the-limits paradigm is a valuable assessment method for CR in young and elderly subjects

Evaluation of the German version of the Adult Attention-Deficit/Hyperactivity Disorder self-report screening scale for DSM-5 as a screening tool for adult Attention-Deficit/Hyperactivity Disorder in primary care

Adult attention-deficit/hyperactivity disorder (ADHD) is common, but often undiagnosed. A valid and time-efficient screening tool for primary care is needed. Objective of this study is to evaluate the German version of the Adult ADHD Self-Report Scale for DSM-5 (ASRS-5) and its feasibility, acceptability, and reliability as a screening tool for adult ADHD in primary care. A multi-centered prospective, diagnostic study was performed. We recruited 262 patients in primary care practices and at an ADHD Outpatient Service of a department of psychiatry in Germany. Patients from 18 to 65 years with suspected or diagnosed ADHD were included by medical doctors, as well as non-ADHD patients as “negative controls.” Participants filled in the ASRS-5 and a sociodemographic questionnaire. The Integrated Diagnosis of Adult ADHD, revised version (IDA-R) performed by trained interviewers was used for validation. Feasibility, acceptability, and credibility in primary care practices were examined through a semi-structured interview. The German version of the ASRS-5 showed comparable psychometric properties to the English original version (sensitivity 95.6% and specificity 72.3%). For factor structure, a parallel analysis suggested one latent dimension. Performing confirmatory factor analysis, the best fit was achieved for a general factor with one correlated error. Internal consistency results in Raykovs Omega = 0.86 and Cronbach’s α  = 0.88. The ASRS-5 was assessed positively in terms of feasibility, acceptability, and credibility by interviewed general practitioners. Potential problems were raised for “treatment options,” “stigmatization,” and “knowledge gaps.” In conclusion, the German version of the ASRS-5 offers a promising tool to improve adult ADHD patients’ diagnosis and healthcare

Eleven strategies for making reproducible research and open science training the norm at research institutions

Across disciplines, researchers increasingly recognize that open science and reproducible research practices may accelerate scientific progress by allowing others to reuse research outputs and by promoting rigorous research that is more likely to yield trustworthy results. While initiatives, training programs, and funder policies encourage researchers to adopt reproducible research and open science practices, these practices are uncommon inmanyfields. Researchers need training to integrate these practicesinto their daily work. We organized a virtual brainstorming event, in collaboration with the German Reproducibility Network, to discuss strategies for making reproducible research and open science training the norm at research institutions. Here, weoutline eleven strategies, concentrated in three areas:(1)offering training, (2)adapting research assessment criteria and program requirements, and (3) building communities. We provide a brief overview of each strategy, offer tips for implementation,and provide links to resources. Our goal is toencourage members of the research community to think creatively about the many ways they can contribute and collaborate to build communities,and make reproducible research and open sciencetraining the norm. Researchers may act in their roles as scientists, supervisors, mentors, instructors, and members of curriculum, hiring or evaluation committees. Institutionalleadership and research administration andsupport staff can accelerate progress by implementing change across their institution

Eleven strategies for making reproducible research and open science training the norm at research institutions

Across disciplines, researchers increasingly recognize that open science and reproducible research practices may accelerate scientific progress by allowing others to reuse research outputs and by promoting rigorous research that is more likely to yield trustworthy results. While initiatives, training programs, and funder policies encourage researchers to adopt reproducible research and open science practices, these practices are uncommon inmanyfields. Researchers need training to integrate these practicesinto their daily work. We organized a virtual brainstorming event, in collaboration with the German Reproducibility Network, to discuss strategies for making reproducible research and open science training the norm at research institutions. Here, weoutline eleven strategies, concentrated in three areas:(1)offering training, (2)adapting research assessment criteria and program requirements, and (3) building communities. We provide a brief overview of each strategy, offer tips for implementation,and provide links to resources. Our goal is toencourage members of the research community to think creatively about the many ways they can contribute and collaborate to build communities,and make reproducible research and open sciencetraining the norm. Researchers may act in their roles as scientists, supervisors, mentors, instructors, and members of curriculum, hiring or evaluation committees. Institutionalleadership and research administration andsupport staff can accelerate progress by implementing change across their institution