Designing Studies for Comparing Interviewer Variance Components in Two Groups of Survey Interviewers

Methodological studies of interviewer effects often seek to identify factors that influence the magnitude of interviewer variance for particular survey questions. There is a long history of work in this area, and results from studies like this have informed current interviewing practice. Unfortunately, many studies of this type suffer from one or more of the following limitations in terms of their designs: 1) a failure to randomly assign interviewers to the treatments being compared; 2) a failure to formally test for differences in the variance components between the two groups; and 3) insufficient statistical power for comparison of the variance components. This paper (and the presentation at the interviewer effects workshop) will outline “ideal” designs for these types of studies, and present an example of successful implementation of the “ideal” design in practice. Statistical models enabling formal testing for differences in variance components will be introduced and illustrated using easy-to-use multilevel modeling procedures, and a SAS macro for performing power analyses when designing these studies will also be outlined

Power Analysis of Trials with Multilevel Data. M. Moerbeek and S. Teerenstra (2016). Boca Raton, FL: Chapman & Hall/CRC Press. 288 Pages, ISBN: 9781498729895.

Peer Reviewedhttps://deepblue.lib.umich.edu/bitstream/2027.42/140024/1/bimj1789.pdfhttps://deepblue.lib.umich.edu/bitstream/2027.42/140024/2/bimj1789_am.pd

The Error Properties of Interviewer Observations and their Implications for Nonresponse Adjustment of Survey Estimates.

Interviewer observations are an important source of auxiliary information in survey research. Interviewers can record observations for all units in a sample, and selected observations may be associated with both key survey variables and response propensity. Survey statisticians use auxiliary variables with these properties to compute post-survey nonresponse adjustments to survey estimates that reduce both bias and variance in the estimates engendered by nonresponse. Unfortunately, interviewer observations are typically judgments and estimates, making them prone to error. To date, no studies have considered the implications of these errors for the effectiveness of nonresponse adjustments, effective observational strategies leading to reduced error rates, predictors of observation accuracy in face-to-face surveys, or alternative estimation methods for mitigating the effects of the errors on estimates. This dissertation presents results from three research studies designed to fill these important gaps in the existing literature. The first study 1) analyzes the error properties of two interviewer observations collected in the National Survey of Family Growth (NSFG), finding accuracy rates ranging from 72-78% and evidence of systematic errors; 2) examines the effectiveness of nonresponse adjustments based in part on the observations, finding evidence of associations with key NSFG variables and response propensity but only slight shifts in estimates; and 3) simulates the implications of errors in the observations for the effectiveness of weighting class adjustments for nonresponse, finding that adjustments based on the error-prone observations attenuate possible reductions in bias. The second study uses multilevel modeling techniques to identify several respondent- and interviewer-level predictors of accuracy in the two NSFG observations, including those supported by social psychological theories of what leads to improved judgment accuracy. The third study develops pattern-mixture model (PMM) estimators of means for the case when an auxiliary variable is error-prone, true values for the variable are collected from survey respondents, and the true values are predictive of unit nonresponse under a non-ignorable missing data mechanism. Simulation studies show that the PMM estimators have several favorable properties in these situations relative to other popular estimators, and R code is provided implementing the PMM approaches.Ph.D.Survey MethodologyUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/89715/1/bwest_1.pdfhttp://deepblue.lib.umich.edu/bitstream/2027.42/89715/2/bwest_2.pdfhttp://deepblue.lib.umich.edu/bitstream/2027.42/89715/3/bwest_3.pd

The Need to Account for Complex Sampling Features when Analyzing Establishment Survey Data: An Illustration using the 2013 Business Research and Development and Innovation Survey (BRDIS)

The importance of correctly accounting for complex sampling features when generating finite population inferences based on complex sample survey data sets has now been clearly established in a variety of fields, including those in both statistical and non-statistical domains. Unfortunately, recent studies of analytic error have suggested that many secondary analysts of survey data do not ultimately account for these sampling features when analyzing their data, for a variety of possible reasons (e.g., poor documentation, or a data producer may not provide the information in a public-use data set). The research in this area has focused exclusively on analyses of household survey data, and individual respondents. No research to date has considered how analysts are approaching the data collected in establishment surveys, and whether published articles advancing science based on analyses of establishment behaviors and outcomes are correctly accounting for complex sampling features. This article presents alternative analyses of real data from the 2013 Business Research and Development and Innovation Survey (BRDIS), and shows that a failure to account for the complex design features of the sample underlying these data can lead to substantial differences in inferences about the target population of establishments for the BRDIS

Impact of real-time ultrasound guidance on complications of percutaneous dilatational tracheostomy: a propensity score analysis

Abstract Introduction Recent studies have demonstrated the feasibility of real-time ultrasound guidance during percutaneous dilatational tracheostomy, including in patients with risk factors such as coagulopathy, cervical spine immobilization and morbid obesity. Use of real-time ultrasound guidance has been shown to improve the technical accuracy of percutaneous dilatational tracheostomy; however, it is unclear if there is an associated reduction in complications. Our objective was to determine whether the peri-procedural use of real-time ultrasound guidance is associated with a reduction in complications of percutaneous dilatational tracheostomy using a propensity score analysis. Methods This study reviewed all percutaneous dilatational tracheostomies performed in an 8-year period in a neurocritical care unit. Percutaneous dilatational tracheostomies were typically performed by trainees under guidance of the attending intensivist. Bronchoscopic guidance was used for all procedures with addition of real-time ultrasound guidance at the discretion of the attending physician. Real-time ultrasound guidance was used to guide endotracheal tube withdrawal, guide tracheal puncture, identify guidewire entry level and confirm bilateral lung sliding. The primary outcome was a composite of previously defined complications including (among others) bleeding, infection, loss of airway, inability to complete procedure, need for revision, granuloma and early dislodgement. Propensity score analysis was used to ensure that the relationship of not using real-time ultrasound guidance with the probability of an adverse outcome was examined within groups of patients having similar covariate profiles. Covariates included were age, gender, body mass index, diagnosis, Acute Physiology and Chronic Health Evaluation II score, timing of tracheostomy, positive end-expiratory pressure and presence of risk factors including coagulopathy, cervical spine immobilization and prior tracheostomy. Results A total of 200 patients underwent percutaneous dilatational tracheostomy during the specified period, and 107 received real-time ultrasound guidance. Risk factors for percutaneous dilatational tracheostomy were present in 63 (32%). There were nine complications in the group without real-time ultrasound guidance: bleeding (n = 4), need for revision related to inability to ventilate or dislodgement (n = 3) and symptomatic granuloma (n = 2). There was one complication in the real-time ultrasound guidance group (early dislodgement). The odds of having an adverse outcome for patients receiving real-time ultrasound guidance were significantly lower (odds ratio = 0.08; 95% confidence interval, 0.009 to 0.811; P = 0.032) than for those receiving a standard technique while holding the propensity score quartile fixed. Conclusions The use of real-time ultrasound guidance during percutaneous dilatational tracheostomy was associated with a significant reduction in procedure-related complications.http://deepblue.lib.umich.edu/bitstream/2027.42/111730/1/13054_2015_Article_924.pd

A Simple and Flexible Rating Method for Predicting Success

Abstract This paper first presents a brief review of potential rating tools and methods for predicting success in the NCAA basketball tournament, including those methods (such as the Ratings Percentage Index, or RPI) that receive a great deal of weight in selecting and seeding teams for the tournament. The paper then proposes a simple and flexible rating method based on ordinal logistic regression and expectation (the OLRE method) that is designed to predict success for those teams selected to participate in the NCAA tournament. A simulation based on the parametric Bradley-Terry model for paired comparisons is used to demonstrate the ability of the computationally simple OLRE method to predict success in the tournament, using actual NCAA tournament data. Given that the proposed method can incorporate several different predictors of success in the NCAA tournament when calculating a rating, and has better predictive power than a model-based approach, it should be strongly considered as an alternative to other rating methods currently used to assign seeds and regions to the teams selected to play in the tournament

Can conversational interviewing improve survey response quality without increasing interviewer effects?

Peer Reviewedhttps://deepblue.lib.umich.edu/bitstream/2027.42/141370/1/rssa12255_am.pdfhttps://deepblue.lib.umich.edu/bitstream/2027.42/141370/2/rssa12255.pd

nvestigating the Utility of Interviewer Observations on the Survey Response Process

TBD