OBJECTIVE: To examine response rate information from mailed physician questionnaires reported in published articles.

DATA SOURCES/STUDY SETTING: Citations for articles published between 1985 and 1995 were obtained using a key word search of the Medline, PsychLit, and Sociofile databases.

STUDY DESIGN: A 5 percent random sample of relevant citations was selected from each year.

DATA COLLECTION/EXTRACTION METHODS: Citations found to be other than physician surveys were discarded and replaced with the next randomly assigned article. Selected articles were abstracted using a standardized variable list.

PRINCIPAL FINDINGS: The average response rate for mailed physician questionnaires was 61 percent. The average response rate for large sample surveys (> 1,000 observations) was 52 percent. In addition, only 44 percent of the abstracted articles reported a discussion of response bias, and only 54 percent reported any type of follow-up.

CONCLUSIONS: (1) Response rates have remained somewhat constant over time, and (2) researchers need to document the efforts used to increase response rates to mailed physician questionnaires

Cummings, S

Konrad, T

Savitz, L

PubMed

Carolina Digital Repository

Reported Response Rates to MailedPhysician QuestionnairesSimone M. Cummings, Lucy A. Savitz, and Thomas R. KonradObjective. To examine response rate information from mailed physician question-naires reported in published articles.Data Sources/Study Setting. Citations for articles published between 1985 and1995 were obtained using a key word search of the Medline, PsychLit, and Sociofiledatabases.Study Design. A 5 percent random sample of relevant citations was selected fromeach year.Data Collection/Extraction Methods. Citations found to be other than physiciansurveys were discarded and replaced with the next randomly assigned article. Selectedarticles were abstracted using a standardized variable list.Principal Findings. The average response rate for mailed physician questionnaireswas 61 percent. The average response rate for large sample surveys (> 1,000 observa-tions) was 52 percent. In addition, only 44 percent of the abstracted articles reporteda discussion of response bias, and only 54 percent reported any type of follow-up.Conclusions. (1) Response rates have remained somewhat constant over time, and(2) researchers need to document the efforts used to increase response rates to mailedphysician questionnaires.Key Words. Response rates, physician surveysThe mailed questionnaire is probably the most frequently used method forsurveying physicians in the field of health services research. Mailed surveysare less costly than other alternatives, such as telephone surveys and face-to-face interviews (Hurd, Nnadi Okolo, Hartzema, et al. 1990; Maheux, Legault,and Lambert 1989; Shosteck and Fairweather 1979; Kanuk and Berenson1975; Linsky 1975; Warwick and Lininger 1975); however, they tend to resultin lower response rates (Hurd, Nnadi Okolo, Hartzema, et al. 1990; Shosteckand Fairweather 1979; Dillman 1978; Kanuk and Berenson 1975; Linsky1975; Warwick and Lininger 1975) and thus are more likely to obtain resultsthat are biased in favor of the sample population most interested in the surveytopic (Donald 1960; Fowler 1988; Stinchcombe,Jones, and Sheatsley 1981).13471348 HSR: Health Services Research 35:6 (February 2001)BACKGROUND AND AIMSAnecdotal evidence suggests that response rates to mailed physician ques-tionnaires have been declining over time (CASRO 1982; Cartwright 1978).However, no gold standard for an acceptable response rate exists. Warwickand Lininger (1975) and Grady and Wallston (1988) suggest that responserates of50 percent are very good for mailed questionnaires. Isaac and Michael(1971) disagree, indicating that a response rate of at least 80 percent isnecessary to obtain good estimates. Gehlbach (1993) reports that althoughresponse rates of at least 80 percent are very good, rates below 80 percent(and above 40 percent) are not necessarily unacceptable. DeMaio (1980) andFowler (1988), however, report that even response rates of 80 percent maybe unsatisfactory if nonresponse bias is present.Nonresponse bias is, in fact, the most important factor in assessing theeffect of a response rate on the validity of a study (Fowler 1988; Grady andWallston 1988). If nonresponders are similar to responders in every way,the response rate will not affect generalizability to the surveyed population(Fowler 1988; Grady and Wallston 1988). Thus, even questionnaires withrelatively low response rates and for which no systematic differences betweenresponders and nonresponders exist could be considered valid. Unfortu-nately, similarities between nonresponders and responders are often difficultto assess (Fowler 1988; Grady and Wallston 1988). Many researchers attemptto address this problem by increasing the amount of follow-up as a meansof increasing the response rate, thereby decreasing the nonresponse rate.However, two studies of physicians have concluded that late respondersdo not differ significantly from earlier responders, suggesting that nonre-sponse bias is not necessarily reduced by an increased response rate (Sobaland Ferentz 1989; Berk 1985). Any type of systematic bias in response hasthe potential to result in biased conclusions-even when response rates areThis study was conducted as part of a larger project funded by the Agency for Health CarePolicy and Research, Public Health Service, U. S. Department of Health and Human Servicesgrant 1-ROI-HS06745-OlA2.Address correspondence to Simone M. Cu=mings, M.H.A., Ph.D., Assistant Professor, GraduateProgram in Health Care Administration, Simmons College, 300 The Fenway, Room S-309,Boston, MA 02115. Lucy A. Savitz, M.B.A., Ph.D. is Research Assistant Professor, Departmentof Health Policy and Administration, University of North Carolina at Chapel Hill. Thomas R.Konrad, Ph.D. is Director, Programn on Health Professions and Primary Care, Cecil G. ShepsCenter for Health Services Research, University of North Carolina at Chapel Hill. This article,submitted to Health Services Research onJune 7, 1999, was revised and accepted for publicationon January 31, 2000.Response Rates to Questionnaireshigh (DeMaio 1980; Guadagnoli and Cunningham 1989; Gilbert, Long-mate, and Branch 1992). Reporting whether there is any systematic biasbetween responders and nonresponders, therefore, is essential to determiningthe validity of published research (Fowler 1988; Grady and Wallston 1988;Gehlbach 1993).The present study seeks to determine the following:1. What are the average response rates to mailed physician question-naires, and have these response rates been declining over time?2. How well does the literature report on potential biases and variablesaffecting response rates when results from mailed physician ques-tionnaires are published?METHODIn order to establish an approximation of the universe of published reportsthat use data from mailed physician questionnaires, three databases, Medline,PsychLit, and Sociofile were searched using "physicians," "questionnaire,""questionnaires," "survey," and "surveys" as key words. The citations recov-ered from the databases were downloaded electronically by year (1986-1995)and saved in EndNoteg, a bibliographic database software program. A 5percent random sample of articles reporting data from mailed questionnaires(approximately 27 articles) was selected for analysis from each ofthe ten years.Citations of articles based on surveys other than mailed physician question-naires (such as patient surveys and telephone interviews) were discarded andreplaced with the next randomly assigned article. Data were abstracted forthe following variables:* the response rate and its components;* length of the questionnaire;* amount of follow-up;* extent of nonresponse bias;* whether the study was anonymous;* whether a return envelope was included;* whether the validity of the survey instrument was discussed;* whether a pilot study had been conducted;* year of publication;* name ofjournal;13491350 HSR: Health Services Research 35:6 (Februaty 2001)* author(s);* survey objectives; and* the population being sampled.These items were included for each article in a comparative matrix to theextent that they were reported.RESULTSResponse RatesAverage, median, minimum, and maximum response rates for the full sampleof mailed physician questionnaires and for the subsample of questionnairesused in large studies (> 1,000 observations) are summarized in Table 1 andTable 2, respectively. The overall average response rate to a mailed physicianquestionnaire was 61 percent, while the overall average response rate for largestudies was 52 percent.These findings are comparable to those of Cartwright (1978), whofound an overall average response rate of 71 percent to mailed physicianquestionnaires published between 1961 and 1977, and to those of Asch,Jedrziewski, and Christakis (1997), who reported an overall average responserate of 54 percent to mailed physician questionnaires published in 1991.Analysis of abstracted response rate data indicates that response ratesto mailed physician questionnaires have not been declining over time.Table 1: Average Response Rates Between 1986 and 1995Number Average Median Minimum maximumin Response Response R esns ResponseSample Rate Rate Rate RateYear (N) (6) (96) (9) (9)1995 27 63 62 19 911994 25 60 61 17 871993 25 64 65 15 1001992 25 65 64 39 911991 27 57 57 12 901990 26 63 67 19 931989 25 60 62 21 921988 26 63 62 24 961987 26 59 62 11 861986 25 58 62 14 91Average: 61.20 62.40Response Rates to QuestionnairesTable 2: Average Response Rates for Studies with Sample SizeGreater than 1,000, 1986-1995Number Average Median Minimum Maximumin Response Response Response ResponseSampk Rate Rate Rate RateYear (N) (%) (%) (%) (%)1995 8 54 60 19 841994 11 49 58 17 761993 7 52 60 15 651992 9 57 54 39 821991 10 54 57 12 901990 6 43 43 19 681989 8 58 58 21 861988 8 50 43 25 821987 6 51 59 11 661986 11 53 54 20 81Average: 52.10 54.60Figure 1 clearly indicates that response rates to mailed physician question-naires did not decline between 1986 and 1995 and, in fact, suggests thatresponse rates have remained somewhat constant over this ten-year period.Assessment of ValidityNinety-five percent of the studies sampled reported a response rate. Threepercent of those, however, did not include the number of individuals in thesample or the number of individuals responding. In addition, although theaverage response rate was fairly low, only 44 percent of the articles discussedFigure 1: Response Rates to Mailed Physician Questionnaires.toZ,-0r_Ull1986 1987 1988 1989 199() 1991 1992 1993 1994 1995Year400-- Ave-rage Response Rate2() - .. - Aver-age Response Rate ----------______(saniple > 1000)10(if) A. _nn.n. __ ____6135110,1352 HSR: Health Services Research 35:6 (February 2001)any type of response bias, and only 18 percent of the articles performed anytype of comparison between responders and nonresponders.Reporting ofInfluential VariablesThe reporting ofvariables that affect a study's response rate (and nonresponserate) was very sketchy at best. A little over half (54 percent) of the articlesmentioned whether any type of follow-up had been conducted. Only aboutone-fifth of the articles mentioned whether a return envelope was included(21 percent) or whether a pilot study was conducted (20 percent). Slightlyfewer articles (17 percent) mentioned whether the study was anonymous.Even more discouraging, only 8 percent of the articles discussed the validityof the survey instrument.DISCUSSION AND CONCLUSIONReported response rates are often used by researchers as a quick proxy forsurvey quality because documenting every variable that affects the validity ofa survey's results is impractical. Factors such as the length ofthe questionnaire,wording of survey questions, the inclusion of leading questions, and theordering of survey questions, for example, are often not acknowledged orrecognized by researchers even though they have the potential to seriouslyaffect the validity ofa survey's results. Instead, researchers report the responserate under the sometimes questionable assumption that the most appropriatemethods to ensure the validity of a survey's results have been taken.This study found an overall average response rate of 61 percent tomailed physician questionnaires and a 52 percent response rate for largesample mailed physician questionnaires, both ofwhich are well below the 80percent rate that most researchers suggest as being adequate. However, fewconclusions as to the validity of the results obtained from these questionnairescan be made due to the paucity of information included within the publishedreports, particularly with respect to the presence of nonresponse bias. Lessthan one-fifth of the articles discussed the existence of any type of differ-ences between nonresponders and responders to the study questionnaires.We acknowledge that this information is often difficult to obtain; however,two methods can be used to facilitate an analysis of differences betweenresponders and nonresponders. First, care can be taken when developingthe sampling frame to ensure that certain relevant variables with respectto the population being surveyed are noted so that comparisons betweenresponders and nonresponders may later be made. Second, researchers canattempt to interview the group of nonresponders (or a random sample ofResponse Rates to Questionnairesnonresponders) by phone or in person to determine how they differ fromresponders on select descriptive items. The use of both of these techniqueswill better enable researchers to assess the extent of nonresponse bias presentin research that utilizes mailed physician questionnaires.When an analysis ofdifferences between nonresponders and respondersis conducted that indicates few differences on relevant characteristics betweenthe two groups, one can feel more confident about accepting the results ofstudies with low response rates. Without this enhanced information, readersare left at a disadvantage and should interpret the literature with caution.Readers should be aware of the implications and potential limitations ofstudy results not clearly portrayed as we have discussed. Unless otherwisesubstantiated, reported findings should be viewed as potentially biased andthus may provide skewed results, inconsistent results, or both.In examining trends in reported response rates, we found that physi-cian response rates have not been declining over time and, in fact, haveremained somewhat constant. This finding suggests that researchers shouldnot accept very low response rates from mailed physician questionnairesunder the misconception that physician response rates to mailed physicianquestionnaires are consistently low. As noted by Berry and Kanouse (1987),exclusively focusing extended efforts to increase the response rate may, infact, introduce other forms of bias since varying follow-up approaches maydifferentially appeal to respondents. Rather, researchers need to focus on allof the elements that make a data collection effort reliable and valid.The overall objective of any researcher should be to publish reportsthat contain sufficient information so that a reader is able to make a decisionfrom the published report regarding the internal and external validity of thestudy's results. To this end, we suggest the following variables, at a minimum,be included in published reports based on mailed questionnaires:1. specification of the sampling frame;2. length and focus of the questionnaire;3. number of questionnaires mailed;4. number of questionnaires returned;5. number of returned questionnaires discarded and reasons for do-ing so;6. differences between nonresponders and responders; and7. number and method of follow-up.The availability of these variables will enable researchers to make soundjudgments regarding a study's validity and will also facilitate the replicationof a study's results.13531354 HSR: Health Services Research 35:6 (February 2001)ACKNOWLEDGMENTSSpecial thanks to Janice Pope for her assistance with data management andto the staff and researchers of the Cecil G. Sheps Center of the University ofNorth Carolina at Chapel Hill for their support and guidance.REFERENCESAsch, D. A., M. KJedrziewski, and N. A. Christakis. 1997. "Response Rates to MailSurveys Published in MedicalJournals."Journal ofClinical Epidemiology 50 (10):1129-36.Berk, M. L. 1985. "Interviewing Physicians: The Effect of Improved Response Rate."AmericanJournal ofPublic Health 75 (11): 1338-40.Berry, S. H., and D. E. Kanouse. 1987. "Physician Response to Mailed Survey: AnExperiment in Timing of Payment." Public Opinion Qyarterly 51 (1): 102-14.Cartwright, A. 1978. "Professionals as Responders: Variations in and Effects of Re-sponse Rates to Questionnaires, 1961-77." British Medical journal 2 (6149):1419-21.Council of American Survey Research Organizations (CASRO). 1982. "On the Def-inition of Response Rates." Special Report. New York: Council of AmericanSurvey Research Organizations.DeMaio, T.J. 1980. "Refusals: Who, Where and Why." Public Opinion Quarterly 44 (2):223-33.Dillman, D. A. 1978. Mail and Tekphone Surveys: The Total Design Method New York:Wiley.Donald, M. N. 1960. "Implications of Non-Response for the Interpretation of MailQuestionnaire Data." Public Opinion Quarterly 24 (1): 99-114.Fowler, F.J. 1988. Survey Research Methods. Beverly Hills, CA: Sage.Gehlbach, S. H. 1993. Interpreting the Medical Literature. New York: McGraw-Hill.Gilbert, G. H.,J. Longmate, and L. G. Branch. 1992. "Factors Influencing the Effec-tiveness of Mailed Health Surveys." Public Health Reports 107 (5): 576-84.Grady, K. E., and B. S. Wallston. 1988. Research in Health Care Settings. Newbury Park,CA: Sage.Guadagnoli, E., and S. Cunningham. 1989. "The Effects ofNonresponse and Late Re-sponse on a Survey of Physician Attitudes." Evaluation and the Health Professions12 (3): 318-28.Hurd, P. D., E. Nnadi Okolo, A. G. Hartzema, C. 0. Airhihenbuwa, and M. Wang.1990. "Research Design and Data Collection Methods." In Health Research Designand Methodology, edited by E. Nnadi Okolo, pp. 97-148. Boca Raton, FL: CRCPress.Isaac, S., and W B. Michael. 1971. Handbook in Research and Evaluation. San Diego:Edits.Kanuk, L., and C. Berenson. 1975. "Mail Surveys and Response Rates: A LiteratureReview."Journal ofMarketing Research 12 (November): 440-53.Response Rates to Questionnaires 1355Linsky, A. S. 1975. "Stimulating Responses to Mailed Questionnaires: A Review."Public Opinion Quarterly 39 (1): 82-101.Maheux, B., C. Legault, andJ. Lambert. 1989. "Increasing Response Rates in Physi-cians' Mail Surveys: An Experimental Study." AmericanJournal ofPublic Health79 (5): 638-39.Shosteck, H., and W R. Fairweather. 1979. "Physician Response Rates to Mail andPersonal Interview Surveys." Public Opinion Quarterly 43 (2): 206-17.Sobal,J., and K S. Ferentz. 1989. "Comparing Physicians' Responses to the First andSecond Mailings of a Questionnaire." Evaluation and the Health Professions 12 (3):329-39.Stinchcombe, A. L., C.Jones, and P. Sheatsley. 1981. "Non-Response Bias for AttitudeQuestions." Public Opinion Quarterly 45 (3): 359-75.Warwick, D. P., and C. A. lininger. 1975. The Sample Survey: Theory and Practice. NewYork: McGraw-Hill.

Reported response rates to mailed physician questionnaires.

http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1089194

Reported response rates to mailed physician questionnaires.

Abstract

Similar works

Full text

Available Versions

Carolina Digital Repository