&lt;p&gt;&lt;b&gt;Background and Purpose:&lt;/b&gt; A perceived weakness of the modified Rankin Scale is potential for interobserver variability. We undertook a systematic review of modified Rankin Scale reliability studies.&lt;/p&gt;



&lt;p&gt;&lt;b&gt;Methods:&lt;/b&gt; Two researchers independently reviewed the literature. Crossdisciplinary electronic databases were interrogated using the following key words: Stroke*; Cerebrovasc*; Modified Rankin*; Rankin Scale*; Oxford Handicap*; Observer variation*. Data were extracted according to prespecified criteria with decisions on inclusion by consensus.&lt;/p&gt;



&lt;p&gt;&lt;b&gt;Results:&lt;/b&gt; From 3461 titles, 10 studies (587 patients) were included. Reliability of modified Rankin Scale varied from weighted κ=0.95 to κ=0.25. Overall reliability of mRS was κ=0.46; weighted κ=0.90 (traditional modified Rankin Scale) and κ=0.62; weighted κ=0.87 (structured interview).&lt;/p&gt;



&lt;p&gt;&lt;b&gt;Conclusion:&lt;/b&gt; There remains uncertainty regarding modified Rankin Scale reliability. Interobserver studies closest in design to large-scale clinical trials demonstrate potentially significant interobserver variability.&lt;/p&gt

Dawson, J.

Lees, K.R.

Quinn, T.J.

Walters, M.

English

Enlighten

       Quinn, T.J., Dawson, J. , Walters, M. , and Lees, K.R. (2009) Reliability of the modified Rankin Scale: a systematic review. Stroke, 40 (10). pp. 3393-3395. ISSN 0039-2499  http://eprints.gla.ac.uk/18438/  Deposited on: 18 January 2012   Enlighten – Research publications by members of the University of Glasgow http://eprints.gla.ac.uk Terence J. Quinn, Jesse Dawson, Matthew R. Walters and Kennedy R. LeesReliability of the Modified Rankin Scale : A Systematic ReviewISSN: 1524-4628 Copyright © 2009 American Heart Association. All rights reserved. Print ISSN: 0039-2499. OnlineStroke is published by the American Heart Association. 7272 Greenville Avenue, Dallas, TX 72514doi: 10.1161/STROKEAHA.109.5572562009, 40:3393-3395: originally published online August 13, 2009Stroke  http://stroke.ahajournals.org/content/40/10/3393located on the World Wide Web at: The online version of this article, along with updated information and services, is http://www.lww.com/reprintsReprints: Information about reprints can be found online at    journalpermissions@lww.com410-528-8550. E-mail: Fax:Kluwer Health, 351 West Camden Street, Baltimore, MD 21202-2436. Phone: 410-528-4050. Permissions: Permissions & Rights Desk, Lippincott Williams & Wilkins, a division of Wolters   http://stroke.ahajournals.org//subscriptions/Subscriptions: Information about subscribing to Stroke is online at  at GLASGOW UNIV LIB on January 18, 2012http://stroke.ahajournals.org/Downloaded from Reliability of the Modified Rankin ScaleA Systematic ReviewTerence J. Quinn, MRCP; Jesse Dawson, MRCP; Matthew R. Walters, MD; Kennedy R. Lees, MDBackground and Purpose—A perceived weakness of the modified Rankin Scale is potential for interobserver variability.We undertook a systematic review of modified Rankin Scale reliability studies.Methods—Two researchers independently reviewed the literature. Crossdisciplinary electronic databases were interrogatedusing the following key words: Stroke*; Cerebrovasc*; Modified Rankin*; Rankin Scale*; Oxford Handicap*; Observervariation*. Data were extracted according to prespecified criteria with decisions on inclusion by consensus.Results—From 3461 titles, 10 studies (587 patients) were included. Reliability of modified Rankin Scale varied fromweighted 0.95 to 0.25. Overall reliability of mRS was 0.46; weighted 0.90 (traditional modified RankinScale) and 0.62; weighted 0.87 (structured interview).Conclusion—There remains uncertainty regarding modified Rankin Scale reliability. Interobserver studies closest in design tolarge-scale clinical trials demonstrate potentially significant interobserver variability. (Stroke. 2009;40:3393-3395.)Key Words: clinical trials  clinometrics  modified Rankin Scales  outcome assessment  systematic reviewThe modified Rankin Scale (mRS) is the most prevalentfunctional outcome measure in contemporary strokeresearch.1 A weakness of mRS grading is the potential forinterobserver variability. Variability implies end pointmisclassification and can weaken statistical power.1 Vari-ous attempts to quantify mRS reliability have beenreported.Clinical trial use of the mRS is global2 and often used byresearch nurses and professions allied to medicine.2 A con-temporary, systematic review of the international literature,including allied healthcare journals, was performed.MethodologyTwo clinical researchers independently reviewed the literature.Throughout the process, we adhered to Meta-analysis of ObservationalStudies in Epidemiology (MOOSE) guidelines for meta-analysis.3ParticipantsStudy populations included human stroke survivors only. We usedno restrictions for mRS assessor.Study MethodologyAll studies purporting to measure mRS reliability through patientinterview (inter-/intraobserver variability) were reviewed.Bias and Trial QualityAs a minimum data set to allow assessment of trial quality, wecollected details on: patient selection; blinding of mRS interviewers(to others scores); mRS methodology; number of patients/observers;details of withdrawals; and dropouts.OutcomesNo restriction on the basis of mRS assessment methodology wasapplied. Studies using mRS derivatives (Rankin Scale and OxfordHandicap Scale) were reviewed for references but not included.Search StrategyA comprehensive battery of crossdiscipline electronic databaseswere interrogated: AMED; British Nursing Index; CINAHL; Em-base; Health and Psychosocial Instruments; Internurse.com; Med-line; and PsychINFO (all inception to December 2008). Key wordswere formulated using MeSH headings and designed to be asinclusive as possible (Figure).To identify studies not yet in print, proceedings of scientificmeetings were hand-searched: International Stroke Conference; Eu-ropean Stroke Conference; and World Stroke Congress (January2006 to November 2008). Bibliographies of retrieved articles weresearched for further references and the process repeated until no newarticles were found.We retrieved full text of articles that either reviewer suspectedmay be relevant. Data were extracted according to prespecifiedcriteria. Decisions on inclusion were by consensus. Where poten-tially relevant data were not published, electronic contact withoriginal authors was attempted. For studies not published in English,professional translation services were used.StatisticsTo allow for comparison and where data permitted, we describedresults using ; quadratic weighted , and percentage agreement.Based on previous work suggesting a beneficial effect of astructured interview approach,4 we performed separate analysiscomparing “structured” and “traditional” mRS. A one-groupdescriptive study using average absolute difference with a fixedeffects model was performed using MIX software Version 1.7(www.mix-for-meta-analysis.info).Received May 25, 2009; accepted June 16, 2009.From the Department Cardiovascular and Medical Sciences, Faculty of Medicine, University of Glasgow, Glasgow, UK.Correspondence to Terence J. Quinn, MRCP, Gardiner Institute of Cardiovascular and Medical Sciences, Western Infirmary, Glasgow G11 6NT, UK.E-mail Tjq1t@clinmed.gla.ac.uk© 2009 American Heart Association, Inc.Stroke is available at http://stroke.ahajournals.org DOI: 10.1161/STROKEAHA.109.5572563393 at GLASGOW UNIV LIB on January 18, 2012http://stroke.ahajournals.org/Downloaded from ResultsThe review profile is detailed in the Figure. Ten studiesinvolving 587 patients were included in the final analysis(reference list available as supplemental data; available athttp://stroke.ahajournals.org). Median number of includedpatients was 47; median number of researchers performingmRS was 2.Interobserver variability of mRS varied from “nearperfect” (weighted 0.95) to “poor” (0.25). Overall,reliability was “moderate” for the 2 approaches to mRS(Table 1 and 2). Three studies4 – 6 (162 patients) measuredintraobserver variability of mRS. Overall reliability wasvery good with combined weighted 0.94 and percentageagreement 84%.In the included studies, diverse methodologies were used toadminister and study mRS (Supplemental Table, availableonline at http://stroke.ahajournals.org). No study met our“minimum” criteria to allow assessment of quality: no de-scription of patient selection (n5); no data on blinding(n5); inadequate description of mRS methodology (n2);and no description of location/timing of mRS (n2). As aresult, we included all relevant studies regardless of method-ological quality.DiscussionWe have demonstrated that overall reliability of mRS ismoderate, but there remains potential for improvement.The effect of structured interview on reliability remainsunproven; apparent benefits seen with combined  analysiswere lost when “weighted” s were applied. We should becautious in interpreting these data; only 4 studies purportedto examine the structured approach and almost two thirdsof the data came from studies performed by the authors ofthe original structured interview. The nonparametric natureof  does not allow for comparative meta-analysis and sothe safest conclusion is that structuring mRS may partlyimprove mRS reliability, but effects have not beenconsistent.It is interesting that studies with larger numbers of patientsand observers reported poorer reliability. The importance ofinterobserver reliability in clinical trials becomes apparentwhen the number of potential end point assessors is consid-ered. In the recent Stroke-Acute Ischemic NXY Treatment(SAINT) trials, 1000 assessors from 25 countries weretrained in outcome assessment.7 We have shown that numbersof assessors included in mRS reliability studies are consider-ably smaller. The ideal methodology to assess mRS reliabilitywould involve observers of differing backgrounds and fromdiffering international centers. Only one study approachesthis “ideal” and it reports a concerningly low reliability forstandard mRS.4 We should note that all included studiesmeasured reliability across mRS 0 to 5. For clinical trials, agrade of mRS 6 (death) is often added; addition of thisobjective end point may improve overall reliability.Figure. Profile of the systematic review.Table 1. Reliability of Traditional mRSStudy * Weighted * Agreement, %†van Swieten, 19888 0.56 (0.45–0.68) 0.91 (0.71–1.00) 65%Wolfe, 1991 N/A 0.87 (0.84–0.97) 80%Berger, 19999 0.56 (0.41–0.71) 0.88 (0.58–1.00) N/AWilson, 200210 0.44 (0.29–0.62) 0.78 (0.53–1.00) 57%Newcommon, 200311 0.72 (0.55–0.89) N/A N/AWilson, 2005 0.25 (0.16–0.35) 0.71 (0.53–0.88) 43%de Caneda, 200612 0.45 (0.31–0.60) 0.45 (0.58–0.90) N/AGur, 200713 N/A 0.95 (0.89–1.00) N/AMeyer, 200814 N/A 0.90 (0.59–1.00) N/AQuinn, 2009 0.64 (0.48–0.79) 0.91 (0.65–1.00) 72%Totals 0.46 (0.41–0.51) 0.90 (0.86–0.94) 71%*95% CIs in parentheses.†Agreement between observers.NA indicates not available.Table 2. Reliability of mRS Using a StructuredInterview ApproachStudy * Weighted *Agreement,%†Wilson, 200210 0.70 (0.56–00.85) 0.93 (0.67–1.00) 78%Newcommon, 200311 0.34 (0.17–0.55) N/A 50%Wilson, 2005 0.74 (0.64–0.84) 0.91 (0.73–1.00) 81%Quinn, 2009 0.50 (0.34–0.68) 0.74 (0.455–1.00) 63%Totals 0.62 (0.56–0.69) 0.87 (0.75–1.00) 73%*95% CIs in parentheses.†Agreement between observers.NA indicates not available.3394 Stroke October 2009 at GLASGOW UNIV LIB on January 18, 2012http://stroke.ahajournals.org/Downloaded from Intraobserver studies suggested excellent reliability. How-ever, 2 studies measured mRS at distinct periods in thepatient’s recovery and thus are prone to recall bias andpotential for functional improvement between assessments.4,5Although theoretically interesting, intraobserver variabilitymay be of less relevance to clinical trials, in which primaryoutcome assessment is performed once only.There was heterogeneity between studies in several impor-tant aspects of methodology (Supplemental Table). As anexample, only one study6 made use of the recognized mRStraining resource.7 This study reported no beneficial effect ofstructured interview, perhaps suggesting that structured ap-proach is unnecessary if assessors are adequately trained.Collation of studies with differences in methodology poten-tially weakens our meta-analysis but is perhaps necessary;recent review of mRS reported substantial heterogeneity in itsapplication.1 Quality of studies varied and for all thoseincluded, certain data were incomplete. Again this weakensour analysis because we were unable to exclude potentiallybiased studies.Heterogeneity was further evident in the statistical meth-ods used in the studies: ; weighted ; intraclass correla-tion coefficient; and percentage agreement were all used.All are appropriate and there is no accepted optimal test.However, the resultant data were not readily interchange-able and offered limited potential for meta-analysis with-out access to original individual patient data. Our own useof statistics demands discussion. No universally acceptedanalysis method for multiple  statistics has been de-scribed. Recognizing this limitation, we used a groupanalysis technique that made the fewest assumptions of theunderlying data.Accepting these limitations, our study does have certainstrengths. Literature searching was comprehensive andsystematic, considering reports from non-English and“nonmedical” sources and excluding studies that would nothelp describe mRS reliability in a clinical trial setting.There remains uncertainty regarding mRS reliability.Available studies are likely underpowered and have designflaws that limit generalization. Studies closest in design tolarge-scale trials demonstrate potentially significant variabil-ity. Although we await definitive data on mRS reliability, wemust acknowledge a degree of interobserver variability isinherent in mRS.AcknowledgmentsWe are grateful to the National Health Service library services ofNorth Glasgow for document retrieval and Mr Aaron Ward fortranslation services. We acknowledge all those researchers whoresponded to our electronic enquiries.DisclosuresK.R.L., M.W., J.D., and T.J.Q. have been awarded academic grantsupport from the Chief Scientist Office to continue work ondeveloping stroke outcome assessments using mRS and all havepublished data in support of mRS as the optimal end point for acutestroke trials.References1. Quinn TJ, Dawson J, Walters MR, Lees KR. Functional outcomemeasures in contemporary stroke trials. Int J Stroke. 2009;4:200–206.2. Quinn TJ, Dawson J, Walters MR, Lees KR. Variability in modifiedRankin scoring across a large cohort of international observers.Stroke. 2008;39:2975–2979.3. Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D,Moher D, Becker BJ, Sipe TA, Thacker SB for the Meta-analysis ofObservational Studies in Epidemiology (MOOSE) Group. Meta-analysisof Observational Studies in Epidemiology: a proposal for reporting.JAMA. 2000;283:2008–2012.4. Wilson JT, Hareendran A, Hendry A, Potter J, Bone I, Muir KW.Reliability of the modified Rankin Scale across multiple raters: benefitsof a structured interview. Stroke. 2005;36:777–781.5. Wolfe CDA, Taub NA, Woodrow EJ, Burney PGJ. Assessment of scalesof disability and handicap for stroke patients. Stroke. 1991;22:1242–1244.6. Quinn TJ, Dawson J, Walters MR, Lees KR. Exploring the reliability ofthe modified Rankin Scale. Stroke. 2009;40:762–766.7. Quinn TJ, Lees KR, Hardemark HG, Dawson J, Walters MR. Initialexperiences of a digital training resource for modified Rankin Scaleassessment in clinical trials. Stroke. 2007;38:2257–2261.8. van Swieten, Koudstaal PJ, Visser MC, Schouten HJA, Gijn JV. Inter-observer agreement for-the assessment of Handicap in Stroke Patients.Stroke. 1988;19:604–607.9. Berger K, Weltermann B, Kolominsky-Rabas P, Meves S, HeuschmannP, Böhner J, Neundörfer B, Hense HW, Bu¨ttner T. The reliability ofstroke scales: The German version of NIHSS, ESS and Rankin scales.Fortschritte der Neurologie-Psychiatrie. 1999;67:81–93.10. Wilson JT, Hareendran A, Grant M, Baird T, Schulz UG, Muir KW, BoneI. Improving the assessment of outcomes in stroke: use of a structuredinterview to assign grades on the modified Rankin Scale. Stroke. 2002;33:2243–2246.11. Newcommon NJ, Green TL, Hayley E, Cooke T, Hill MD. Improving theassessment of outcomes in stroke: use of a structured interview to assigngrades on the modified Rankin scale. Stroke. 2003;34:377–378.12. de Caneda, MAG, Fernandes JG, de Almeida AG, Mugnol FE. Reliabilityof neurological assessment scales in patients with stroke. Arq Neuro-Psiquiatr. 2006;64:690–697.13. Gur AY, Lampl Y, Gross B, Royter V, Shopin L, Bornstein NM. A newscale for assessing patients with vertebrobasilar stroke-the Israeli Verte-brobasilar Stroke Scale (IVBSS): inter-rater reliability and concurrentvalidity. Clinical Neurology & Neurosurgery. 2007;109:317–322.14. Meyer BC, Raman R, Chacon MR, Jensen M, Werner JD. Reliability ofsite-independent telemedicine when assessed by telemedicine-naivestroke practitioners. Journal of Stroke & Cerebrovascular Diseases.2008;17:181–186.Quinn et al Modified Rankin Scale Reliability 3395 at GLASGOW UNIV LIB on January 18, 2012http://stroke.ahajournals.org/Downloaded from 

A new scale for assessing patients with vertebrobasilar stroke-the Israeli Vertebrobasilar Stroke Scale (IVBSS): inter-rater reliability and concurrent validity.

Assessment of scales of disability and handicap for stroke patients.

Bu ¨ttner T. The reliability of stroke scales: The German version of NIHSS, ESS and Rankin scales. Fortschritte der Neurologie-Psychiatrie.

Exploring the reliability of the modified Rankin Scale.

Functional outcome measures in contemporary stroke trials.

Improving the assessment of outcomes in stroke: use of a structured interview to assign grades on the modified Rankin scale.

Improving the assessment of outcomes in stroke: use of a structured interview to assign grades on the modified Rankin Scale. Stroke.

Initial experiences of a digital training resource for modified Rankin Scale assessment in clinical trials.

Interobserver agreement for-the assessment of Handicap in Stroke Patients.

Reliability of neurological assessment scales in patients with stroke. Arq NeuroPsiquiatr.

Reliability of site-independent telemedicine when assessed by telemedicine-naive stroke practitioners.

Reliability of the modified Rankin Scale across multiple raters: benefits of a structured interview.

Variability in modified Rankin scoring across a large cohort of international observers.

Reliability of the modified Rankin Scale: a systematic review

Abstract

Similar works

Full text

Available Versions

Enlighten

Enlighten: Publications

Crossref