Middel, Berrie

van Sonderen, Eric

English

University of Groningen

   University of GroningenResponsiveness and Validity of 3 Outcome Measures of Motor Function After StrokeRehabilitationMiddel, Berrie; van Sonderen, EricPublished in:StrokeDOI:10.1161/STROKEAHA.110.580365IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionPublisher's PDF, also known as Version of recordPublication date:2010Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Middel, B., & van Sonderen, E. (2010). Responsiveness and Validity of 3 Outcome Measures of MotorFunction After Stroke Rehabilitation. Stroke, 41(7), E463-E464.https://doi.org/10.1161/STROKEAHA.110.580365CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-amendment.Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.ISSN: 1524-4628 Copyright © 2010 American Heart Association. All rights reserved. Print ISSN: 0039-2499. OnlineStroke is published by the American Heart Association. 7272 Greenville Avenue, Dallas, TX 72514DOI: 10.1161/STROKEAHA.110.580365  2010;41;e463-e464; originally published online May 13, 2010; StrokeBerrie Middel and Eric van Sonderen  Stroke RehabilitationResponsiveness and Validity of 3 Outcome Measures of Motor Function After http://stroke.ahajournals.org/cgi/content/full/41/7/e463located on the World Wide Web at: The online version of this article, along with updated information and services, is http://www.lww.com/reprintsReprints: Information about reprints can be found online at    journalpermissions@lww.com410-528-8550. E-mail: Fax:Kluwer Health, 351 West Camden Street, Baltimore, MD 21202-2436. Phone: 410-528-4050. Permissions: Permissions & Rights Desk, Lippincott Williams & Wilkins, a division of Wolters   http://stroke.ahajournals.org/subscriptions/Subscriptions: Information about subscribing to Stroke is online at  at Rijksuniversiteit Groningen on October 25, 2010 stroke.ahajournals.orgDownloaded from Letters to the EditorStroke welcomes Letters to the Editor and will publish them, if suitable, as space permits. They should not exceed 750 words (including references)and may be subject to editing or abridgment. Please submit letters in duplicate, typed double-spaced. Include a fax number for the corresponding author anda completed copyright transfer agreement form (available online at http://stroke.ahajournals.org and http://submit-stroke.ahajournals.org).Responsiveness and Validity of 3 OutcomeMeasures of Motor Function AfterStroke RehabilitationTo the Editor:With interest we read the recently published letter of Dr Sivan,“Interpreting Effect Size to Estimate Responsiveness of OutcomeMeasures,”1 as a response to a paper by Hsieh et al2 in whichthey provided indices of the magnitude of treatment-relatedintraindividual change assessed with the Fugl-Meyer Assessment(FMA), Action Research Arm Test (ARAT), and Wolf MotorFunction Test performance time (WMFT-TIME) and functionalability scores (WMFT-FAS). As an effect size index, Hsieh et alused the method of the so-called standardized response mean(SRM) by which mean change in scores over time is divided bythe SD of these change scores (see Formula A).(A) SRMX changeSD(Xchange)As Sivan argued in his letter,1 the interpretation of themagnitude of intraindividual change estimated with a SRMmay lead to overestimation or underestimation of treatment-related effects when the widely used thresholds of Cohen3 areused. These thresholds for classification of the magnitude ofmean differences were developed with an effect size indexbased on standardizing these mean differences using thepooled SD (see Formula B). Dunlap et al convincingly arguedthat only the pooled SD should be used to compute effect size(ES) for correlated designs and concluded that if the SDpooledis corrected for the amount of correlation between themeasures, then the ES estimate will be an overestimate of theactual ES.4 It is essential for clinical investigators to under-stand the differences between the SRM and ES in classifyingtreatment-related change in terms of Cohen’s thresholds (ES0.20 indicating a “trivial” change, ES between 0.20 and0.50 “small,” ES of 0.50 to 0.80 a moderate, andES 0.80 a large change).3(B) ESpX changeSD(pooled)Hsieh et al refer to our earlier work concerning the risk ofmisclassification of an SRM when using Cohen’s thresh-olds5,6 in their response to Sivan’s critic. However, theircalculation of adjusted ES estimates7 is based on a falseassumption. Consequently, adjusting ES for the size of thecorrelation between baseline and follow-up as computed byHsieh et al7 in the Table leads to an ES estimate more thantwice the magnitude of the ES computed using the SDpooledwhen the correlation between the baseline and follow-upscores is at least 0.8.4Adjustment of a SRM to ES comprises 2 components.First, Cohen introduced a (2) correction as necessary for anappropriate use of his tables for sample size calculation. Thiscorrection for looking in Cohen’s power tables is necessarybecause these assume 2(N-1) degrees of freedom (2 indepen-dent samples), whereas in, for example, pre-/posttest evalu-ation, only n-1 are actually available3 (pp 46 to 48). Thus,following Cohen’s theory, “multiplying SRM by 2 (ap-proximately 1.41) compensates for the sample size tables’assumption of double the error variance”3 (p 46).Second, because the t test prescribed in “own control”study designs (baseline to follow-up) is based on correlatedmeans3 (p 48), we also have to compensate for the correlation(r) between paired observations. Therefore, according toCohen, the relative size of the standardizing unit for the SRMto the ESpooled is not(C) dd1rbut(D) 21r)Thus, the difference between means for paired (dependent)samples needs to be standardized by a value “which is 2 (1r)as large as would be the case were they independent”3 (p 49).As was shown in an earlier publication, (d/2)/(1r) isequivalent to the SRM and alternatively SRM *  2 * (1r)Table. Effect Size Estimates SRM, ESP, and ES as Adjusted by Hsieh et alScale r r2A MeanChangeBSDchangeCSDpooledA/B EffectSize d (SRM)A/C EffectSizepooled (ESP)Adjusted EffectSize7 (ESP)/(1r)FMA 0.901 0.81 5.75 4.06 8.67 1.42 0.66 2.06ARAT 0.915 0.84 4.68 4.95 11.24 0.95 0.41 1.40WMFT-TIME 0.594 0.35 2.13 5.56 5.91 0.38 0.36 0.56WMFT-FAS 0.951 0.90 0.30 0.23 0.73 1.30 0.41 1.85FMA indicates Fugl-Meyer Assessment; ARAT, Action Research Arm Test; WMFT, Wolf Motor Function Test performance time (WMFT-TIME) andfunctional ability scores (WMFT-FAS).(Stroke. 2010;41:e463-e464.)© 2010 American Heart Association, Inc.Stroke is available at http://stroke.ahajournals.org DOI: 10.1161/STROKEAHA.110.580365e463 at Rijksuniversiteit Groningen on October 25, 2010 stroke.ahajournals.orgDownloaded from is equivalent to d and both indices will vary with the size of r.Hsieh et al have adjusted their ES but may have overlookedthe fact that their method of adjustment of SRM to an ES isincomplete. As shown in the Table, correct transformation oftheir SRM estimates into ES suitable for Cohen’s classifica-tion resulted in a moderate effect for Fugl-Meyer Assessmentand small effects for Action Research Arm Test, Wolf MotorFunction Test–TIME, and Wolf Motor Function Test–FAS,respectively, which seems in sharp contrast with the conclusionof Hsieh et al. Extreme high correlations between baseline andfollow-up (except Wolf Motor Function Test–TIME) are mainlyresponsible for overestimation of effect size by the method ofHsieh et al. Minor deviations from published indices are due tocalculations without individual data.DisclosuresNone.Berrie Middel, PhDUniversity Medical Center GroningenDepartment of Health SciencesSubdivision of Public HealthDepartment of Oral Sciences and Clinical EpidemiologyUniversity of GroningenGroningen, The NetherlandsEric van Sonderen, PhDUniversity Medical Center GroningenDepartment of Health SciencesSubdivision of Health PsychologyUniversity of GroningenGroningen, The Netherlands1. Sivan M. Interpreting effect size to estimate responsiveness of outcomemeasures. Stroke. 2009;40:e709–e711.2. Hsieh YW, Wu CY, Lin KC, Chang YF, Chen CL, Liu JS. Responsivenessand validity of three outcome measures of motor function after strokerehabilitation. Stroke. 2009;40:1386–1391.3. Cohen J. The T Test for Means. Statistical Power Analysis for the Behav-ioural Sciences, 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates;1988:19–74.4. Dunlap WP, Cortina JM, Vaslow JB, Burke MJ. Meta-analysis of exper-iments with matched groups or repeated measures designs. PsychologicalMethods. 1996;1:170–177.5. Middel B, Van Sonderen FLP. Statistical significant change versusrelevant or important change in (quasi) experimental design: some con-ceptual and methodological problems in estimating magnitude ofintervention-related change in health services research. InternationalJournal of Integrated Care. 2002;2:1–21.6. Middel B, Van Sonderen FLP. Erratum. Statistical significant changeversus relevant or important change in (quasi) experimental design: someconceptual and methodological problems in estimating magnitude ofintervention-related change in health services research. InternationalJournal of Integrated Care. 2008;8:1–2.7. Hsieh YW, Wu CY, Lin KC. Response to Letter by Sivan. Stroke. In press.e464 Letters to the Editor at Rijksuniversiteit Groningen on October 25, 2010 stroke.ahajournals.orgDownloaded from 

Responsiveness and Validity of 3 Outcome Measures of Motor Function After Stroke Rehabilitation

ARTS repository - University of Groningen

Proceedings - University of Groningen

Middel, B.,

Sonderen, E. van,

University of Groningen Digital Archive

Dissertations of the University of Groningen

   University of GroningenResponsiveness and validity of 3 outcome measures of motor function after strokerehabilitation (letter)Middel, B.; van Sonderen, E.Published in:StrokeIMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionPublisher's PDF, also known as Version of recordPublication date:2010Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Middel, B., & van Sonderen, E. (2010). Responsiveness and validity of 3 outcome measures of motorfunction after stroke rehabilitation (letter). Stroke, 41(7), E463-E464.CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 12-11-2019ISSN: 1524-4628 Copyright © 2010 American Heart Association. All rights reserved. Print ISSN: 0039-2499. OnlineStroke is published by the American Heart Association. 7272 Greenville Avenue, Dallas, TX 72514DOI: 10.1161/STROKEAHA.110.580365  2010;41;e463-e464; originally published online May 13, 2010; StrokeBerrie Middel and Eric van Sonderen  Stroke RehabilitationResponsiveness and Validity of 3 Outcome Measures of Motor Function After http://stroke.ahajournals.org/cgi/content/full/41/7/e463located on the World Wide Web at: The online version of this article, along with updated information and services, is http://www.lww.com/reprintsReprints: Information about reprints can be found online at    journalpermissions@lww.com410-528-8550. E-mail: Fax:Kluwer Health, 351 West Camden Street, Baltimore, MD 21202-2436. Phone: 410-528-4050. Permissions: Permissions & Rights Desk, Lippincott Williams & Wilkins, a division of Wolters   http://stroke.ahajournals.org/subscriptions/Subscriptions: Information about subscribing to Stroke is online at  at Rijksuniversiteit Groningen on October 25, 2010 stroke.ahajournals.orgDownloaded from Letters to the EditorStroke welcomes Letters to the Editor and will publish them, if suitable, as space permits. They should not exceed 750 words (including references)and may be subject to editing or abridgment. Please submit letters in duplicate, typed double-spaced. Include a fax number for the corresponding author anda completed copyright transfer agreement form (available online at http://stroke.ahajournals.org and http://submit-stroke.ahajournals.org).Responsiveness and Validity of 3 OutcomeMeasures of Motor Function AfterStroke RehabilitationTo the Editor:With interest we read the recently published letter of Dr Sivan,“Interpreting Effect Size to Estimate Responsiveness of OutcomeMeasures,”1 as a response to a paper by Hsieh et al2 in whichthey provided indices of the magnitude of treatment-relatedintraindividual change assessed with the Fugl-Meyer Assessment(FMA), Action Research Arm Test (ARAT), and Wolf MotorFunction Test performance time (WMFT-TIME) and functionalability scores (WMFT-FAS). As an effect size index, Hsieh et alused the method of the so-called standardized response mean(SRM) by which mean change in scores over time is divided bythe SD of these change scores (see Formula A).(A) SRM XchangeSD(Xchange)As Sivan argued in his letter,1 the interpretation of themagnitude of intraindividual change estimated with a SRMmay lead to overestimation or underestimation of treatment-related effects when the widely used thresholds of Cohen3 areused. These thresholds for classification of the magnitude ofmean differences were developed with an effect size indexbased on standardizing these mean differences using thepooled SD (see Formula B). Dunlap et al convincingly arguedthat only the pooled SD should be used to compute effect size(ES) for correlated designs and concluded that if the SDpooledis corrected for the amount of correlation between themeasures, then the ES estimate will be an overestimate of theactual ES.4 It is essential for clinical investigators to under-stand the differences between the SRM and ES in classifyingtreatment-related change in terms of Cohen’s thresholds (ES0.20 indicating a “trivial” change, ES between 0.20 and0.50 “small,” ES of 0.50 to 0.80 a moderate, andES 0.80 a large change).3(B) ESpX changeSD(pooled)Hsieh et al refer to our earlier work concerning the risk ofmisclassification of an SRM when using Cohen’s thresh-olds5,6 in their response to Sivan’s critic. However, theircalculation of adjusted ES estimates7 is based on a falseassumption. Consequently, adjusting ES for the size of thecorrelation between baseline and follow-up as computed byHsieh et al7 in the Table leads to an ES estimate more thantwice the magnitude of the ES computed using the SDpooledwhen the correlation between the baseline and follow-upscores is at least 0.8.4Adjustment of a SRM to ES comprises 2 components.First, Cohen introduced a (2) correction as necessary for anappropriate use of his tables for sample size calculation. Thiscorrection for looking in Cohen’s power tables is necessarybecause these assume 2(N-1) degrees of freedom (2 indepen-dent samples), whereas in, for example, pre-/posttest evalu-ation, only n-1 are actually available3 (pp 46 to 48). Thus,following Cohen’s theory, “multiplying SRM by 2 (ap-proximately 1.41) compensates for the sample size tables’assumption of double the error variance”3 (p 46).Second, because the t test prescribed in “own control”study designs (baseline to follow-up) is based on correlatedmeans3 (p 48), we also have to compensate for the correlation(r) between paired observations. Therefore, according toCohen, the relative size of the standardizing unit for the SRMto the ESpooled is not(C) d d1rbut(D) 21r)Thus, the difference between means for paired (dependent)samples needs to be standardized by a value “which is2 (1r)as large as would be the case were they independent”3 (p 49).As was shown in an earlier publication, (d/2)/(1r) isequivalent to the SRM and alternatively SRM *  2 * (1r)Table. Effect Size Estimates SRM, ESP, and ES as Adjusted by Hsieh et alScale r r2A MeanChangeBSDchangeCSDpooledA/B EffectSize d (SRM)A/C EffectSizepooled (ESP)Adjusted EffectSize7 (ESP)/(1r)FMA 0.901 0.81 5.75 4.06 8.67 1.42 0.66 2.06ARAT 0.915 0.84 4.68 4.95 11.24 0.95 0.41 1.40WMFT-TIME 0.594 0.35 2.13 5.56 5.91 0.38 0.36 0.56WMFT-FAS 0.951 0.90 0.30 0.23 0.73 1.30 0.41 1.85FMA indicates Fugl-Meyer Assessment; ARAT, Action Research Arm Test; WMFT, Wolf Motor Function Test performance time (WMFT-TIME) andfunctional ability scores (WMFT-FAS).(Stroke. 2010;41:e463-e464.)© 2010 American Heart Association, Inc.Stroke is available at http://stroke.ahajournals.org DOI: 10.1161/STROKEAHA.110.580365e463 at Rijksuniversiteit Groningen on October 25, 2010 stroke.ahajournals.orgDownloaded from is equivalent to d and both indices will vary with the size of r.Hsieh et al have adjusted their ES but may have overlookedthe fact that their method of adjustment of SRM to an ES isincomplete. As shown in the Table, correct transformation oftheir SRM estimates into ES suitable for Cohen’s classifica-tion resulted in a moderate effect for Fugl-Meyer Assessmentand small effects for Action Research Arm Test, Wolf MotorFunction Test–TIME, and Wolf Motor Function Test–FAS,respectively, which seems in sharp contrast with the conclusionof Hsieh et al. Extreme high correlations between baseline andfollow-up (except Wolf Motor Function Test–TIME) are mainlyresponsible for overestimation of effect size by the method ofHsieh et al. Minor deviations from published indices are due tocalculations without individual data.DisclosuresNone.Berrie Middel, PhDUniversity Medical Center GroningenDepartment of Health SciencesSubdivision of Public HealthDepartment of Oral Sciences and Clinical EpidemiologyUniversity of GroningenGroningen, The NetherlandsEric van Sonderen, PhDUniversity Medical Center GroningenDepartment of Health SciencesSubdivision of Health PsychologyUniversity of GroningenGroningen, The Netherlands1. Sivan M. Interpreting effect size to estimate responsiveness of outcomemeasures. Stroke. 2009;40:e709–e711.2. Hsieh YW, Wu CY, Lin KC, Chang YF, Chen CL, Liu JS. Responsivenessand validity of three outcome measures of motor function after strokerehabilitation. Stroke. 2009;40:1386–1391.3. Cohen J. The T Test for Means. Statistical Power Analysis for the Behav-ioural Sciences, 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates;1988:19–74.4. Dunlap WP, Cortina JM, Vaslow JB, Burke MJ. Meta-analysis of exper-iments with matched groups or repeated measures designs. PsychologicalMethods. 1996;1:170–177.5. Middel B, Van Sonderen FLP. Statistical significant change versusrelevant or important change in (quasi) experimental design: some con-ceptual and methodological problems in estimating magnitude ofintervention-related change in health services research. InternationalJournal of Integrated Care. 2002;2:1–21.6. Middel B, Van Sonderen FLP. Erratum. Statistical significant changeversus relevant or important change in (quasi) experimental design: someconceptual and methodological problems in estimating magnitude ofintervention-related change in health services research. InternationalJournal of Integrated Care. 2008;8:1–2.7. Hsieh YW, Wu CY, Lin KC. Response to Letter by Sivan. Stroke. In press.e464 Letters to the Editor at Rijksuniversiteit Groningen on October 25, 2010 stroke.ahajournals.orgDownloaded from 

https://pure.rug.nl/ws/files/2557436/Middel_2010_Stroke.pdf

Responsiveness and Validity of 3 Outcome Measures of Motor Function After Stroke Rehabilitation

Abstract

Similar works

Full text

Available Versions

University of Groningen

ARTS repository - University of Groningen

Proceedings - University of Groningen

ARTS repository - University of Groningen

University of Groningen Digital Archive

Dissertations of the University of Groningen

University of Groningen