Reproducibility and responsiveness of evaluative outcome measures - Theoretical considerations illustrated by an empirical example

Objective: This article outlines basic methods for quantifying reproducibility and responsiveness of evaluative outcome measures. Methods: The background noise in stable patients provides the desired information to quantify the reproducibility. From this, the smallest real difference (SRD) for longitudinal differences can be derived. We propose to use the SRD to define responsiveness: from all patients who change according to an external criterion, we take the percentage that changes at least SRD on the outcome measure. A more complete picture of the responsiveness of the outcome measure arises when the receiver operating characteristic (ROC) is constructed, corresponding to all possible cut-off change scores. The proposed methods are illustrated with an empirical example. Results: In the illustration the methods appeared to be very useful and complemented each other. We could evaluate whether the observed change score was larger than that expected due to chance. With the methods it was possible to evaluate both the ability of an instrument to detect change if there is a real change in health status (sensitivity to change) and the ability to detect absence of change if there is no real change (specificity to change). Conclusion: We presented the use of SRDs and ROC curves for quantifying reproducibility and responsiveness. We started with the basic notions and arrived at methods that are both understandable and useful

A method for research programming in the field of evidence-based medicine

Objective: This paper presents the method we used in programming research on the efficacy of therapeutic interventions for nine chronic benign pain disorders. Methods: We started with an inventory to identify commonly applied interventions. For these interventions we searched the literature to identify gaps in evidence. First, we searched for recent reviews, of which we assessed the methodologic quality. If only reviews of poor or moderate quality were found, a new systematic review was recommended. When we found no recent reviews, we searched for randomized controlled trials (RCTs). If there was insufficient or inconsistent evidence, the performance of a new RCT was advised. This structured procedure resulted in a list with topics for which new systematic reviews were recommended and topics for which new RCTs were needed. These lists were the starting points for priority setting by four Centers for Pain Management and Research. All members of the centers were asked to state the priority of each topic. The resulting hierarchy of topics for their own center was discussed in a meeting in each center, giving participants the opportunity to elucidate their views and considerations. Results: The final result was a robust priority list for the need of research (systematic reviews and RCTs) on chronic benign pain syndromes. Discussion: The strength and weaknesses of this approach are discussed. This method of priority setting is by no means restricted to treatments for chronic pain; it is also a useful approach for programming research to enable evidence-based medicine in other fields of interest

Measuring sick leave: a comparison of self-reported data on sick leave and data from company records

The objective of this study was to compare sick leave data obtained from questionnaires with data from company records. During a period of 12 months, questionnaires were completed monthly for 6 months and then at 9 and 12 months. The sensitivity and specificity of questionnaires for detecting an episode of sick leave were determined, using the company records as a reference standard. In addition, the duration of sick leave episodes reported in the two data sets was compared. In this analysis, company records were not assumed to be superior, and agreement was assessed with intraclass correlation coefficients (ICCs). The sensitivity of questionnaires for detecting an episode of sick leave was 55% (95% CI = 0.50-0.60) and the specificity 83% (95% CI = 0.72-0.94). The ICC for all episodes was 0.58 (95% CI = 0.47-0.67). The only satisfactory ICC (0.87; 95% CI = 0.74-0.93) was found for the questionnaires at 9 and 12 months. No large systematic differences were found between the duration of episodes reported in the two data sets. In conclusion, in our study, the sensitivity of questionnaires for detecting an episode of sick leave was very low. Furthermore, when episodes were recalled, there was little agreement on the duration of the episode between questionnaire data and data in the company records. Based on these results and considering the risk of missing questionnaires, data on sick leave gathered from company records are clearly preferable as an outcome measure in research

Splinting vs surgery in the treatment of carpal tunnel syndrome - A randomized controlled trial

Carpal tunnel syndrome (CTS) can be treated with nonsurgical or surgical options. However, there is no consensus on the most effective method of treatment. To compare the short-term and long-term efficacy of splinting and surgery for relieving the symptoms of CTS. A randomized controlled trial conducted from October 1998 to April 2000 at 13 neurological outpatient clinics in the Netherlands. A total of 176 patients with clinically and electrophysiologically confirmed idiopathic CTS were assigned to wrist splinting during the night for at least 6 weeks (89 patients) or open carpal tunnel release (87 patients); 147 patients (84%) completed the final follow-up assessment 18 months after randomization. General improvement, number of nights waking up due to symptoms, and severity of symptoms. In the intention-to-treat analyses, surgery was more effective than splinting on all outcome measures. The success rates (based on general improvement) after 3 months were 80% for the surgery group (62/78 patients) vs 54% for the splinting group (46/86 patients), which is a difference of 26% (95% confidence interval [CI], 12%-40%; P <.001). After 18 months, the success rates increased to 90% for the surgery group (61/68 patients) vs 75% for the splinting group (59/79 patients), which is a difference of 15% (95% CI, 3%-27%; P =.02). However, by that time 41% of patients (32/79) in the splint group had also received the surgery treatment. Treatment with open carpal tunnel release surgery resulted in better outcomes than treatment with wrist splinting for patients with CT

Conservative treatment options for carpal tunnel syndrome: a systematic review of randomised controlled trials

Carpal tunnel syndrome (CTS) is a common disorder, for which various conservative treatment options are available. The objective of this study is to determine the efficacy of the various conservative treatment options for relieving the symptoms of CTS. Computer-aided searches of MEDLINE (1/1966 to 3/2000), EMBASE (1/1988 to 2/2000) and the Cochrane Controlled Trials Register (2000, issue 1) were conducted, together with reference checking. Included were randomised controlled trials evaluating the efficacy of conservative treatment options in a study population of CTS patients, with a full report published in English, German, French or Dutch. Two reviewers independently selected the studies. Fourteen randomised controlled trials were included in the review. Assessment of methodological quality and data-extraction was independently performed by two reviewers. A rating system, based on the number of studies and their methodological quality and findings, was used to determine the strength of the available evidence for the efficacy of the treatment. Diuretics, pyridoxine, non-steroidal anti-inflammatory drugs, yoga and laser-acupuncture seem to be ineffective in providing short-term symptom relief (varying levels of evidence) and steroid injections seem to be effective (limited evidence). There is conflicting evidence for the efficacy of ultrasound and oral steroids. For providing long-term relief from symptoms there is limited evidence that ultrasound is effective, and that splinting is less effective than surgery. In conclusion, there is still little known about the efficacy of most conservative treatment options for CTS. To establish stronger evidence more high quality trials are neede

Clinimetric properties of the motor activity log for the assessment of arm use in hemiparetic patients

Background and Purpose - The Motor Activity Log (MAL) is a semistructured interview for hemiparetic stroke patients to assess the use of their paretic arm and hand (amount of use [AOU]) and quality of movement [QOM]) during activities of daily living. Scores range from 0 to 5. The following clinimetric properties of the MAL were quantified: internal consistency (Cronbach α), test-retest agreement (Bland and Altman method), cross-sectional construct validity (correlation between AOU and QOM and with the Action Research Arm (ARA] test), longitudinal construct validity (correlation of change on the MAL during the intervention with a global change rating [GCR] and with change on the ARA), and responsiveness (effect size). Methods - Two baseline measurements 2 weeks apart and 1 follow-up measurement immediately after 2 weeks of intensive exercise therapy either with or without immobilization of the unimpaired arm (forced use) were performed in 56 chronic stroke patients. Results - Internal consistency was high (AOU: a=0.88; QOM: a=0.91). The limits of agreement were -0.70 to 0.85 and -0.61 to 0.71 for AOU and QOM, respectively. The correlation with the ARA score (Spearman p) was 0.63 (AOU and QOM). However, the improvement on the MAL during the intervention was only weakly related to the GCR and to the improvement on the ARA, Spearman p was between 0.16 and 0.22. The responsiveness ratio was 1.9 (AOU) and 2.0 (QOM). Conclusion - The MAL is internally consistent and relatively stable in chronic stroke patients not undergoing an intervention. The cross-sectional construct validity of the MAL is reasonable, but the results raise doubt about its longitudinal construct validity

The usefulness of evaluative outcome measures in patients with multiple sclerosis

To select the most useful evaluative outcome measures for early multiple sclerosis, we included 156 recently diagnosed patients in a 3-year follow-up study, and assessed them on 23 outcome measures in the domains of disease-specific outcomes, physical functioning, mental health, social functioning and general health. A global rating scale (GRS) and the Expanded Disability Status Scale (EDSS) were used as external criteria to determine the minimally important change (MIC) for each outcome measure. Subsequently, we determined whether the outcome measures could detect their MIC reliably. From these, per domain the outcome measure that was found to be most sensitive to changes (responsive) was identified. At group level, 11 outcomes of the domains of physical functioning, mental health, social functioning and general health could reliably detect the MIC. Of these 11, the most responsive measures per domain were the Medical Outcome Study 36 Short Form sub-scale physical functioning (SF36pf), the Disability and Impact Profile (DIP) sub-scale psychological, the Rehabilitation Activities Profile sub-scale occupation (RAPocc) and the SF36 sub-scale health, respectively. Overall, the most responsive measures were the SF36pf and the RAPocc. In individual patients, none of the measures could reliably detect the MIC. In sum, in the early stages of multiple sclerosis the most useful evaluative outcome measures for research are the SF36pf (physical functioning) and the RAPocc (social functioning). © The Author (2006). Published by Oxford University Press on behalf of the Guarantors of Brain. All rights reserved