The intra- and interrater reliability of the action research arm test: A practical test of upper extremity function in patients with stroke

Objectives: To determine the intra- and interrater reliability of the Action Research Arm (ARA) test, to assess its ability to detect a minimal clinically important difference (MCID) of 5.7 points, and to identify less reliable test items. Design: Intrarater reliability of the sum scores and of individual items was assessed by comparing (1) the ratings of the laboratory measurements of 20 patients with the ratings of the same measurements recorded on videotape by the original rater, and (2) the repeated ratings of videotaped measurements by the same rater. Interrater reliability was assessed by comparing the ratings of the videotaped measurements of 2 raters. The resulting limits of agreement were compared with the MCID. Patients: Stratified sample, based on the intake ARA score, of 20 chronic stroke patients (median age, 62yr; median time since stroke onset, 3.6yr; mean intake ARA score, 29.2). Main Outcome Measures: Spearman's rank-order correlation coefficient (Spearman's rho); intraclass correlation coefficient (ICC); mean difference and limits of agreement, based on ARA sum scores; and weighted kappa, based on individual items. Results: All intra- and interrater Spearman's rho and ICC values were higher than .98. The mean difference between ratings was highest for the interrater pair (.75; 95% confidence interval, .02-1.48), suggesting a small systematic difference between raters. Intrarater limits of agreement were -1.66 to 2.26; interrater limits of agreement were -2.35 to 3.85. Median weighted kappas exceeded .92. Conclusion: The high intra- and interrater reliability of the ARA test was confirmed, as was its ability to detect a clinically relevant difference of 5.7 points

How is your mind-set? Proof of concept for the measurement of the level of emotional development

Background In persons with intellectual and developmental disabilities, not only cognitive brain functions, but also socio-emotional processing networks may be impaired. This study aims to validate the Scale of Emotional Development—Short (SED-S) to provide an instrument for the assessment of socio-emotional brain functions. Method The SED-S was applied in 160 children aged 0–12 years. Criterion validity was investigated at item and scale level in terms of the agreement between the scale classification and the child’s chronological age. Additionally, interrater reliability and internal consistency were assessed. Results For the majority of items, the expected response pattern emerged, showing the highest response probabilities in the respective target age groups. Agreement between the classification of the different SED-S domains and chronological age was high (κw = 0.95; exact agreement = 80.6%). Interrater reliability at domain level ranged from κw = .98 to 1.00 and internal consistency was high (α = .99). Conclusion The study normed the SED-S in a sample of typically developing children and provides evidence for criterion validity on item, domain and scale level

Validation of a novel scoring system for changes in skeletal manifestations of hypophosphatasia in newborns, infants, and children: The Radiographic Global Impression of Change scale

Hypophosphatasia (HPP) is the heritable metabolic disease characterized by impaired skeletal mineralization due to low activity of the tissue-nonspecific isoenzyme of alkaline phosphatase. Although HPP during growth often manifests with distinctive radiographic skeletal features, no validated method was available to quantify them, including changes over time. We created the Radiographic Global Impression of Change (RGI-C) scale to assess changes in the skeletal burden of pediatric HPP. Site-specific pairs of radiographs of newborns, infants, and children with HPP from three clinical studies of asfotase alfa, an enzyme replacement therapy for HPP, were obtained at baseline and during treatment. Each pair was scored by three pediatric radiologists ( raters ), with nine raters across the three studies. Intrarater and interrater agreement was determined by weighted Kappa coefficients. Interrater reliability was assessed using intraclass correlation coefficients (ICCs) and by two-way random effects analysis of variance (ANOVA) and a mixed-model repeated measures ANOVA. Pearson correlation coefficients evaluated relationships of the RGI-C to the Rickets Severity Scale (RSS), Pediatric Outcomes Data Collection Instrument Global Function Parent Normative Score, Childhood Health Assessment Questionnaire Disability Index, 6-Minute Walk Test percent predicted, and Z-score for height in patients aged 6 to 12 years at baseline. Eighty-nine percent (8/9) of raters showed substantial or almost perfect intrarater agreement of sequential RGI-C scores (weighted Kappa coefficients, 0.72 to 0.93) and moderate or substantial interrater agreement (weighted Kappa coefficients, 0.53 to 0.71) in patients aged 0 to 12 years at baseline. Moderate-to-good interrater reliability was observed (ICC, 0.57 to 0.65). RGI-C scores were significantly (p ≤ 0.0065) correlated with the RSS and with measures of global function, disability, endurance, and growth in the patients aged 6 to 12 years at baseline. Thus, the RGI-C is valid and reliable for detecting clinically important changes in skeletal manifestations of severe HPP in newborns, infants, and children, including during asfotase alfa treatment. © 2018 The Authors. Journal of Bone and Mineral Research Published by Wiley Periodicals Inc

Interrater reliability of surveillance for ventilator-associated events and pneumonia

OBJECTIVETo compare interrater reliabilities for ventilator-associated event (VAE) surveillance, traditional ventilator-associated pneumonia (VAP) surveillance, and clinical diagnosis of VAP by intensivists.DESIGNA retrospective study nested within a prospective multicenter quality improvement study.SETTINGIntensive care units (ICUs) within 5 hospitals of the Centers for Disease Control and Prevention Epicenters.PATIENTSPatients who underwent mechanical ventilation.METHODSWe selected 150 charts for review, including all VAEs and traditionally defined VAPs identified during the primary study and randomly selected charts of patients without VAEs or VAPs. Each chart was independently reviewed by 2 research assistants (RAs) for VAEs, 2 hospital infection preventionists (IPs) for traditionally defined VAP, and 2 intensivists for any episodes of pulmonary deterioration. We calculated interrater agreement using κ estimates.RESULTSThe 150 selected episodes spanned 2,500 ventilator days. In total, 93–96 VAEs were identified by RAs; 31–49 VAPs were identified by IPs, and 29–35 VAPs were diagnosed by intensivists. Interrater reliability between RAs for VAEs was high (κ, 0.71; 95% CI, 0.59–0.81). Agreement between IPs using traditional VAP criteria was slight (κ, 0.12; 95% CI, −0.05–0.29). Agreement between intensivists was slight regarding episodes of pulmonary deterioration (κ 0.22; 95% CI, 0.05–0.39) and was fair regarding whether episodes of deterioration were attributable to clinically defined VAP (κ, 0.34; 95% CI, 0.17–0.51). The clinical correlation between VAE surveillance and intensivists’ clinical assessments was poor.CONCLUSIONSProspective surveillance using VAE criteria is more reliable than traditional VAP surveillance and clinical VAP diagnosis; the correlation between VAEs and clinically recognized pulmonary deterioration is poor.Infect Control Hosp Epidemiol 2017;38:172–178</jats:sec

Manual muscle testing and hand-held dynamometry in people with inflammatory myopathy : an intra- and interrater reliability and validity study

Manual muscle testing (MMT) and hand-held dynamometry (HHD) are commonly used in people with inflammatory myopathy (IM), but their clinimetric properties have not yet been sufficiently studied. To evaluate the reliability and validity of MMT and HHD, maximum isometric strength was measured in eight muscle groups across three measurement events. To evaluate reliability of HHD, intra-class correlation coefficients (ICC), the standard error of measurements (SEM) and smallest detectable changes (SDC) were calculated. To measure reliability of MMT linear Cohen`s Kappa was computed for single muscle groups and ICC for total score. Additionally, correlations between MMT8 and HHD were evaluated with Spearman Correlation Coefficients. Fifty people with myositis (56±14 years, 76% female) were included in the study. Intra-and interrater reliability of HHD yielded excellent ICCs (0.75-0.97) for all muscle groups, except for interrater reliability of ankle extension (0.61). The corresponding SEMs% ranged from 8 to 28% and the SDCs% from 23 to 65%. MMT8 total score revealed excellent intra-and interrater reliability (ICC>0.9). Intrarater reliability of single muscle groups was substantial for shoulder and hip abduction, elbow and neck flexion, and hip extension (0.64-0.69); moderate for wrist (0.53) and knee extension (0.49) and fair for ankle extension (0.35). Interrater reliability was moderate for neck flexion (0.54) and hip abduction (0.44); fair for shoulder abduction, elbow flexion, wrist and ankle extension (0.20-0.33); and slight for knee extension (0.08). Correlations between the two tests were low for wrist, knee, ankle, and hip extension; moderate for elbow flexion, neck flexion and hip abduction; and good for shoulder abduction. In conclusion, the MMT8 total score is a reliable assessment to consider general muscle weakness in people with myositis but not for single muscle groups. In contrast, our results confirm that HHD can be recommended to evaluate strength of single muscle groups

Reliability of 4-m and 6-m walking speed tests in elderly people with cognitive impairment

[Abstract] The purpose of this study was to evaluate the interrater and test–retest reliability of 4-m and 6-m walking speed tests in elderly people with cognitive impairment. 50 subjects aged 65 and over with cognitive impairment were selected from an adult day-care centre and a nursing home. To examine interrater reliability, 21 people were evaluated independently by two researchers who administered the 4-m and 6-m walking tests in each evaluation. For test–retest reliability, two observers administered the tests to the same 29 subjects, with a time interval of one week. Intraclass correlation coefficients (ICCs) were calculated to examine interrater and test–retest reliability. The ICCs for interrater reliability reached values of 0.96 and 0.88 for the 4-m and 6-m walking tests, respectively. In the test–retest study, the time interval was 7.4 1.17 days. The ICCs were 0.91 for the 4-m test and 0.86 for the 6-m test. The results support the use of walking tests in elderly people with cognitive impairment

The Validity, Reliability, Measurement Error, and Minimum Detectable Change of the 30‐Second Fast‐Paced Walk Test in Persons with Knee Osteoarthritis: A Novel Test of Short‐Distance Walking Ability

Objective To develop and establish the reliability, validity, measurement error, and minimum detectable change of a novel 30‐second fast‐paced walk test (30SFW) in persons with knee osteoarthritis (OA) that is easy to administer and can quantify walking performance in persons of all abilities. Methods Twenty females with symptomatic knee OA (mean age [SD] 58.30 [8.05] years) and 20 age‐ and sex‐matched asymptomatic controls (57.25 [8.71] years) participated in the study. Participants completed questionnaires of demographic and clinical data, the Knee Injury and Osteoarthritis Outcome Score (KOOS), and the 36‐item Short Form Health Survey (SF‐36) followed by 30SFW performance. Participants returned 2‐7 days later and performed the 30SFW again. Results The knee OA group reported function that was worse than controls (all KOOS subscales; P \u3c 0.0001). The 30SFW intrarater and interrater reliability were excellent [ICC(2,1) = 0.95‐0.99]. Knee OA participants walked a shorter distance in the 30SFW than controls (mean [SD]: OA 44.4 m [9.5 m]; control 58.1 m [7.8 m]; P \u3c 0.0001). Positive strong correlations were found between the 30SFW and the KOOS–Activity of Daily Living, SF‐36‐Physical Functioning, and SF‐36‐Physical Health Composite scores (P \u3c 0.0001). A nonsignificant, weak correlation between 30SFW and SF‐36‐Mental Health scores was present (r = 0.32, P = 0.05). Conclusion The 30SFW has excellent intrarater and interrater reliability. The 30SFW demonstrated excellent known groups, convergent, and discriminant validity as a measure of short‐distance walking ability in persons with knee OA. Clinicians and researchers should consider using the 30SFW to quantify walking ability in persons with knee OA and assess walking ability change

Measurement Error in Research on Human Resources and Firm Performance: Additional Data and Suggestions for Future Research

Gerhart and colleagues and Huselid and Becker recently debated the presence and implications of measurement error in measures of human resource practices. This paper presents data from three more studies, one of large organizations from different industries at the corporate level, one from commercial banks, and the other of autonomous business units at the level of the job. Results of all three studies provide additional evidence that single respondent measures of HR practices contain large amounts of measurement error. Implications for future research are discussed

The diagnosis of mental disorders: the problem of reification

A pressing need for interrater reliability in the diagnosis of mental disorders emerged during the mid-twentieth century, prompted in part by the development of diverse new treatments. The Diagnostic and Statistical Manual of Mental Disorders (DSM), third edition answered this need by introducing operationalized diagnostic criteria that were field-tested for interrater reliability. Unfortunately, the focus on reliability came at a time when the scientific understanding of mental disorders was embryonic and could not yield valid disease definitions. Based on accreting problems with the current DSM-fourth edition (DSM-IV) classification, it is apparent that validity will not be achieved simply by refining criteria for existing disorders or by the addition of new disorders. Yet DSM-IV diagnostic criteria dominate thinking about mental disorders in clinical practice, research, treatment development, and law. As a result, the modernDSMsystem, intended to create a shared language, also creates epistemic blinders that impede progress toward valid diagnoses. Insights that are beginning to emerge from psychology, neuroscience, and genetics suggest possible strategies for moving forward