A Study of Chance-Corrected Agreement Coefficients for the Measurement of Multi-Rater Consistency

Chance corrected agreement coefficients such as the Cohen and Fleiss Kappas are commonly used for the measurement of consistency in the decisions made by clinical observers or raters. However, the way that they estimate the probability of agreement (Pe) or cost of disagreement (De) 'by chance' has been strongly questioned, and alternatives have been proposed, such as the Aickin Alpha coefficient and the Gwet AC1 and AC2 coefficients. A well known paradox illustrates deficiencies of the Kappa coefficients which may be remedied by scaling Pe or De according to the uniformity of the scoring. The AC1 and AC2 coefficients result from the application of this scaling to the Brennan-Prediger coefficient which may be considered a simplified form of Kappa. This paper examines some commonly used multi-rater agreement coefficients including AC1 and AC2. It then proposes an alternative subject-by-subject scaling approach that may be applied to weighted and unweighted multi-rater Cohen and Fleiss Kappas and also Intra-Class Correlation (ICC) coefficients

Reformulation and Generalisation of the Cohen and Fleiss Kappas

The assessment of consistency in the categorical or ordinal decisions made by observers or raters is an important problem especially in the medical field. The Fleiss Kappa, Cohen Kappa and Intra-class Correlation (ICC), as commonly used for this purpose, are compared and a generalised approach to these measurements is presented. Differences between the Fleiss Kappa and multi-rater versions of the Cohen Kappa are explained and it is shown how both may be applied to ordinal scoring with linear, quadratic or other weighting. The relationship between quadratically weighted Fleiss and Cohen Kappa and pair-wise ICC is clarified and generalised to multi-rater assessments. The AC1 coefficient is considered as an alternative measure of consistency and the relevance of the Kappas and AC1 to measuring content validity is explore

Early social distancing policies in Europe, changes in mobility & COVID-19 case trajectories: insights from Spring 2020

Background Social distancing have been widely used to mitigate community spread of SARS-CoV-2. We sought to quantify the impact of COVID-19 social distancing policies across 27 European counties in spring 2020 on population mobility and the subsequent trajectory of disease. Methods We obtained data on national social distancing policies from the Oxford COVID-19 Government Response Tracker and aggregated and anonymized mobility data from Google. We used a pre-post comparison and two linear mixed-effects models to first assess the relationship between implementation of national policies and observed changes in mobility, and then to assess the relationship between changes in mobility and rates of COVID-19 infections in subsequent weeks. Results Compared to a pre-COVID baseline, Spain saw the largest decrease in aggregate population mobility (~70%), as measured by the time spent away from residence, while Sweden saw the smallest decrease (~20%). The largest declines in mobility were associated with mandatory stay-at-home orders, followed by mandatory workplace closures, school closures, and non-mandatory workplace closures. While mandatory shelter-in-place orders were associated with 16.7% less mobility (95% CI: -23.7% to -9.7%), non-mandatory orders were only associated with an 8.4% decrease (95% CI: -14.9% to -1.8%). Large-gathering bans were associated with the smallest change in mobility compared with other policy types. Changes in mobility were in turn associated with changes in COVID-19 case growth. For example, a 10% decrease in time spent away from places of residence was associated with 11.8% (95% CI: 3.8%, 19.1%) fewer new COVID-19 cases. Discussion This comprehensive evaluation across Europe suggests that mandatory stay-at-home orders and workplace closures had the largest impacts on population mobility and subsequent COVID-19 cases at the onset of the pandemic. With a better understanding of policies’ relative performance, countries can more effectively invest in, and target, early nonpharmacological interventions

Impacts of Social Distancing Policies on Mobility and COVID-19 Case Growth in the US

Social distancing remains an important strategy to combat the COVID-19 pandemic in the United States. However, the impacts of specific state-level policies on mobility and subsequent COVID-19 case trajectories have not been completely quantified. Using anonymized and aggregated mobility data from opted-in Google users, we found that state-level emergency declarations resulted in a 9.9% reduction in time spent away from places of residence. Implementation of one or more social distancing policies resulted in an additional 24.5% reduction in mobility the following week, and subsequent shelter-in-place mandates yielded an additional 29.0% reduction. Decreases in mobility were associated with substantial reductions in case growth 2 to 4 weeks later. For example, a 10% reduction in mobility was associated with a 17.5% reduction in case growth 2 weeks later. Given the continued reliance on social distancing policies to limit the spread of COVID-19, these results may be helpful to public health officials trying to balance infection control with the economic and social consequences of these policies.Comment: Co-first Authors: GAW, SV, VE, and AF contributed equally. Corresponding Author: Dr. Evgeniy Gabrilovich, [email protected] 32 pages (including supplemental material), 4 figures in the main text, additional figures in the supplemental materia

Measurement of Rater Consistency by Chance-Corrected Agreement Coefficients

Measurement of consistency in the decisions made by observers or raters is an important problem in clinical medicine. Chance corrected agreement coefficients such as the Cohen and Fleiss Kappas are commonly used for this purpose, though the way that they estimate the probability of agreement 'by chance' has been strongly questioned. Alternatives have been proposed, such as the Aickin Alpha coefficient and the Gwet AC1 and AC2 coefficients which are gaining currency. A well known paradox illustrates deficiencies of the Kappa coefficients which, it is claimed, are remedied by an approach which grades the subjects according to their probability of being hard to score. The AC1 and AC2 coefficients result from the application of this grading to the Brennan-Prediger coefficient which may be considered a simplified form of Kappa. This paper questions the rationale of the hardness probability used by AC1 and proposes an alternative approach that may be applied to weighted and unweighted multi-rater Cohen and Fleiss Kappas and also Intra-Class Correlation (ICC) coefficients

Hyo-Mental Angle and Distance: An Important Adjunct in Airway Assessment of Adult Mucopolysaccharidosis

Background: Mucopolysaccharidosis (MPS) is a rare congenital lysosomal storage disorder with complex airways. High anterior larynx is assessed by thyromental distance (TMD) nasendoscopy. A simpler method to assess this hyoid bone is described. The distance between the central-hyoid and symphysis of the mandible (hyo-mental distance; HMD) and inclination of this line to the horizontal axis (hyo-mental angle; HMA) in neutrally positioned patients is investigated. Methods: HMA, HMD in MPS, and non-MPS were compared, and their correlation with height and weight were assessed. Results: 50 adult MPS patients (M = 32, F = 18, age range = 19–66 years; mean BMI = 26.8 kg/m2) of MPS I, II, III, IV, and VI were compared with 50 non-MPS (M = 25, F = 25; age range = 22–84 years; mean BMI = 26.5 kg/m2). Mean HMA in MPS was 25.72° (−10 to +50) versus 2.42° (−35 to +28) in non-MPS. Mean HMD was 46.5 (25.7–66) millimeters in MPS versus 41.8 (27–60.3) in non-MPS. HMA versus height and weight showed a moderate correlation (r = −0.4, p < 0.05) in MPS and no significant correlation (r < 0.4, p > 0.05) in non-MPS. HMD versus height and weight showed no correlation (r < 0.4, p > 0.05) in both groups. Conclusions: HMA seems more acute in MPS despite nearly the same HMD as non-MPS, signifying a high larynx, which may be missed by TMD