Comparison of Cohen's Kappa and Gwet's AC1 with a mass shooting classification index: A study of rater uncertainty

In order to quantify the degree of agreement between raters when classifying subjects into predefined categories, inter-rater reliability (IRR) experiments are often conducted in the medical field. Originally, percent agreement was used to calculate the extent of agreement between raters; however, it was criticized for not taking into account chance-agreement. Chance-agreement refers to the propensity for raters to guess when classifying nondeterministic subjects to categories. In other words, raters can be certain that some subjects are textbook and are associated with a true category membership, whereas, other subjects are ambiguous and require true random guessing (Schuster & Smith, 2002). A commonly used chance-corrected agreement coefficient has been Cohen's Kappa. Limitations have been associated with the Kappa statistic such as Kappa's tendency to overcorrect for chance-agreement in the presence of high prevalence rates (i.e., highly skewed data). Due to such issues, Gwet (2014) proposed a new chance-corrected agreement coefficient called the AC1 statistic. The purpose of this study was to examine Cohen's Kappa and Gwet's AC1 with respect to prevalence rates and rater uncertainty using a newly developed classification system for mass shooters. A new methodology for identifying textbook and ambiguous subjects was demonstrated. Specifically, the purposes of the present study were (1) to examine how Cohen's Kappa and Gwet's AC1 are affected by prevalence rates and (2) to determine whether there are differences in the observable discrepancies between Cohen's Kappa and Gwet's AC1 for subjects classified as textbook compared to subjects classified as ambiguous. Findings indicated that observable discrepancies between Cohen's Kappa and Gwet's AC1 could be seen in both the textbook and ambiguous conditions. Specifically, analyses suggested that percent agreement was likely to overestimate the extent of true agreement among raters and Cohen's Kappa was likely to underestimate the extent of true agreement among raters. The ambiguous analysis revealed larger discrepancies between Gwet's AC1 and Cohen's Kappa in the presence of highly skewed data, however, discrepancies between Gwet's AC1 and Cohen's Kappa appeared to be more dependent on the number of observable disagreements between raters during the textbook analysis. Recommendations for practice and future research are discussed

Testing a continuum structure of self-determined motivation: A meta-analysis

Self-determination theory proposes a multidimensional representation of motivation comprised of several factors said to fall along a continuum of relative autonomy. The current meta-analysis examined the relationships between these motivation factors in order to demonstrate how reliably they conformed to a predictable continuum-like pattern. Based on data from 486 samples representing over 205,000 participants who completed 1 of 13 validated motivation scales, the results largely supported a continuum-like structure of motivation and indicate that self-determination is central in explaining human motivation. Further examination of heterogeneity indicated that while regulations were predictably ordered across domains and scales, the exact distance between subscales varied across samples in a way that was not explainable by a set of moderators. Results did not support the inclusion of integrated regulation or the 3 subscales of intrinsic motivation (i.e., intrinsic motivation to know, to experience stimulation, and to achieve) due to excessively high interfactor correlations and overlapping confidence intervals. Recommendations for scale refinements and the scoring of motivation are provided

A Critical Evaluation of Remote Sensing Based Land Cover Mapping Methodologies

A novel, disaggregated approach to land cover survey is developed on the basis of land cover attributes; the parameters typically used to delineate land cover classes. The recording of land cover attributes, via objective measurement techniques, is advocated as it eliminates the requirement for surveyors to delineate and classify land cover; a process proven to be subjective and error prone. Within the North York Moors National Park, a field methodology is developed to characterise five attributes: species composition, cover, height, structure and density. The utility of land cover attributes to act as land cover ‘building blocks’ is demonstrated via classification of the field data to the Monitoring Landscape Change in the National Parks (MLCNP), National Land Use Database (NLUD) and Phase 1 Habitat Mapping (P1) schemes. Integration of the classified field data and a SPOT5 satellite image is demonstrated within per-pixel and object-orientated classification environments. Per-pixel classification produced overall accuracies of 81%, 80% and 76% at the field samples for the MLCNP, NLUD and P1 schemes, respectively. However, independent validation produced significantly lower accuracies. These decreases are demonstrated to be a function of sample fraction. Object-orientated classification, exemplified for the MLCNP schema at 3 segmentation scales, achieved accuracies approaching 75%. The aggregation of attributes to classes underutilises the potential of the remotely sensed data to describe landscape variability. Consequently, classification and geostatistical techniques capable of land cover attribute parameterisation, across the study area, are reviewed and exemplified for a sub-pixel classification. Land cover attributes provide a flexible source of field data which has been proven to support multiple land cover classification schemes and classification scales (sub-pixel, pixel and object). This multi-scaled/schemed approach enables the differential treatment of regions, within the remote sensing image, as a function of landscape characteristics and the users’ requirements providing a flexible mapping solution