Examining intra-rater and inter-rater response agreement: A medical chart abstraction study of a community-based asthma care program

<p>Abstract</p> <p>Background</p> <p>To assess the intra- and inter-rater agreement of chart abstractors from multiple sites involved in the evaluation of an Asthma Care Program (ACP).</p> <p>Methods</p> <p>For intra-rater agreement, 110 charts randomly selected from 1,433 patients enrolled in the ACP across eight Ontario communities were re-abstracted by 10 abstractors. For inter-rater agreement, data abstractors reviewed a set of eight fictitious charts. Data abstraction involved information pertaining to six categories: physical assessment, asthma control, spirometry, asthma education, referral visits, and medication side effects. Percentage agreement and the kappa statistic (κ) were used to measure agreement. Sensitivity and specificity estimates were calculated comparing results from all raters against the gold standard.</p> <p>Results</p> <p>Intra-rater re-abstraction yielded an overall kappa of 0.81. Kappa values for the chart abstraction categories were: physical assessment (κ 0.84), asthma control (κ 0.83), spirometry (κ 0.84), asthma education (κ 0.72), referral visits (κ 0.59) and medication side effects (κ 0.51). Inter-rater abstraction of the fictitious charts produced an overall kappa of 0.75, sensitivity of 0.91 and specificity of 0.89. Abstractors demonstrated agreement for physical assessment (κ 0.88, sensitivity and specificity 0.95), asthma control (κ 0.68, sensitivity 0.89, specificity 0.85), referral visits (κ 0.77, sensitivity 0.88, specificity 0.95), and asthma education (κ 0.49, sensitivity 0.87, specificity 0.77).</p> <p>Conclusion</p> <p>Though collected by multiple abstractors, the results show high sensitivity and specificity and substantial to excellent inter- and intra-rater agreement, assuring confidence in the use of chart abstraction for evaluating the ACP.</p

A structured registration program can be validly used for quality assessment in general practice

ABSTRACT: BACKGROUND: Patient information, medical history, clinical outcomes and demographic information, can be registered in different ways in registration programs. For evaluation of diabetes care, data can easily be extracted from a structured registration program (SRP). The usability of data from this source depends on the agreement of this data with that of the usual data registration in the electronic medical record (EMR). Aim of the study was to determine the comparability of data from an EMR and from an SRP, to determine whether the use of SRP data for quality assessment is justified in general practice. METHODS: We obtained 196 records of diabetes mellitus patients in a sample of general practices in the Netherlands. We compared the agreement between the two programs in terms of laboratory and non-laboratory parameters. Agreement was determined by defining accordance between the programs in absent and present registrations, accordance between values of registrations, and whether the differences found in values were also a clinically relevant difference. RESULTS: No differences were found in the occurrence of registration (absent/present) in the SRP and EMR for all the laboratory parameters. Smoking behaviour, weight and eye examination were registered significantly more often in the SRP than in the EMR. In the EMR, blood pressure was registered significantly more often than in the SRP. Data registered in the EMR and in the SRP had a similar clinical meaning for all parameters (laboratory and non-laboratory). CONCLUSIONS: Laboratory parameters showed good agreement and non-laboratory acceptable agreement of the SRP with the EMR. Data from a structured registration program can be used validly for research purposes and quality assessment in general practice

Dealing with heterogeneity of treatment effects: is the literature up to the challenge?

<p>Abstract</p> <p>Background</p> <p>Some patients will experience more or less benefit from treatment than the averages reported from clinical trials; such variation in therapeutic outcome is termed heterogeneity of treatment effects (HTE). Identifying HTE is necessary to individualize treatment. The degree to which heterogeneity is sought and analyzed correctly in the general medical literature is unknown. We undertook this literature sample to track the use of HTE analyses over time, examine the appropriateness of the statistical methods used, and explore the predictors of such analyses.</p> <p>Methods</p> <p>Articles were selected through a probability sample of randomized controlled trials (RCTs) published in <it>Annals of Internal Medicine</it>, <it>BMJ</it>, <it>JAMA</it>, <it>The Lancet</it>, and <it>NEJM </it>during odd numbered months of 1994, 1999, and 2004. RCTs were independently reviewed and coded by two abstractors, with adjudication by a third. Studies were classified as reporting: (1) HTE analysis, utilizing a formal test for heterogeneity or treatment-by-covariate interaction, (2) subgroup analysis only, involving no formal test for heterogeneity or interaction; or (3) neither. Chi-square tests and multiple logistic regression were used to identify variables associated with HTE reporting.</p> <p>Results</p> <p>319 studies were included. Ninety-two (29%) reported HTE analysis; another 88 (28%) reported subgroup analysis only, without examining HTE formally. Major covariates examined included individual risk factors associated with prognosis, responsiveness to treatment, or vulnerability to adverse effects of treatment (56%); gender (30%); age (29%); study site or center (29%); and race/ethnicity (7%). Journal of publication and sample size were significant independent predictors of HTE analysis (p < 0.05 and p < 0.001, respectively).</p> <p>Conclusion</p> <p>HTE is frequently ignored or incorrectly analyzed. An iterative process of exploratory analysis followed by confirmatory HTE analysis will generate the data needed to facilitate an individualized approach to evidence-based medicine.</p

Effects of non-steroidal anti-inflammatory drugs on cancer sites other than the colon and rectum: a meta-analysis

BACKGROUND: Observational studies have consistently shown that aspirin and non-steroidal anti-inflammatory drug (NSAID) use is associated with a close to 50% reduced risk of colorectal cancer. Studies assessing the effects of NSAIDs on other cancers have shown conflicting results. Therefore, we conducted a meta-analysis to evaluate the relationship between NSAID use and cancer other than colorectal. METHODS: We performed a search in Medline (from 1966 to 2002) and identified a total of 47 articles (13 cohort and 34 case-control studies). Overall estimates of the relative risk (RR) were calculated for each cancer site using random effects models. RESULTS: Aspirin use was associated with a reduced risk of cancer of the esophagus and the stomach (RR, 0.51; 95%CI (0.38–0.69), and 0.73; 95%CI (0.63–0.84)). Use of NSAIDs was similarly associated with a lower risk of esophageal and gastric cancers (RR,0.65; 95% CI(0.46–0.92) and RR,0.54; 95%CI (0.39–0.75)). Among other cancers, only the results obtained for breast cancer were fairly consistent in showing a slight reduced risk among NSAID and aspirin users (RR, 0.77; 95%CI (0.66–0.88), and RR, 0.77; 95%CI (0.69–0.86) respectively)). CONCLUSIONS: The results of this meta-analysis show that the potential chemopreventive role of NSAIDs in colorectal cancer might be extended to other gastrointestinal cancers such as esophagus and stomach. Further research is required to evaluate the role of NSAIDs at other cancers sites