Diagnostic Value of Run Chart Analysis: Using Likelihood Ratios to Compare Run Chart Rules on Simulated Data Series

<div><p>Run charts are widely used in healthcare improvement, but there is little consensus on how to interpret them. The primary aim of this study was to evaluate and compare the diagnostic properties of different sets of run chart rules. A run chart is a line graph of a quality measure over time. The main purpose of the run chart is to detect process improvement or process degradation, which will turn up as non-random patterns in the distribution of data points around the median. Non-random variation may be identified by simple statistical tests including the presence of unusually long runs of data points on one side of the median or if the graph crosses the median unusually few times. However, there is no general agreement on what defines “unusually long” or “unusually few”. Other tests of questionable value are frequently used as well. Three sets of run chart rules (Anhoej, Perla, and Carey rules) have been published in peer reviewed healthcare journals, but these sets differ significantly in their sensitivity and specificity to non-random variation. In this study I investigate the diagnostic values expressed by likelihood ratios of three sets of run chart rules for detection of shifts in process performance using random data series. The study concludes that the Anhoej rules have good diagnostic properties and are superior to the Perla and the Carey rules.</p></div

Example run charts with and without a shift in sample mean.

<p>Both charts have 24 useful observations, that is, data points not on the median. The median is calculated from the first 12 data points (baseline). <b>A</b>: No shift. The longest run has 3 data points, and the curve crosses the median 13 times. Only random variation is identified. <b>B</b>: A shift in sample mean of 2 SD was introduced in the last 12 data points. The longest run has 13 data points, which is above the signal limit of the Anhoej rules (8), and there are 6 crossing, which is below the signal limit (8), thus, non-random variation is identified. The plots were created with the qicharts package for R [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0121349#pone.0121349.ref004" target="_blank">4</a>].</p

Likelihood ratios of run chart rules.

<p>The bars show the positive (LR+) and negative (LR-) likelihood ratios for each combination of run chart rules and baseline and post-baseline length shown in the panel header. Each panel is based on 2000 simulations of which 1000 had a shift of 2 SD in sample mean introduced in the post-baseline period.</p

Diagnostic properties of run chart rules based on the results from Table 2.

<p>Diagnostic properties of run chart rules based on the results from <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0121349#pone.0121349.t002" target="_blank">Table 2</a>.</p

Results from runs analyses of 2000 simulated run charts with 24 data points and a shift of 2 SD introduced in the last 12 samples of half the simulations.

<p>Results from runs analyses of 2000 simulated run charts with 24 data points and a shift of 2 SD introduced in the last 12 samples of half the simulations.</p

Smooth operator: Modifying the Anhøj rules to improve runs analysis in statistical process control.

INTRODUCTION:The run chart is one form of statistical process control chart that is particularly useful for detecting persistent shifts in data over time. The Anhøj rules test for shifts by looking for unusually long runs (L) of data points on the same side of the process centre (mean or median) and unusually few crossings (C) of the centre depending on the number of available data points (N). Critical values for C and L have mainly been studied in isolation. But what is really of interest is the joint distribution of C and L, which has so far only been studied using simulated data series. We recently released an R package, crossrun that calculates exact values for the joint probabilities of C and L that allowed us to study the diagnostic properties of the Anhøj rules in detail and to suggest minor adjustments to improve their diagnostic value. METHODS:Based on the crossrun R package we calculated exact values for the joint distribution of C and L for N = 10-100. Furthermore, we developed two functions, bestbox() and cutbox() that automatically seek to adjust the critical values for C and L to balance between sensitivity and specificity requirements. RESULTS:Based on exact values for the joint distribution of C and L for N = 10-100 we present measures of the diagnostic value of the Anhøj rules. The best box and cut box procedures improved the diagnostic value of the Anhøj rules by keeping the specificity and sensitivity close to pre-specified target values. CONCLUSIONS:Based on exact values for the joint distribution of longest run and number of crossings in random data series this study demonstrates that it is possible to obtain better diagnostic properties of run charts by making minor adjustment to the critical values for C and L

Sense and sensibility: on the diagnostic value of control chart rules for detection of shifts in time series data

Abstract Background The aim of this study was to quantify and compare the diagnostic value of The Western Electric (WE) statistical process control (SPC) chart rules and the Anhoej rules for detection of non-random variation in time series data in order to make recommendations for their application in practice. Methods SPC charts are point-and-line graphs showing a measure over time and employing statistical tests for identification of non-random variation. In this study we used simulated time series data with and without non-random variation introduced as shifts in process centre over time. The primary outcome was likelihood ratios of combined tests. Likelihood ratios are useful measures of a test’s ability to discriminate between the true presence or absence of a specific condition. Results With short data series (10 data points), the WE rules 1–4 combined and the Anhoej rules alone or combined with WE rule 1 perform well for identifying or excluding persistent shifts in the order of 2 SD. For longer data series, the Anhoej rules alone or in combination with the WE rule 1 seem to perform slightly better than the WE rules combined. However, the choice of which and how many rules to apply in a given situation should be made deliberately depending on the specific purpose of the SPC analysis and the number of available data points. Conclusions Based on these results and our own practical experience, we suggest a stepwise approach to SPC analysis: Start with a run chart using the Anhoej rules and with the median as process centre. If, and only if, the process shows random variation at the desired level, apply the 3-sigma rule in addition to the Anhoej rules using the mean as process centre

Joint distribution for number of crossings and longest run in independent Bernoulli observations. The R package crossrun.

The R package crossrun computes the joint distribution of the number of crossings and the longest run in a sequence of independent Bernoulli observations. The main intended application is statistical process control where the joint distribution may be used for systematic investigation, and possibly refinement, of existing rules for distinguishing between signal and noise. While the crossrun vignette is written to assist in practical use, this article gives a hands-on explanation of why the procedures works. The article also includes a discussion of limitations of the present version of crossrun together with an outline of ongoing work to meet these limitations. There is more to come, and it is necessary to grasp the basic ideas behind the procedure implemented both to understand these planned extensions, and how presently implemented rules in statistical process control, based on the number of crossings and the longest run, may be refined

Run Charts Revisited: A Simulation Study of Run Chart Rules for Detection of Non-Random Variation in Health Care Processes

<div><p>Background</p><p>A run chart is a line graph of a measure plotted over time with the median as a horizontal line. The main purpose of the run chart is to identify process improvement or degradation, which may be detected by statistical tests for non-random patterns in the data sequence.</p><p>Methods</p><p>We studied the sensitivity to shifts and linear drifts in simulated processes using the shift, crossings and trend rules for detecting non-random variation in run charts.</p><p>Results</p><p>The shift and crossings rules are effective in detecting shifts and drifts in process centre over time while keeping the false signal rate constant around 5% and independent of the number of data points in the chart. The trend rule is virtually useless for detection of linear drift over time, the purpose it was intended for.</p></div