Standard sources of data for influenza surveillance include notifications of laboratory-confirmed cases and notifications from sentinel general practices. These data are not always available in a timely fashion, leading to proposals to use more immediate data sources such as over-the-counter drug sales, ambulance call-outs and web searches to monitor influenza-like illness (ILI). We aimed to assess data from a deputising medical service as another source of data for timely syndromic influenza surveillance. We measured the extent of agreement between the weekly percentage of patients with ILI reported from sentinel general practices and the corresponding weekly percentage reported from a deputising medical service in Victoria, Australia over ten years, from 1999 to 2008. There was good agreement between the two data sources, with suitably narrow limits of agreement. The deputising medical service did not use a standardised definition of ILI and is not supplemented by laboratory confirmation of suspected cases. Nevertheless, the results of this study show that such data can provide low cost and timely ILI surveillance

Coory, M.

Grant, K.

Kelly, H.

University of Queensland eSpace

 www.eurosurveillance.org 1
S urve i ll an ce  an d  ou t b reak  re p o r t s
I n f l u e n z a - l I k e  I l l n e s s  s u r v e I l l a n c e  u s I n g  a  d e p u t I s I n g 
m e d I c a l  s e r v I c e  c o r r e s p o n d s  to  s u r v e I l l a n c e  f r o m 
s e n t I n e l  g e n e r a l  p r a c t I c e s
M Coory (m.coory@uq.edu.au)1, K Grant2, H Kelly2,3
1. School of Population Health, University of Queensland, Herston, Australia
2. Victorian Infectious Diseases Reference Laboratory, Melbourne, Australia
3. School of Population Health, University of Melbourne, Melbourne, Australia
This article was published on 5 November 2009. 
Citation style for this article: Coory M, Grant K, Kelly H. Influenza-like illness surveillance using a deputising medical service corresponds to surveillance from sentinel 
general practices. Euro Surveill. 2009;14(44):pii=19387. Available online: http://www.eurosurveillance.org/ViewArticle.aspx?ArticleId=19387
Standard sources of data for influenza surveillance include 
notifications of laboratory-confirmed cases and notifications from 
sentinel general practices. These data are not always available 
in a timely fashion, leading to proposals to use more immediate 
data sources such as over-the-counter drug sales, ambulance call-
outs and web searches to monitor influenza-like illness (ILI). We 
aimed to assess data from a deputising medical service as another 
source of data for timely syndromic influenza surveillance. We 
measured the extent of agreement between the weekly percentage 
of patients with ILI reported from sentinel general practices and 
the corresponding weekly percentage reported from a deputising 
medical service in Victoria, Australia over ten years, from 1999 to 
2008. There was good agreement between the two data sources, 
with suitably narrow limits of agreement. The deputising medical 
service did not use a standardised definition of ILI and is not 
supplemented by laboratory confirmation of suspected cases. 
Nevertheless, the results of this study show that such data can 
provide low cost and timely ILI surveillance.
Introduction 
In temperate southern Australia, the influenza season occurs 
between May (late autumn) and October (early spring). Sentinel 
general-practitioner (GP) surveillance, operational in Victoria 
during the influenza season, reports weekly on the number of 
patients fulfilling the Australian nationally agreed case definition 
of influenza-like illness (ILI): cough, fever and fatigue. Respiratory 
specimens taken from a proportion of cases permit diagnosis of 
laboratory-confirmed influenza [1]. Not all ILI cases are confirmed 
as influenza. In Victoria, Australia, the proportion of confirmed 
cases between 2003-2007 varied from 18-47%, annually [2]. 
Besides notifications from sentinel GPs, another standard 
method of influenza surveillance is to count the number of 
laboratory-confirmed cases notified to a public health authority 
[1]. Both these standard data sources, which involve laboratory 
testing, are associated with a reporting lag due to the time taken for 
specimen testing and reporting. For instance, the median interval 
between symptom onset and registration for a laboratory test was 
three days for a patient recruited through sentinel GPs in Victoria 
in 2007 and 2008. 
To overcome the problem of delay, surveillance using more 
immediate data sources without laboratory confirmation, referred 
to as syndromic surveillance, have been implemented. These 
include over-the-counter drug sales [3], telephone calls to health 
information lines such as nurse on call [4], ambulance call-outs [5], 
school or workforce absenteeism [6,7], and web searches [8-10].
One surveillance source, previously described by Turner and Kelly 
[11] but not formally assessed, is a deputising medical service, that 
is, an out-of-hours service for GP consultations. Many deputising 
services record the reason for the call-out and the final diagnosis 
in an electronic database, such as the GP house call surveillance 
system in Bordeaux, France [12].  The aim of this study was to 
measure the extent of agreement between ILI surveillance data 
from the deputising service and data from the sentinel GP system 
in Melbourne, Australia, in order to assess whether the former could 
be used for routine influenza surveillance. 
Methods
The Melbourne Medical Deputising Service (MMDS) is a 
deputising, out-of-hours general practice service. Deputising 
doctors attend patients in their homes within a 45 km radius of 
the Melbourne Central Business District. Demographic (e.g. age, 
sex) and clinical data (e.g. diagnosis) are entered by the deputising 
doctor into a customised database, usually within 24 hours of 
the consultation. Access to the data is available on a password-
protected page of the MMDS website. The data are available for 
use in a surveillance system as soon as they are entered, i.e. within 
24 hours of the consultation.
We routinely obtain the proportion of ILI call-outs from the 
MMDS once a week, although they could be obtained daily 
with a 24-hour lag. The weekly data extraction uses a validated 
search algorithm that identifies the number of call-outs for ILI. 
This is divided by the total number of call-outs for that week 
and expressed as a percentage per 100 call-outs. MMDS data 
are available throughout the year. The search algorithm has been 
validated by manual confirmation of the diagnosis of all patients 
identified by the search algorithm for week 34 in the years 2002 
to 2007, a week of high activity for all years in that period,. The 
search algorithm successfully identified ILI call-outs searching 
for the terms ‘flu’ and ‘influ’ and excluding terms such as ‘reflux’ 
2  www.eurosurveillance.org
and ‘fluid’ that included the letters ‘flu’. New exclusion terms, 
‘fluvax’, ‘at risk’ and ‘immunisation’, were added to the algorithm in 
2009 to exclude pandemic H1N1 influenza contacts who received 
prophylactic antiviral treatment. 
For the sentinel GP system, we used the number of consultations 
that met the nationally agreed definition of ILI expressed as a 
percentage of total visits as the comparator. We then assessed 
the degree of consensus between this measure and that from the 
MMDS, using a standard statistical method developed by Bland and 
Altman [13,14]. This method is based on reporting the difference 
between the two measures, and the 95% limits of agreement, which 
provide an interval in which 95% of the differences between the 
two measurements are expected to lie. If the limits of agreement 
describe differences that are not of material importance, the data 
sources can be used interchangeably. 
As described by Bland and Altman [14], it is not unusual for 
the difference between two measures and the standard deviation 
to increase with increasing values of the two measures being 
assessed, and this should be accounted for in the statistical 
analysis, otherwise the limits of agreement will be too wide for 
low values of weekly ILI proportions and too narrow for high 
values. Accordingly, we regressed the difference of the weekly ILI 
percentages on their average, using absolute residuals to estimate 
the standard deviation.
To further assess the comparability of the two surveillance 
systems, we calculated the area under the receiver operator 
characteristic (ROC) curve for the 10 years of data from the 
deputising service against the weeks with higher than expected 
seasonal activity as currently defined by a sentinel GP weekly ILI 
percentage of 1.5%, described by Watts et al. [15]. In the context 
of this study, as described by Bland and Altman [16], an area under 
the ROC curve of 0.5 would mean that the deputising service was 
no better than chance in detecting the influenza season, while a 
value of 1.0 would mean that it was a perfect measure. Confidence 
intervals for the area under the ROC curve were obtained using the 
algorithm of DeLong et al. [17].
Results
From 1999 to 2008, the weekly percentages of ILI reported 
through the deputising service were similar to the percentages seen 
in the sentinel GP system during periods of low seasonal activity, 
but were larger in periods of higher activity, although this was less 
evident in later years (Figure 1). The difference between the two 
F i g u r e  1
Weekly percentage of ILI reported through the deputising service versus the sentinel GP system, Melbourne, Australia, 
1999-2008
0
2
4
6
8
10
12
04
 J
an
 1
99
9
04
 A
pr
 1
99
9
04
 J
ul
 1
99
9
04
 O
ct
 1
99
9
04
 J
an
 2
00
0
04
 A
pr
 2
00
0
04
 J
ul
 2
00
0
04
 O
ct
 2
00
0
04
 J
an
 2
00
1
04
 A
pr
 2
00
1
04
 J
ul
 2
00
1
04
 O
ct
 2
00
1
04
 J
an
 2
00
2
04
 A
pr
 2
00
2
04
 J
ul
 2
00
2
04
 O
ct
 2
00
2
04
 J
an
 2
00
3
04
 A
pr
 2
00
3
04
 J
ul
 2
00
3
04
 O
ct
 2
00
3
04
 J
an
 2
00
4
04
 A
pr
 2
00
4
04
 J
ul
 2
00
4
04
 O
ct
 2
00
4
04
 J
an
 2
00
5
04
 A
pr
 2
00
5
04
 J
ul
 2
00
5
04
 O
ct
 2
00
5
04
 J
an
 2
00
6
04
 A
pr
 2
00
6
04
 J
ul
 2
00
6
04
 O
ct
 2
00
6
04
 J
an
 2
00
7
04
 A
pr
 2
00
7
04
 J
ul
 2
00
7
04
 O
ct
 2
00
7
04
 J
an
 2
00
8
04
 A
pr
 2
00
8
04
 J
ul
 2
00
8
04
 O
ct
 2
00
8
Date
IL
I/
 1
00
 c
on
su
lt
at
io
n
s
Sentinel GPs
Deputising service
GP: general practitioner; ILI: influenza-like illness
 www.eurosurveillance.org 3
ILI data sources was small, but increased during the peak of the 
season, with data from the deputising service recording higher 
values than data from the sentinel GPs (Table).
The 95% limits of agreement increased with increasing ILI 
activity, the importance of which, as noted by Bland and Altman 
[13], is a matter of judgement, rather than a statistical issue. Our 
judgement is that the limits of agreement are appropriately small 
F i g u r e  2
Weekly percentage of influenza-like illness reported through the deputising service versus the sentinel general practitioner system, 
Melbourne, Australia, Victoria, Australia, influenza season 2009
GP: general practitioner; ILI: influenza-like illness
0
1
2
3
4
5
6
27
 A
pr
 2
00
9
04
 M
ay
 2
00
9
11
 M
ay
 2
00
9
18
 M
ay
 2
00
9
25
 M
ay
 2
00
9
01
 J
un
 2
00
9
08
 J
un
 2
00
9
15
 J
un
 2
00
9
22
 J
un
 2
00
9
29
 J
un
 2
00
9
06
 J
ul
 2
00
9
13
 J
ul
 2
00
9
20
 J
ul
 2
00
9
27
 J
ul
 2
00
9
03
 A
ug
 2
00
9
10
 A
ug
 2
00
9
17
 A
ug
 2
00
9
24
 A
ug
 2
00
9
31
 A
ug
 2
00
9
07
 S
ep
 2
00
9
14
 S
ep
 2
00
9
21
 S
ep
 2
00
9
28
 S
ep
 2
00
9
Date
IL
I/
 1
00
 C
on
su
lt
at
io
n
s
Sentinel GPs
Deputising service
T a b l e
95% limits of agreement for deciles of the average* of deputising service and sentinel general practitioner data
(Cumulative percentage of observations)
Average of ILI from deputising service 
and sentinel GP per 100 consultations
Difference between deputising service and sentinel GP ILI per 
100 consultations 95% limits of agreement
(10%) 0.42% 0.0% -0.6%, 0.3%
(20%) 0.58% 0.2% -0.4%, 0.8%
(30%) 0.72% 0.4% -0.3%, 1.1%
(40%) 0.86% 0.6% -0.2%, 1.4%
(50%) 1.00% 0.8% -0.0%, 1.6%
(60%) 1.19% 1.0% 0.1%, 1.9%
(70%) 1.45% 1.3% 0.2%, 2.3%
(80%) 1.87% 1.6% 0.5%, 2.8%
(90%) 2.53% 3.1% 0.8%, 3.6%
* Assessing 95% limits of agreement against the average is the preferred method of assessing whether one set of measurements can substitute for (is 
equivalent to) another [19].
GP: general practitioner; ILI: influenza-like illness.
4  www.eurosurveillance.org
during periods of normal seasonal ILI activity as well as at the 
start and end of the season, and that the wider limits at the peak 
of the season, or in seasons of higher activity, are of no material 
importance. 
The area under the ROC curve was 0.91 (95% confidence 
interval (CI) 0.83, 0.98), confirming very close agreement between 
the systems when dichotomised around ILI activity describing 
normal and higher than expected seasonal activity. 
Having both surveillance systems in place has been very useful 
in the H1N1 influenza pandemic of 2009 as the two surveillance 
systems provided complementary and confirmatory surveillance 
data when influenza A(H1N1)v was the dominant circulating strain 
[20]. As with previous years, however, ILI proportions from the two 
surveillance systems were more similar for lower values (Figure 2).
Discussion
There was good agreement between the weekly percentages 
of ILI in the deputising service and sentinel GP system, although 
the agreement for high ILI values was not as close as for lower 
values. This is probably because the deputising service is an out-
of-hours service, which is likely to have a higher percentage of 
call-outs for acute illnesses, such as influenza. The deputising 
service is also less likely to see non-acute illnesses, effectively 
increasing the ILI percentage relative to sentinel GPs who would 
continue to see patients for chronic diseases during the peak of 
the influenza season. Moreover all ILI consultations are captured 
by the deputing service database, whereas GP data are recorded 
on paper forms which makes complete capture of all ILI patients 
is unlikely. This would reduce the reported ILI percentage from 
sentinel GPs compared with the deputising service.
We did not use the correlation coefficient to assess whether the 
deputising service data were equivalent to the sentinel GP data 
as some authors have done [8], because this approach has been 
questioned in a series of much cited papers by Bland and Altman 
[13,14,18,19]. There are two reasons for not using the correlation 
coefficient to assess equivalence of two data sources: First, if the 
values of the data vary across a wide range, as is the case for ILI 
data from both deputising service and sentinel GPs, the correlation 
coefficient will be close to 1.0 even if one measure is not a good 
substitute for the other. Second, correlation ignores any systematic 
bias between the two measures. To overcome these problems, 
Bland and Altman recommended reporting the difference, or bias, 
between the two measures and the 95% limits of agreement and 
we have followed their advice in this study.
We did not examine agreement for different age groups. However, 
for the most recent five-year period included in the analysis (2004-
2008), the percentage of ILI cases under the age of 15 years was 
similar in the two systems (19.5% in the deputising service versus 
18.8% in the sentinel GP system), while the ILI cases from the 
deputising service were slightly older than those from sentinel 
general practice (mean 40.7 years versus 39.9 years) and showed 
more variation (standard deviation 25.6 versus 20.3). This was 
because of the growing number of out-of-hours consultations by 
the deputising medical service at care facilities for the elderly in 
the latter years of surveillance; 8.6% of ILI cases identified by the 
deputising service were 80 years or older while the corresponding 
percentage for sentinel general practice was only 2.2%.
Deputising medical service surveillance does not use a 
standardised definition of ILI and is not supplemented by laboratory 
confirmation of suspected influenza cases. Nevertheless we have 
shown that data from a deputising medical service can provide low 
cost and timely ILI surveillance throughout the year, equivalent to 
ILI surveillance provided by sentinel GPs. Further confirming its 
utility, surveillance data from the deputising service confirmed the 
onset and peak of ILI activity during the 2009 pandemic in Victoria.
Acknowledgements
We gratefully acknowledge the ongoing support of the Executive Director 
of MMDS, Ms Josie Adams and database support from Steven Long of 
SL Digital.
References
1. Grant K, Carville K, Fielding J, Barr I, Tran T, Riddell M, et al. High proportion 
of influenza B in the 2008 influenza season in Victoria. Communicable 
Diseases Intelligence. 2009; Forthcoming. 
2. Kelly HA, Carville K, Grant K, Jacoby P, Thomas T, Barr I. Estimation of 
Influenza Vaccine Effectiveness from Routine Surveillance Data. PlosOne. 
2009; Forthcoming. 
3. Vergu E, Grais RF, Sarter H, Fagot JP, Lambert B, Valleron AJ, et al. Medication 
sales and syndromic surveillance, France. Emerg Infect Dis. 2006;12(3):416-21. 
4. Espino JU, Hogan WR, Wagner MM. Telephone triage: a timely data source for 
surveillance of influenza-like diseases. AMIA Annu Symp Proc. 2003:215-9. 
5. Coory MD, Kelly H, Tippett V. Assessment of ambulance dispatch data for 
surveillance of influenza-like illness in Melbourne, Australia. Public Health. 
2009;123(2):163-8. 
6. Besculides M, Heffernan R, Mostashari F, Weiss D. Evaluation of school 
absenteeism data for early outbreak detection, New York City. BMC Public 
Health. 2005;5:105. 
7. Zhao H, Joseph C, Phin N. Outbreaks of influenza and influenza-like illness 
in schools in England and Wales, 2005/06. Euro Surveill. 2007;12(5):E3-4. 
8. Ginsberg J, Mohebbi MH, Patel RS, Brammer L, Smolinski MS, Brilliant L. 
Detecting influenza epidemics using search engine query data. Nature. 
2009;457(7232):1012-4. 
9. Polgreen PM, Chen Y, Pennock DM, Nelson FD. Using internet searches for 
influenza surveillance. Clin Infect Dis. 2008;47(11):1443-8. 
10. Butler D. Web data predict flu. Nature. 2008;456(7220):287-8. 
11. Turner J, Kelly H. A medical locum service as a site for sentinel influenza 
surveillance. Euro Surveill. 2005;10(4):pii: 530. Available from: http://www.
eurosurveillance.org/ViewArticle.aspx?ArticleId=530 
12. Flamand C, Larrieu S, Couvy F, Jouves B, Josseran L, Filleul L. Validation of 
a syndromic surveillance system using a general practitioner house calls 
network, Bordeaux, France. Euro Surveill. 2008;13(25):pii: 18905. Available 
from: http://www.eurosurveillance.org/ViewArticle.aspx?ArticleId=18905 
13. Bland JM, Altman DG. Statistical methods for assessing agreement between 
two methods of clinical measurement. Lancet. 1986;1(8476):307-10. 
14. Bland JM, Altman DG. Measuring agreement in method comparison studies. 
Stat Methods Med Res. 1999;8(2):135-60. 
15. Watts CG, Andrews RM, Druce JD, Kelly HA. Establishing thresholds for influenza 
surveillance in Victoria. Aust N Z J Public Health. 2003;27(4):409-12. 
16. Altman DG, Bland JM. Diagnostic tests 3: receiver operating characteristic 
plots. BMJ. 1994;309(6948):188. 
17. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or 
more correlated receiver operating characteristic curves: a nonparametric 
approach. Biometrics. 1988;44(3):837-45. 
18. Altman DG, Bland JM. Measurement in medicine: The analysis of method 
comparison studies. The Statistician. 1983;32:307-17. 
19. Bland JM, Altman DG. Comparing methods of measurement: why 
plotting difference against standard method is misleading. Lancet. 
1995;346(8982):1085-7. 
20. Kelly H, Grant K. Interim analysis of pandemic influenza (H1N1) 2009 in 
Australia: surveillance trends, age of infection and effectiveness of seasonal 
vaccination. Euro Surveill. 2009;14(31):pii: 19288. Available from: http://www.
eurosurveillance.org/ViewArticle.aspx?ArticleId=19288 


Influenza-like illness surveillance using a deputising medical service corresponds to surveillance from sentinel general practices

http://espace.library.uq.edu.au/view/UQ:189902/UQ189902_OA.pdf

Influenza-like illness surveillance using a deputising medical service corresponds to surveillance from sentinel general practices

Abstract

Similar works

Full text

Available Versions

University of Queensland eSpace