In the context of the Community Programme in the field of Health, the European Commission financed a series of initiatives to support the development and use of indicators for planning health services for Rare Diseases (RDs). The European Project for Rare Disease National Plans Development (EUROPLAN) elaborated a set of 59 process and outcome indicators, for monitoring the implementation and for evaluating the impact of the National Plans on RDs. Due to the high number and difficulty in handling the indicators, the subsequent Joint Action "Working for RDs" planned to derive a selection of 21 core indicators that were adopted by the European Union Committee of Experts on RDs in June 2013. The descriptive study carried out in the framework of the Joint Action to select the key indicators to orient policies for RDs shows that core indicators represent an excellent opportunity to share knowledge and comparability among Member States

Amalia Egle Gentile

Domenica Taruscio

Marta De Santis

Rita Maria Ferrelli

Annali dell Istituto Superiore di Sanità

In the context of the Community Programme in the field of Health, the European Commission financed a series of initiatives to support the development and use of indicators for planning health services for Rare Diseases (RDs). The European Project for Rare Disease National Plans Development (EUROPLAN) elaborated a set of 59 process and outcome indicators, for monitoring the implementation and for evaluating the impact of the National Plans on RDs. Due to the high number and difficulty in handling the indicators, the subsequent Joint Action “Working for RDs” planned to derive a selection of 21 core indicators that were adopted by the European Union Committee of Experts on RDs in June 2013. The descriptive study carried out in the framework of the Joint Action to select the key indicators to orient policies for RDs shows that core indicators represent an excellent opportunity to share knowledge and comparability among Member States

Ferrelli, Rita Maria

De Santis, Marta

Gentile, Amalia Egle

Taruscio, Domenica

Annali dell'Istituto Superiore di Sanità

O
r
ig
in
a
l
 a
r
t
ic
l
e
s
 a
n
d
 r
e
v
ie
w
s
342
Ann Ist Super Sanità 2015 | Vol. 51, No. 4: 342-345
DOI: 10.4415/ANN_15_04_15
Key words
•  rare diseases
•  health services
•  health planning
•  indicators
Exploring the usability of EUCERD 
core indicators for rare diseases
Rita Maria Ferrelli1, Marta De Santis2, Amalia Egle Gentile2 and Domenica Taruscio2
1Ufficio Relazioni Esterne, Istituto Superiore di Sanità, Rome, Italy 
2Centro Nazionale Malattie Rare, Istituto Superiore di Sanità, Rome, Italy
Abstract
In the context of the Community Programme in the field of Health, the European 
Commission financed a series of initiatives to support the development and use of 
indicators for planning health services for Rare Diseases (RDs). The European Project 
for Rare Disease National Plans Development (EUROPLAN) elaborated a set of 59 
process and outcome indicators, for monitoring the implementation and for evaluat-
ing the impact of the National Plans on RDs. Due to the high number and difficulty 
in handling the indicators, the subsequent Joint Action “Working for RDs” planned 
to derive a selection of 21 core indicators that were adopted by the European Union 
Committee of Experts on RDs in June 2013. The descriptive study carried out in the 
framework of the Joint Action to select the key indicators  to orient policies for RDs 
shows that core indicators represent an excellent opportunity to share knowledge and 
comparability among Member States. 
BACKGROUND
The Council of the European Union, in its 8 June 
2009 Recommendation on an action in the field of Rare 
Diseases (RDs), recommends that “Member States 
elaborate and adopt a plan or strategy as soon as pos-
sible, preferably by the end of 2013 at the latest, aimed 
at guiding and structuring relevant actions in the field 
of Rare Diseases within the framework of their health 
and social systems” [1]. The development and use of 
indicators is an integral part of planning and designing 
Health Services, as they help to analyze and compare 
performance across population groups or geographic 
areas, and they can be useful in determining policy pri-
orities. In the specific field of RDs, building a set of 
harmonized indicators on strategies/plans is a complex 
task because there are thousands of diseases that show 
different and specific issues and necessities. Due to the 
heterogeneity of RDs, the low number of patients/dis-
ease and the geographical spread, many indicators used 
for most common diseases are not applicable. More-
over Countries’ Health Care Systems do not always fol-
low the same organizational scheme across Europe and 
this is another difficulty. Nonetheless, in this context, 
the pooling of resources and the general consensus are 
the necessary basis to generate indicators. In order, to 
support this process, the programme of Community ac-
tion in the field of Public Health co-funded the Euro-
pean Project for Rare Disease National Plans Develop-
ment (EUROPLAN), a three-year project (2008-2011) 
coordinated by the Italian National Centre for Rare 
Diseases at National Institute of Health (Istituto Supe-
riore di Sanità – ISS).
EUROPLAN aimed at identifying indicators to as-
sess RDs initiatives with a view on monitoring the im-
plementation and evaluating the impact of the national/
regional plans on RDs, as well as the involved cost in 
maintaining this public health information system. 
EUROPLAN was structured along the areas of inter-
est listed in the European Union (EU) proposal for a 
Council Recommendation, the EU Communication on 
RDs, namely:
• Area 1. Plans and strategies in the field of RDs;
• Area 2. Adequate definition, codification and inven-
torying of RDs;
• Area 3. Research on RDs;
• Area 4. Centres of Expertise and European Refer-
ence Networks for RDs;
• Area 5. Gathering the expertise on RDs at European 
level;
• Area 6. Empowerment of patient organisations;
• Area 7. Sustainability.
For each area the following points were discussed:
• background;
•  key Message;
•  rationale;
• health context.
Standard definition used to describe the features of 
indicators took into account the main literature on in-
Address for correspondence: Rita Maria Ferrelli, Uffficio Relazioni Esterne, Istituto Superiore di Sanità, Viale Regina Elena 299, 00161 Rome, Italy. 
E-mail: rita.ferrelli@iss.it. 
Usability of EUCERD CoRE inDiCatoR
O
r
ig
in
a
l
 a
r
t
ic
l
e
s
 a
n
d
 r
e
v
ie
w
s
343
trinsic properties (understandability, reliability, validity, 
consistency, sensitivity, specificity, feasibility), resource 
demand (availability, sustainability, implementation, 
workload demand, timeliness) and decision making 
(applicability, coherence, comprehensiveness, policy 
relevance) [2-8].
EUROPLAN developed a set of 59 indicators, that 
were organised in process and outcome indicators, to 
monitor the implementation and to evaluate the im-
pact of National Plans or Strategies for RDs [9]. While 
all indicators were considered relevant, from an op-
erational perspective, they were quite numerous and 
difficult to handle. Therefore, the subsequent Joint 
Action “Working for Rare Diseases” (EJA) of the Eu-
ropean Union Committee of Experts on Rare Diseases 
(EUCERD) planned to derive a selection of core in-
dicators. In order to reduce the complexity of the list, 
EUROPLAN 2012-15 (as WP4 of the EJA) identified 
a series of 21 core indicators, according to criteria of 
usefulness and feasibility. The selection of core indica-
tors has been the result of the work carried out in sev-
eral procedural steps by both teams of EUROPLAN 
and the European Organisation for Rare Diseases 
(EURORDIS), a non-governmental patient-driven al-
liance of patient organisations and individuals active in 
the field of RDs, dedicated to improving the quality of 
life of all people living with RDs in Europe. The process 
foresaw two independent methodologies for selecting 
the “core indicators”:  
• Delphi process (carried out by the ISS) with the col-
laboration of Ministry of Health representatives of 27 
Member States (MS), 10 EURORDIS advisors and 4 
experts. Participants were asked to select indicators 
according to two criteria: usefulness and feasibility of 
data collection. The process took place from 1 De-
cember 2012 to 14 February 2013. A satisfactory level 
of agreement was reached at the first round of the 
Delphi;
•  EURORDIS approach (carried out by EURORDIS) 
with the participation of 8 EURORDIS advisors in con-
junction with their Ministry of Health lead contacts on 
national plans/strategies.
The drafting group elaborated the first version of the 
Recommendations, merging the results from the two 
methodologies and circulated them among Ministry of 
Health representatives of 27 Member States, 10 EU-
RORDIS Advisors and 4 experts; their comments and 
amendments were included for the preparation of the 
second draft, that was discussed and amended during 
the EUROPLAN Workshop “Key Indicators for Na-
tional Plans” (ISS, Rome, 25 March 2013). The sub-
sequent drafts circulated among EUCERD members, 
until adoption. EUCERD adopted the list of 21 core 
indicators on 6 June 2013 [10].
Following EUCERD adoption of the core indicators 
in June 2013, a follow-up study about the use of indi-
cators was deemed necessary for their fine-tuning and 
for improving their potential to orient policies for RDs. 
Therefore, a study has been planned in order to explore 
the degree of usability of the core indicators in selected 
EU Member States and to identify problems in relation 
to the use of the indicators.
METHOD
The descriptive study was structured in two compo-
nents:
1. a survey on the use of the indicators in selected EU 
Member States;
2. an exploratory collection of lessons to take into ac-
count for strengthening potential indicators to orient 
policies for RDs.
The sample size was defined after consultation with 
the EJA team that opted to carry out the study on a lim-
ited number of Countries. The selection of the Coun-
tries was done with purposive sampling, according to 
activities carried out in former and successful collabora-
tion with the Italian National Centre for Rare Diseases, 
namely: Bulgaria, Croatia, Italy, Romania and Spain.
The 21 core indicators were entered in the web-based 
survey system SurveyMonkey (www.surveymonkey.
com). The representatives of the Ministries of Health, 
directly involved in planning, monitoring and evaluating 
National Plans for RDs in the five selected Members 
States, were invited by E-mail to fill-in the question-
naire. Data were analysed by means of descriptive uni-
variate analysis.
The second component of the study included open-
answer questions about:
• positive aspects highlighted while using the indica-
tors;
• problems faced while using the indicators;
•  opportunity to integrate the indicators.
RESULTS
The main results of the survey for the indicators are 
summarised in Table 1. Concerning the second part of 
the study, the following answers were collected in rela-
tion to the positive aspects highlighted while using the 
indicators:
•  collection of important information about National 
Plans on RDs and easiness to answer;
•  excellent opportunity to share knowledge and compa-
rability among countries;
• political usefulness;
•  chance to adapt national RDs policies to best exam-
ples available and recommended at EU level;
•  tool to timely follow up and report on national RDs 
activities;
• possibility to follow-up the progress;
•  focus on relevant issues regarding development and 
implementation of National Plans which are common 
to the 28 EU Member States;
• harmonisation of monitoring procedures and criteria 
and assessment of common RDs policies in the 28 EU 
Member States;
• usefulness to capture and to describe the situation as 
far as the general measures adopted by countries.
Problems faced while using the indicators relate to 
the fact that:
• quantitative indicators may not reflect qualitative im-
provements;
•  comparing Member States’ different rules and regu-
lation to make clear the different organization of the 
countries through short answers can be very difficult;
•  indicator on participation in European Reference 
Rita Maria Ferrelli, Marta De Santis, Amalia Egle Gentile and Domenica Taruscio
O
r
ig
in
a
l
 a
r
t
ic
l
e
s
 a
n
d
 r
e
v
ie
w
s
344
Networks is difficult to use (since European Reference 
Networks for RDs are not yet established); 
•  visibility of what is done for RDs patients in pro-
grammes to support in their daily life integration (in 
place in all Countries) is low;
•  regionalisation may hamper data collection from re-
gions to the national level; e.g. even though Spain has 
a national policy with public funds for research and for 
the national plan, there are regional plans on RDs with 
regional funds and the regional information is difficult 
to collect. Another example is done by Centers of Ex-
pertise: in Spain there are National Centers adhering 
to the national policy and regional centers adhering to 
the regional policies. If the indicator mixes up the two 
terms, the answer cannot be done.
In relation to the opportunity of integrating the in-
dicators, the participants to the survey believe that 
increasing their number would entail a return to the 
“starting line”. However, they did not exclude a possible 
future integration with EUCERD criteria on centres 
of expertise for RDs and Rare Disease Reference Net-
works, and they also suggest including information on 
new born screening policies and on the genetic diagno-
sis and genetic counselling policies.
DISCUSSION AND CONCLUSIONS
The objective of the Core Indicators for Rare Disease 
National Plans and Strategies is to capture relevant 
data and information on the process of planning, fol-
low up and implementation of the plans and strategies. 
They are instrumental for the decision-making process 
related to the adoption, assessment and further devel-
opment of RDs public policies. Their use is also for the 
report to the European Commission and for the annual 
reporting that Member States are required to do in view 
of the drafting of the “Report on the State of the Art on 
Rare Disease Activities in Europe of the EU Commit-
tee of Experts on Rare Diseases”, published on a yearly 
basis by the Expert Group on Rare Diseases, former 
EUCERD.
The indicators are structured around the seven the-
matic areas of the European Council Recommenda-
tions: 
• plans and strategies in the field of RDs; 
•  adequate definition, codification and inventorying of 
RDs; 
•  research; 
•  centres of expertise and European Reference Net-
works for RDs; 
•  gathering the expertise on RDs at European level; 
•  empowerment of patient organisations; 
•  sustainability. 
As binary variables, they give at a glance the picture 
of the Country in relation to the examined issues. As 
such, they may not be able to give all the related nu-
ances. Qualitative information will be necessary to fill 
in the gap.
Indicators are supposed to be SMART: Specific, 
Measurable, Appropriate, Realistic, Time-bound [11]. 
Therefore, their use will have to be related to the tar-
get group. The factors that need to be changed and the 
objectives under study will have to be achievable and 
Table 1
Main results of the survey on the usability of EUCERD core indicators
Normative aspects for 
developing a RDs plan
The selected EU Member States either are working on or have a regulation embedded in national law. They 
adopted EU Rare Disease definition and applied it on National Plans/National Strategies.
A RDs advisory committee is present and includes all stakeholders. A permanent and official patients’ 
representation in plan development, monitoring and assessment is also present in the selected EU Member 
States.
Centres of Expertise on 
RDs and guidelines
Croatia and Spain have already a national policy for establishing Centres of Expertise on RDs fully implemented; 
the other Countries are working on. As far as the number of national and regional Centres of Expertise adhering 
to the national policy is concerned, the answers show a heterogeneous picture, reflecting different local 
situations. The participation of Centres of Expertise in European Reference Networks is mainly still in progress. 
The Member States who answered the question on clinical guidelines confirmed the existence of a policy for 
developing Clinical Practice Guidelines.
Information system 
on RDs 
All Countries support an information system on RDs at national level. They participate in Orphanet Joint Action 
and a few Countries produce information in national language(s).
Help lines for RDs Italy, Romania and Spain have help lines for RDs.
Registries Except for Croatia, the other Countries have a national policy on registry and data collection on RDs. In detail: 
• Bulgaria and Spain have a policy for a national/centralised registry and data collection;
• Italy has a policy for national/centralised and regional registry and data collection;
• Romania representative responded “Do not know”.
Research on RDs A few Countries refer the existence of specific RDs research programs in the framework of national research 
programs. No specific public funds are allocated to the research on RDs in the selected Countries. All the 
selected countries, excepted for Croatia, participate in European and international research initiatives
Therapy and patients’ 
integration
All Countries, with the exception of Bulgaria, refer the existence of a governmental system for compassionate 
use of medicinal products.
In relation to patients’ integration, the respondents refer the existence of programs, in some Countries 
specifically addressing RDs patients, in other generally directed to persons with a disability.
Economic sustainability The status of either policy or decision to ensure long-term funding and/or sustainability of the measures in the 
RDs plan/strategy is heterogeneous, since a few Countries are already committed, while other are working on it.
EUCERD: European Union Committee of Experts on Rare Disease; RD: rare disease.
Usability of EUCERD CoRE inDiCatoR
O
r
ig
in
a
l
 a
r
t
ic
l
e
s
 a
n
d
 r
e
v
ie
w
s
345
REFERENCES
1. European Union. Council Recommendation of 8 June 
2009 on an action in the field of rare diseases (2009/ C 
151/02). Official Journal of the European Union C171/7, 
3/7/2009. Available from: http://eur-lex.europa.eu/Lex-
UriServ/LexUriServ.do?uri=OJ:C:2009:151:0007:0010:E
N:PDF.
2. Cambridge advanced learner’s dictionary. Cambridge: Cam-
bridge University Press; 2009. 
3. McGlynn EA. Six challenges in measuring the quality of 
health care. Health Aff 1997;16(3):7-21. 
4. Miller GA (Ed). WordNet. Princeton University. The 
Trustees of Princeton University; 2009. Available from: 
http://wordnet.princeton.edu/.
5. Murray CJL. Health Statistics 2. Towards good practice 
for health statistics: lessons from the Millennium De-
velopment Goal health indicators. Lancet 2007;369:862-
73.
6. Nikolai LA, Bazley JD, Jones JP. Intermediate accounting. 
11 ed. Mason OH: South-Western Cengage Learning; 
2009. 
7. Porta M. A dictionary of epidemiology. 5th ed. New York: 
Oxford University Press; 2008.
8. WordNet (r) 2.1 (2005). http://onlinedictionary.dataseg-
ment.com/word/applicability.
9. EUROPLAN – European Project for Rare Disease National 
Plans Development. Available from:  www.europlanproject.
eu/Resources/docs/2008-2011_3.EUROPLANIndica-
tors.pdf.
10. EUCERD – European Union Committee of Experts on 
Rare Disease. EUCERD Recommendation on core indica-
tors for rare disease national plans/strategies. Available from: 
http://ec.europa.eu/health/rare_diseases/docs/eucerd_in-
dicators_en.pdf.
11. Doran GT. There’s a S.M.A.R.T. way to write manage-
ment’s goals and objectives. Manag Rev (AMA FORUM) 
1981;70(11):35-6.
attainable, acceptable for the target group and realisti-
cally achievable according to the available resources in 
terms of time, money and knowledge.
In conclusion, the pilot study on the usability of the 
core indicators highlighted their usefulness in giving 
a snapshot of the main areas of concern for national 
planning for RDs. Moreover, a synthetic representa-
tion allows for defining common policies at European 
level; e.g. the situation about RDs coding, while show-
ing heterogeneity in the system adopted for coding 
RDs in the Country, allows for identifying a system 
that is likely to be used in most, hopefully all, of the 
Member States.
The core indicators represent an excellent oppor-
tunity to share knowledge and comparability among 
Member States. However, it is important to acknowl-
edge the strengths and weaknesses of the single tools 
and to be aware that quantitative indicators may not 
reflect qualitative substantial aspects of the issue that 
they are measuring. This may be studied with appropri-
ate and different tools, according to the objectives that 
have been set for the matter under study.
Acknowledgements
This work was carried out thanks to the support of 
the EUCERD Joint Action: Working for Rare Diseases 
(Grant 20112201), which was funded by the European 
Commission (Executive Agency for Health and Con-
sumers, now CHAFEA) in the context of the European 
Union’s Second Community Programme in the field of 
Health.
Conflict of interest statement
There are no potential conflicts of interest or any fi-
nancial or personal relationships with other people or 
organizations that could inappropriately bias conduct 
and findings of this study.
Submitted on 26 March 2015.
Accepted on 3 September 2015.


Exploring the usability of EUCERD core indicators for rare diseases

Directory of Open Access Journals

http://www.annali-iss.eu/article/download/305/165

Exploring the usability of EUCERD core indicators for rare diseases

Abstract

Similar works

Full text

Available Versions

Annali dell'Istituto Superiore di Sanità

Directory of Open Access Journals