Alban, L

Tomassone, L

English

Institutional Research Information System University of Turin

3 - 5 MAY 2022 · COPENHAGEN · DENMARKABSTRACTSwww.icahs4.org4th International Conference on Animal Health Surveillance3- 5 May 2022 · DenmarkAnimal Health Surveillance64  [O18] A USERS’ APPROACH TO ASSESSING EVALUATION TOOLS FOR INTEGRATED SURVEILLANCE OF AMU AND AMR  - THE UPDATED COEVALAMR METHODOLOGY  Lis Alban1, Marion Bordier2, Barbara Haesler3, Lucie Collineau4, Laura Tomassone5, Houda Bennai 6, Cécile Aenishaenslin7, Madelaine Norström8, Carola Sauter-Louis9, Maurizio Aragrande10, Maria-Eleni Filippitzi11, Marianne Sandberg12  1Danish Agriculture & Food Council, University of Copenhagen, København V, Denmark 2Astre, Univ Montpellier, Cirad, Inrae, Université de Montpellier, National Institute of Veterinary Research, Montpellier Cedex 5, France 3Royal Veterinary College, Royal Veterinary College, Dept. of Pathobiology and Population Sciences,, Hatfield, United Kingdom 4French Agency for Food, Environmental and Occupational Health & Safety (Anses), Lyon, France 5University of Turin, Department of Veterinary Sciences, Grugliasco, Italy 6Department of Pathobiology and Population Sciences, Royal Veterinary College, Royal Veterinary College, London, United Kingdom, Dept. of Pathobiology and Population Sciences, London, United Kingdom 7Faculty of Veterinary Medicine, Université de Montréal, Faculty of Veterinary Medicine, University of Montreal, Saint-Hyacinthe (Quebec), Université de Montréal, Canada, Canada 8Norwegian Veterinary Institute, Norwegian Veterinary Institute, Aas, Norway, Oslo, Norway 9Friedrich-Loeffler-Institut, Friedrich-Loeffler-Institut, Greifswald, Germany, Institute of Epidemiology,, Greifswald - Insel Riems, Germany 10University of Bologna, Department of Agricultural and Food Sciences, University of Bologna, Bologna, Italy, Department of Agricultural and Food Sciences, Bologna, Italy 11Department of Epidemiology and Public Health, Sciensano Belgium, Aristotle University of Thessaloniki, Veterinary Epidemiology, Laboratory of Animal Health Economics, Thessaloniki, Belgium 12National Food Institute, National Food Institute, Technical University of Denmark, Research Group for Genomic Epidemiology, Lyngby, Denmark  Summary Regular evaluation of any surveillance is needed to ensure system effectiveness. In an international research network, called CoEvalAMR, we have developed guidelines for evaluation of integrated surveillance of antimicrobial use (AMU) and antimicrobial resistance (AMR). Different evaluation tools have been assessed using a methodology developed with a focus on the user’s experience in a country case format. The original methodology consisted of a brief description of the case study, followed by an assessment of 11 functional aspects and eight content themes and a SWOT-like analysis. Based on the first round of cases, we have updated the methodology to increase the evaluation coverage and standardisation. The update was based on 1) a brainstorm with CoEvalAMR network members, 2) results of a questionnaire with 23 expected tool users, and 3) a comparison with the SISOT framework developed by the Tripartite (FAO/WHO/OIE). Tool users prefer easy-to-use tools addressing many issues in a way which can be communicated straightforwardly to decision-makers and stakeholders. However, no tool can do this, and each tool has its advantages and limitations. This makes it important to reiterate that the choice of tool depends upon the specific evaluation purpose and that adequate resources, time and training are needed before an evaluation will provide a useful output. Our updated methodology can be used by tool users to share experience regarding different tools, and hereby assisting others in identifying the most suited tool for their purposes.  Introduction/background  It is a common good to keep antimicrobials effective for the coming generations. One way of supporting this goal is to have surveillance covering different domains and sectors in an integrated manner (1).   To ensure surveillance effectiveness and efficiency, there is a need to evaluate existing surveillance programmes at regular intervals (2). Several tools have been developed by either research groups or national/international organisations to assist in such evaluations. Evaluations may be done by different types of professionals with diverse levels of experience in surveillance evaluation, various access to detailed data, and time to dedicate to the evaluation. Moreover, evaluations may be pursued with different goals. This makes it pertinent to choose the right tool for a given evaluation context, team and question.   During 2019-2020, an international network of scientists called CoEvalAMR8 aimed to develop guidance for assessment of integrated surveillance of antimicrobial use (AMU) and antimicrobial resistance  8 https://coevalamr.fp7-risksur.eu/ 65  (AMR). In this first phase of the CoEvalAMR project, a methodology was developed to gather user feedback on evaluation tools for integrated surveillance for AMU and AMR in an easy and standardised way (3). The focus was on gathering users’ subjective experience on the application of the tools, and the approach chosen was partly inspired by “Trip Advisor”. This implied use of questions such as “things that I liked or that the tool covered well” or “things that I struggled with when using this tool” (3).  The original methodology consisted of four different approaches, which complemented each other. The first covered a brief description of the case study. This was followed by 11 pre-defined functional aspects including workability regarding the need for data, time and people. The third approach covered an assessment of seven predefined content themes related to the tools’ scope.  The functional aspects and content themes were scored semi-quantitatively using a scale from 1-4, and a comment was requested explaining the score. The fourth approach consisted of the subjective perception of the tool assessors based on a strengths, weaknesses, opportunities, and threats (SWOT) using a SWOT-like approach.  During the first phase of the project, six tools were assessed using the described methodology, by applying them to eight national surveillance systems as country cases. The tools were: ATLASS, ECoSur, ISSEP, NEOH, PMP-AMR and SURVTOOLS. Each tool was assessed in between one and four country cases. A comprehensive description of this work can be found in Sandberg et al. (3) whereas Nielsen et al. (4) contains a description of the Danish work in detail. Moreover, a description of users’ experience for each country case studies can be found on the website of CoEval-AMR9.   Hence, ample experience was collected regarding assessment of the tools and the developed methodology. It was concluded that the methodology worked well, but that it could be improved with respect to coverage and standardisation. These aspects are currently being dealt within the second phase of the CoEvalAMR project, which will run from 2021 to 2022. The objective of the present paper is to present the updated methodology as well as the considerations behind the update.  Materials and Methods Lessons learned during phase 1 of relevance for updating of the methodology: In spring and summer 2021, a series of monthly virtual meetings within working group 4 (WG4) began, allowing members to convene and discuss how to update the methodology. The suggestions for further improvement largely centered on how to increase the objectivity by providing clearer definitions for the aspects to score as well as the different levels of the scores. Moreover, it was of interest to broaden the coverage of the methodology to e.g., include additional functional aspects or content themes. Finally, it was investigated whether the four questions behind the SWOT-like analysis could be clarified further.  Analysis of expectations of tool users: In the first phase, a survey was conducted to gather information on evaluation of existing or planned AMU/AMR surveillance and people’s use of available tools as well as their expectations for the tools. An analysis of the 23 answers to the questionnaire undertaken by Ruegg et al. (5) was used during the updating of the methodology. The respondents pointed among others to: i) The tools should be standardised, ii) they should provide clear results and evidence of data integration quality that can be used with confidence in research or to inform decision making, iii) Standardised guidance should be available regarding which tool to use, depending on the evaluation needs, iv) All tools should be free and easy to use with strong services available to users to provide guidance, v) It should be possible to undertake different levels of evaluation from superficial to deep, to enable either a rapid “general overview” evaluation to be conducted with the ability to evaluate certain components deeper. Hence, essentially, people want a one-stop shop for everything and standardised tools that are flexible and easy-to-use.  Comparison with the SISOT Evaluation Matrix: It was decided to compare the developed methodology with the assessment process used in the SISOT evaluation matrix, recently developed by the Tripartite (WHO/FAO/OIE) of the United Nations (UN) (6). Hereby, it could be investigated whether and how to broaden the coverage. Moreover, suggestions for standardization of scores could be obtained.   Results The first approach, which contains an introduction to the case study, was amplified by a brief description of the tool. For this, nine aspects were identified such as sector covered, type of tool, languages, accessibility etc.   9 https://guidance.fp7-risksur.eu/welcome/case-studies/ 66   Table 1 shows the final list of functional aspects. Four new were identified on top of the existing used in the first phase: i) Number of people to be interviewed, ii) Costs, iii) Required level of knowledge of users, and iv) Training to get acquainted with the tool. One aspect was moved to the tool description, and two were deleted because they were considered redundant.   Table 1. Description of the updated list of 12 functional aspects, sorted into five groups – scales and scores were defined for each aspect (not shown)  Group Functional aspect Ease of use User-friendliness related to wording, guidance and layout of the tool or framework  Analysis and interpretation of evaluation data   Amount and complexity of data required, where complexity is defined as different kinds of data from multiple sources in different formats or primary data collection required Scope Can address the stated evaluation objectives  Evaluation of OH aspects (collaboration across sectors /disciplines, knowledge integration, added value of OH approach, etc.) Pre-requisites before use  Required level of knowledge of users regarding surveillance, epidemiology and evaluation  Training to get acquainted with the tool or framework Time and resources Costs related to the access and use of the tool or framework  Number of people in the evaluation team  Number of people to be interviewed   Duration of the evaluation process Outputs Generation of actionable evaluation outputs   Similarly, two new content themes were suggested during the updating: “governance” and “impact”. The “impact” content theme would be divided into short-term, intermediate and long-term effects in line with the recommendation by Aenishaenslin et al. (1). This would result in a total of nine content themes. It was decided to maintain the scale from 1-4 for the functional aspects and the content themes (Figure 1).   Visualisation of the results was improved by trying out different approaches such as radar diagrams or green dots or stars as a Trip Advisor-like way of presenting the scoring of functional aspects and content themes.  Finally, the phrasing of the four questions for the SWOT-like analysis was updated emphasising further on strengths, weaknesses, opportunities and threats.     67  Figure 1. Overview of the updated CoEvalAMR methodology focusing on users’ assessment of evaluation tools. Numbers in bracket represent the number of questions to address the subject     Discussion In the first part of the CoEvalAMR network project, it was found that the users scored the individual functional aspects and content themes in a slightly subjective way. As the project went along, a higher degree of consensus arose regarding interpretation of the methodology including way of scoring (3). We expect that with the update of the methodology, the degree of subjectivity will be lower, but still present.   The respondents of the questionnaire survey undertaken as part of phase 1 of CoEvalAMR pointed to the need for standardisation of tools (5). In response to that, we have focused on standardising our methodology by introducing clearer definitions and scales. It is relevant to discuss to which extent further standardisation of our methodology is needed. One may argue that standardisation is an essential requirement in academia, but a less important issue for persons involved with the authorities. Focus in the latter sector is on the process initiated as part of evaluation, where the tool may act as an initiator of a larger process. In other words, the intention of our work is not to compare tools, but to describe the tools to such an extent that the future users will be guided in choosing the right tool for their purpose.   According to the survey undertaken among evaluation users, the users prefer tools that are easy to use, without much need for preparation or training (5). However, results of such evaluations may not be sufficiently valuable. Still, it is relevant to discuss the balance between required training, allocated resources, details and overview. To address this, the intended outcome of the evaluation becomes crucial. This reiterates the need for careful description of the evaluation purpose before choosing the evaluation tool. In our updating of the methodology, we have been inspired by the SISOT matrix developed by the Tripartite (6). This matrix is very detailed and can be used for evaluating different kinds of tools and resources for use in zoonotic risk-reducing activities. It is flexible and can also be used to assess evaluation tools. The questions and possible ways of answering show how well-developed the SISOT matrix is.   Our revised CoEvalAMR tool is targeting integrated surveillance for AMU and AMR. It is simpler and quicker to use, while it still contains most of the elements that form part of the SISOT matrix.  The case studies reported by Sandberg et al. (3) and Nielsen et al. (4) covered both integrated surveillance programmes and single programmes. According to the Tripartite, multisectoral means that more than one sector is working together in a joint program or response to an event. However, it does not imply that all sectors must work together on all aspects. Similarly, multidisciplinary means collaboration across several disciplines. Taking a One Health approach means that all relevant sectors and disciplines are involved (2). One of the keywords for integrated surveillance is relevance, although this may be difficult to judge in some cases. The methodology we have developed is useful to provide an overview of the advantages and disadvantages of the tool investigated, irrespective of whether the tool was used for evaluation of an integrated or non-integrated surveillance system.  Evaluation of One Health surveillance is an active field, and there is a growing number of these evaluation tools/frameworks becoming available. In Sandberg et al. (3), six tools were retained for evaluation. The ambition in phase 2 of CoEvalAMR is to apply the updated evaluation methodology to other tools or frameworks, in accordance with the needs of the network members. Other persons 68  involved in surveillance evaluation are welcome to make use of our methodology and to share their results.  References  1. Aenishaenslin et al. (2021) Front. Vet. Sci. 8:611931. 2. WHO/FAO/OIE (2019) Taking a Multisectoral One Health Approach: A Tripartite Guide to Addressing Zoonotic Diseases in Countries 3. Sandberg et al. (2021) Front. Vet. Sci. 8:620998.  4. Nielsen et al. (2020) Clin. Microbiol. Infect. 26: 1606-16011. 5. Ruegg et al., in prep.  6. WHO/FAO/OIE (2021) SISOT Evaluation Matrix – User manual

A users’ approach to assessing evaluation tools for integrated surveillance of AMU and AMR - the updated CoEvalAMR methodology

https://iris.unito.it/bitstream/2318/1873416/1/Alban.pdf

A users’ approach to assessing evaluation tools for integrated surveillance of AMU and AMR - the updated CoEvalAMR methodology

Abstract

Similar works

Full text

Available Versions

Institutional Research Information System University of Turin