Challenges in assessments of health technologies
In recent years there have been major advances in the development of health technology assessment (HTA). However, HTA still has certain limitations when assessing technologies, which
fi are complex, i.e. consist of several interacting components, target different groups or organisational levels, have multiple and variable outcomes, and/or permit a certain degree of flexibility or tailoring;
fi are context-dependent, with HTA usually focusing on the technology rather than on the system within which it is used;
fi perform differently depending on the way they are implemented; and/or
fi have distinct effects on different individuals.
Logic models are one important means of conceptualising and handling complexity in HTAs or systematic reviews (SRs) of complex technologies, as well as integrating the findings of multi-component HTAs. A logic model is described as “… a graphic description of a system … designed to identify important elements and relationships within that system”. When evaluating complex health technologies, logic models can serve an instrumental purpose at every stage of the HTA/SR process, from scoping the topic of the HTA/SR, including formulating the question and defining the intervention; conducting the HTA/SR; interpreting results and making the HTA/SR relevant for decision makers to implement in policy and practice.
Purpose and scope of the guidance
This guidance is targeted at commissioners, producers and users of guidelines, HTAs and SRs with an interest in using logic models as an overarching framework for their work. It aims to make the use of logic models as straightforward as possible by facilitating the systematic identification or development as well as utilisation of different types and sub-types of logic models. In principle, logic models are a useful tool in any kind of SR or HTA, as they aid with the conceptualisation of the intervention and the review question. This is particularly useful for complex technologies, where conceptualising the intervention and its implementation within a system is critical. In addition, logic models can enhance communication within the HTA/SR team and with relevant stakeholders.
Three types of logic model are described: With a priori logic models the logic model is specified upfront and remains unchanged during the HTA/SR process. With iterative logic models the logic model is subject to continual modification throughout the course of an HTA/SR. The staged logic model harnesses the strengths of both a priori and iterative approaches by pre-specifying revision points at which major data inputs are anticipated. In
| 6
addition, two subtypes are identified, namely logic models that seek to represent structure (system-based) and those that focus on processes or activities (process-orientated).
This guidance offers direction on how to choose between distinct types and sub-types of logic models, describes each logic model type and its application in detail, and provides templates for getting started with the development of an HTA/SR-specific logic model.
Development of the guidance
This guidance was informed by a combination of (i) systematic searches for published examples of logic models supplemented by purposive sampling of iterative logic modelling approaches; (ii) searches for existing guidance on the use of logic models in primary research, SRs and HTAs; (iii) development of two draft templates for system-based and process-orientated logic models in an iterative process within the research team and in consultation with external methodological experts; and (iv) application of these draft templates in multiple SRs and one HTA of different complex health technologies covering technical, educational and policy interventions in environmental health, e-learning for health professionals and models of palliative care.
Application of this guidance
For a comprehensive integrated assessment of a complex technology we have developed a five step process, the INTEGRATE-HTA model. In Step 1 the HTA objective and the technology are defined with the support from a panel of stakeholders. A system-based logic model is developed in Step 2. It provides a structured overview of technology, the context, implementation issues, and relevant patient groups. It then frames the assessment of the effectiveness, as well as economic, ethical, legal, and socio-cultural aspects in Step 3. In Step 4 a graphical overview of the assessment results, structured by the logic model, is provided. Step 5 is a structured decision-making process informed by the HTA (and is thus not formally part of the HTA but follows it). Logic models therefore form an integral element of the INTEGRATE-HTA model but may also be useful in individual steps.
This guidance starts off by offering support in identifying and, as needed, adapting existing logic models from the literature or developing an HTA-/SR-specific logic model de novo. In either case, the user will need to decide upfront whether to pursue an a priori, staged or iterative approach to logic modelling, and the guidance offers criteria on how to decide between these distinct types of logic modelling. The system-based and process-orientated logic model templates provide a starting point for the de novo development of either type of logic model. The guidance also discusses the advantages and drawbacks of adopting the system-based or process-orientated sub-type, and offers some general considerations in applying logic models, such as the variety of data sources used, transparency in reporting and necessary trade-offs between comprehensiveness and complexity of the logic model in communicating with stakeholders.
For a priori logic modelling, a six-step process comprises: (1) defining the PICO elements of the HTA/SR as well as relevant aspect of context and implementation; (2) deciding on a system- vs. process-orientated logic model subtype with the former focusing on a conceptualization of the question and the latter more concerned with an explanation of the pathways from the intervention to the outcomes; (3) populating the logic model template with information obtained through literature searches, discussions within the author team and consultations with content experts; (4) asking stakeholders for input and refining the logic model accordingly; (5) repeating steps 3 and 4 until all members of the author team agree that the logic model accurately represents the
framework for the specific HTA/SR; and (6) publishing the final logic model with the protocol of the HTA or SR. This logic model remains unchanged during the HTA/SR process.
For iterative logic modelling, a five-step process includes: (1) creating an initial logic model as a starting point for subsequent exploration, where a logic model template is used to create an initial logic model de novo; (2) identifying data on the whole system or entire process, or on individual components of the model, where data may come from stakeholders, the review team, ongoing primary research or the published literature; (3) making
7 |
changes to the initial logic model repeatedly and at any point of the review and documenting these changes carefully; (4) creating a new numbered version of the logic model, where changes are considered substantive or stepwise; and (5) recording a definitive version of the logic model for the purpose of publication within the final HTA/SR report. It is recognised that this version of the logic model is only definitive with regard to the specific project timeframe and may well be subject to subsequent modification by the HTA/SR team, or indeed by other teams.
For staged logic modelling, a four-step process consists of: (1) developing an initial logic model, using one of the templates and various mechanisms to populate them, in particular input from stakeholders and literature searches; (2) pre-specifying points within the HTA/SR process at which significant inputs, defined in terms of quantity or importance, are likely to have an impact on the structure and content of the HTA/SR and thus the logic model; (3) revisiting the logic model at the pre-specified review and revision points, and creating new and clearly labelled versions, documenting how and based on which data sources changes were made; and (4) presenting selected versions of the logic model, as a minimum the initial and the final logic models, in the HTA/ SR report.
Conclusions
Logic models are an important tool when conducting HTAs or SRs of complex health technologies, as they enhance transparency on underlying assumptions and help understand complexity by depicting the entire system, its parts and the interactions between intervention and outcomes; they also play a key role in integrating across different parts of a multi-component HTA. Nonetheless, logic models are not a panacea in addressing or resolving complexity and each type shows its specific strengths and limitations. This guidance provides a stateof-the-art overview of current practices in the use of logic models within HTAs and SRs. By providing templates for generating a logic model de novo, it aims to make the process of logic model development and application as straightforward as possible. Certain types and sub-types of logic models are more or less suitable depending on the technology concerned and the HTA/SR question addressed and approach pursued. This guidance offers a series of considerations on how to choose between a priori, iterative and staged logic models, illustrated with example applications of each type
