Comprehensive assessment of complex technologies: integrating various aspects in health technology assessment

Objectives: Despite recent development of health technology assessment (HTA) methods, there are still methodological gaps for the assessment of complex health technologies. The INTEGRATE-HTA guidance for effectiveness, economic, ethical, socio-cultural, and legal aspects, deals with challenges when assessing complex technologies, such as heterogeneous study designs, multiple stakeholder perspectives, and unpredictable outcomes. The objective of this article is to outline this guidance and describe the added value of integrating these assessment aspects. Methods: Different methods were used to develop the various parts of the guidance, but all draw on existing, published knowledge and were supported by stakeholder involvement. The guidance was modified after application in a case study and in response to feedback from internal and external reviewers. Results: The guidance consists of five parts, addressing five core aspects of HTA, all presenting stepwise approaches based on the assessment of complexity, context, and stakeholder involvement. The guidance on effectiveness, health economics and ethics aspects focus on helping users choose appropriate, or further develop, existing methods. The recommendations are based on existing methods’ applicability for dealing with problems arising with complex interventions. The guidance offers new frameworks to identify socio-cultural and legal issues, along with overviews of relevant methods and sources. Conclusions: The INTEGRATE-HTA guidance outlines a wide range of methods and facilitates appropriate choices among them. The guidance enables understanding of how complexity matters for HTA and brings together assessments from disciplines, such as epidemiology, economics, ethics, law, and social theory. This indicates relevance for a broad range of technologies

Guidance on the use of logic models in health technology assessments of complex interventions

Challenges in assessments of health technologies In recent years there have been major advances in the development of health technology assessment (HTA). However, HTA still has certain limitations when assessing technologies, which fi are complex, i.e. consist of several interacting components, target different groups or organisational levels, have multiple and variable outcomes, and/or permit a certain degree of flexibility or tailoring; fi are context-dependent, with HTA usually focusing on the technology rather than on the system within which it is used; fi perform differently depending on the way they are implemented; and/or fi have distinct effects on different individuals. Logic models are one important means of conceptualising and handling complexity in HTAs or systematic reviews (SRs) of complex technologies, as well as integrating the findings of multi-component HTAs. A logic model is described as “… a graphic description of a system … designed to identify important elements and relationships within that system”. When evaluating complex health technologies, logic models can serve an instrumental purpose at every stage of the HTA/SR process, from scoping the topic of the HTA/SR, including formulating the question and defining the intervention; conducting the HTA/SR; interpreting results and making the HTA/SR relevant for decision makers to implement in policy and practice. Purpose and scope of the guidance This guidance is targeted at commissioners, producers and users of guidelines, HTAs and SRs with an interest in using logic models as an overarching framework for their work. It aims to make the use of logic models as straightforward as possible by facilitating the systematic identification or development as well as utilisation of different types and sub-types of logic models. In principle, logic models are a useful tool in any kind of SR or HTA, as they aid with the conceptualisation of the intervention and the review question. This is particularly useful for complex technologies, where conceptualising the intervention and its implementation within a system is critical. In addition, logic models can enhance communication within the HTA/SR team and with relevant stakeholders. Three types of logic model are described: With a priori logic models the logic model is specified upfront and remains unchanged during the HTA/SR process. With iterative logic models the logic model is subject to continual modification throughout the course of an HTA/SR. The staged logic model harnesses the strengths of both a priori and iterative approaches by pre-specifying revision points at which major data inputs are anticipated. In | 6 addition, two subtypes are identified, namely logic models that seek to represent structure (system-based) and those that focus on processes or activities (process-orientated). This guidance offers direction on how to choose between distinct types and sub-types of logic models, describes each logic model type and its application in detail, and provides templates for getting started with the development of an HTA/SR-specific logic model. Development of the guidance This guidance was informed by a combination of (i) systematic searches for published examples of logic models supplemented by purposive sampling of iterative logic modelling approaches; (ii) searches for existing guidance on the use of logic models in primary research, SRs and HTAs; (iii) development of two draft templates for system-based and process-orientated logic models in an iterative process within the research team and in consultation with external methodological experts; and (iv) application of these draft templates in multiple SRs and one HTA of different complex health technologies covering technical, educational and policy interventions in environmental health, e-learning for health professionals and models of palliative care. Application of this guidance For a comprehensive integrated assessment of a complex technology we have developed a five step process, the INTEGRATE-HTA model. In Step 1 the HTA objective and the technology are defined with the support from a panel of stakeholders. A system-based logic model is developed in Step 2. It provides a structured overview of technology, the context, implementation issues, and relevant patient groups. It then frames the assessment of the effectiveness, as well as economic, ethical, legal, and socio-cultural aspects in Step 3. In Step 4 a graphical overview of the assessment results, structured by the logic model, is provided. Step 5 is a structured decision-making process informed by the HTA (and is thus not formally part of the HTA but follows it). Logic models therefore form an integral element of the INTEGRATE-HTA model but may also be useful in individual steps. This guidance starts off by offering support in identifying and, as needed, adapting existing logic models from the literature or developing an HTA-/SR-specific logic model de novo. In either case, the user will need to decide upfront whether to pursue an a priori, staged or iterative approach to logic modelling, and the guidance offers criteria on how to decide between these distinct types of logic modelling. The system-based and process-orientated logic model templates provide a starting point for the de novo development of either type of logic model. The guidance also discusses the advantages and drawbacks of adopting the system-based or process-orientated sub-type, and offers some general considerations in applying logic models, such as the variety of data sources used, transparency in reporting and necessary trade-offs between comprehensiveness and complexity of the logic model in communicating with stakeholders. For a priori logic modelling, a six-step process comprises: (1) defining the PICO elements of the HTA/SR as well as relevant aspect of context and implementation; (2) deciding on a system- vs. process-orientated logic model subtype with the former focusing on a conceptualization of the question and the latter more concerned with an explanation of the pathways from the intervention to the outcomes; (3) populating the logic model template with information obtained through literature searches, discussions within the author team and consultations with content experts; (4) asking stakeholders for input and refining the logic model accordingly; (5) repeating steps 3 and 4 until all members of the author team agree that the logic model accurately represents the framework for the specific HTA/SR; and (6) publishing the final logic model with the protocol of the HTA or SR. This logic model remains unchanged during the HTA/SR process. For iterative logic modelling, a five-step process includes: (1) creating an initial logic model as a starting point for subsequent exploration, where a logic model template is used to create an initial logic model de novo; (2) identifying data on the whole system or entire process, or on individual components of the model, where data may come from stakeholders, the review team, ongoing primary research or the published literature; (3) making 7 | changes to the initial logic model repeatedly and at any point of the review and documenting these changes carefully; (4) creating a new numbered version of the logic model, where changes are considered substantive or stepwise; and (5) recording a definitive version of the logic model for the purpose of publication within the final HTA/SR report. It is recognised that this version of the logic model is only definitive with regard to the specific project timeframe and may well be subject to subsequent modification by the HTA/SR team, or indeed by other teams. For staged logic modelling, a four-step process consists of: (1) developing an initial logic model, using one of the templates and various mechanisms to populate them, in particular input from stakeholders and literature searches; (2) pre-specifying points within the HTA/SR process at which significant inputs, defined in terms of quantity or importance, are likely to have an impact on the structure and content of the HTA/SR and thus the logic model; (3) revisiting the logic model at the pre-specified review and revision points, and creating new and clearly labelled versions, documenting how and based on which data sources changes were made; and (4) presenting selected versions of the logic model, as a minimum the initial and the final logic models, in the HTA/ SR report. Conclusions Logic models are an important tool when conducting HTAs or SRs of complex health technologies, as they enhance transparency on underlying assumptions and help understand complexity by depicting the entire system, its parts and the interactions between intervention and outcomes; they also play a key role in integrating across different parts of a multi-component HTA. Nonetheless, logic models are not a panacea in addressing or resolving complexity and each type shows its specific strengths and limitations. This guidance provides a stateof-the-art overview of current practices in the use of logic models within HTAs and SRs. By providing templates for generating a logic model de novo, it aims to make the process of logic model development and application as straightforward as possible. Certain types and sub-types of logic models are more or less suitable depending on the technology concerned and the HTA/SR question addressed and approach pursued. This guidance offers a series of considerations on how to choose between a priori, iterative and staged logic models, illustrated with example applications of each type

Die Erfassung von Publikationen aus der Datennutzung: Verfahren, Herausforderungen und Nutzen; Ein Erfahrungsbericht von Forschungsdatenzentren

Publikationen sind im Wissenschaftssystem das wesentliche Medium zur Kommunikation von Erkenntnissen und werden häufig als Indikator für wissenschaftliche Leistung herangezogen. Für Forschungsdatenzentren (FDZ) erfüllen Publikationen weitere Funktionen: Verknüpft mit den zugrundeliegenden Forschungsdaten dienen sie als wichtige Informationsquelle für potentielle Datennutzer:innen. Zudem wird die Anzahl der auf Basis von Nachnutzungen entstandenen Publikationen als ein Evaluationskriterium für die Datendienste der FDZ herangezogen und das, obwohl die Erfassung dieser Publikationen mit einer Reihe von Herausforderungen verbunden ist. Der vorliegende Artikel beschreibt die verschiedenen Vorgehensweisen von FDZ bei der Erfassung von Publikationen, die aus der Nutzung von Forschungsdaten entstehen, und diskutiert damit einhergehende Probleme der Fehl-, Über- und Untererfassung. Es zeigt sich, dass die verschiedenen Verfahren der Erfassung mit je unterschiedlichen Schwierigkeiten behaftet sind, die ihre Verwendung als Leistungsindikator relativieren. Einen Königsweg gibt es derzeit nicht; stattdessen wird eine Kombination aus unterschiedlichen Vorgehensweisen empfohlen, die auch in Abhängigkeit von der jeweiligen Größe des Datenbestands eines FDZ und den jeweiligen Nutzer:innen zu bewerten ist

Ethical analysis in HTA of complex health interventions

Background: In the field of health technology assessment (HTA), there are several approaches that can be used for ethical analysis. However, there is a scarcity of literature that critically evaluates and compares the strength and weaknesses of these approaches when they are applied in practice. In this paper, we analyse the applicability of some selected approaches for addressing ethical issues in HTA in the field of complex health interventions. Complex health interventions have been the focus of methodological attention in HTA. However, the potential methodological challenges for ethical analysis are as yet unknown. Methods: Six of the most frequently described and applied ethical approaches in HTA were critically assessed against a set of five characteristics of complex health interventions: multiple and changing perspectives, indeterminate phenomena, uncertain causality, unpredictable outcomes, and ethical complexity. The assessments are based on literature and the authors’ experiences of developing, applying and assessing the approaches. Results: The Interactive, participatory HTA approach is by its nature and flexibility, applicable across most complexity characteristics. Wide Reflective Equilibrium is also flexible and its openness to different perspectives makes it better suited for complex health interventions than more rigid conventional approaches, such as Principlism and Casuistry. Approaches developed for HTA purposes are fairly applicable for complex health interventions, which one could expect because they include various ethical perspectives, such as the HTA Core Model® and the Socratic approach. Conclusion: This study shows how the applicability for addressing ethical issues in HTA of complex health interventions differs between the selected ethical approaches. Knowledge about these differences may be helpful when choosing and applying an approach for ethical analyses in HTA. We believe that the study contributes to increasing awareness and interest of the ethical aspects of complex health interventions in general

Why is it so difficult to integrate ethics in Health Technology Assessment (HTA)? The epistemological viewpoint

Ethics has been identified as a key element in Health Technology Assessment (HTA) since its conception. However, ethical issues are still not frequently addressed explicitly in HTA. Several valuable reasons have been identified. The basis of the article is the claim that ethics is often not part of HTA for “epistemological reasons”. Hence, the main aim of the contribution is to explore in more details and emphasize them by using the fact/value dichotomy. Our conclusion is that current HTA configuration is dominantly based on the comparison among objective and empirically testable “facts”, whilst ethics is not empirically testable. In this sense, there is a sort of “epistemological gap”, which can explain why it is so difficult to integrate ethics in HTA. We suggest that the epistemological differences among the various domains of HTA are addressed more explicitly

Guidance on the integrated assessment of complex health technologies: the INTEGRATE-HTA model

Challenges in assessments of health technologies In recent years there have been major advances in the development of health technology assessment (HTA). However, HTA still has certain limitations when assessing technologies which are complex, i.e. consist of several interacting components, target different groups or organizational levels, have multiple and variable outcomes, and/or permit a certain degree of flexibility or tailoring (Craig et al., 2008), fi are context-dependent - current HTA usually focusses on the technology, not on the system within which it is used, fi perform differently depending on the way they are implemented, fi have different effects on different individuals. Furthermore, HTA usually assesses and appraises aspects side-by-side, while decision-making needs an integrated perspective on the value of a technology. In the EU-funded INTEGRATE-HTA project, we developed concepts and methods to deal with these challenges, which are described in six guidances. Because of the interactions, an integrated assessment needs to start from the beginning of the assessment. This guidance provides a systematic five-step-process for an integrated assessment of complex technologies (the INTEGRATE-HTA Model). Purpose and scope of the guidance The aim of the INTEGRATE-HTA project is to provide concepts and methods that enable a patient-centred, comprehensive, and integrated assessment of complex health technologies. The purpose of this guidance is to structure the overall HTA-process. The INTEGRATE-HTA Model outlines an integrated scoping process, a coordinated application of assessment methods for different aspects and an integrated and structured decision-making process. It is intended for HTA agencies, HTA researchers and those engaged in the evaluation of complex health technologies. As it links the assessment to the decision-making process, it also addresses HTA commissioners and other stakeholders using or planning HTAs. While all technologies are arguably complex, some are more complex than others. Applying this guidance might lead to a more thorough and therefore more time-consuming process. Depending on the degree of complexity, one might choose to follow the whole process as described in this guidance, or only focus on certain steps. The guidance provides an operational definition to assess the complexity of technologies which can be used to identify specific aspects that will need more attention than others. What the guidance does not provide is a post-hoc solution for assessments that have already been completed. | 6 Development of the guidance The INTEGRATE-HTA Model presented in this guidance was developed based on a systematic literature search on approaches for integration, on the experiences of traditional HTAs, as well as on the other methodological guidances developed in the INTEGRATE-HTA project. It was tested in a case study on palliative care and iteratively revised during the practical application. The guidance was again revised after internal and external peer-review. Application of this guidance For a comprehensive integrated assessment of a complex technology, we developed a five-step process, the INTEGRATE-HTA model. In Step 1, the HTA objective and the technology are defined with the support from a panel of stakeholders. An initial logic model is developed in Step 2. The initial logic model provides a structured overview of the technology, the context, implementation issues, and relevant patient groups. It then frames the assessment of the effectiveness, as well as economic, ethical, legal, and socio-cultural aspects in Step 3. In Step 4, a graphical overview of the assessment results, structured by the logic model, is provided. Step 5 is a structured decision-making process informed by the HTA (and is thus not formally part of the HTA, but follows it). fi Step 1: In step 1, the technology under assessment and the objective of the HTA are defined. Especially for complex technologies, such as palliative care, the definition of the technology alone is a challenge that must not be underestimated. It is recommended to do this based on a tentative literature review and with the support of stakeholder advisory panels (SAPs) which should comprise clinical experts, academics, patients, possibly their relatives and/or other caretakers, and the public. The setting of an objective considering all relevant aspects of complexity and structured by assessment criteria is important. The assessment criteria will usually reflect values of the stakeholders as well as the input from the theoretical, methodological and empirical literature. fi Step 2: In step 2, an initial logic model is developed (see Guidance on the use of logic models in health technology assessments of complex interventions). The model provides a structured overview on participants, interventions, comparators, and outcomes. Parallel to this, groups of patients that are distinguished by different preferences and treatment moderators (see Guidance for the assessment of treatment moderation and patients’ preferences) are identified. Specific context and implementation issues are also identified as part of the initial logic model (see Guidance for the Assessment of Context and Implementation in Health Technology Assessments (HTA) and Systematic Reviews of Complex Interventions). The product of this step is the logic model as a graphical representation of all aspects and their interactions that are relevant for the assessment of the complex technology. fi Step 3: In step 3, the logic model serves as a conceptual framework that guides the evidence assessment. Depending on the specific aspect (e.g. effectiveness, economic, ethical, socio-cultural, or legal aspects) different methods are available for the assessment (see Guidance for assessing effectiveness, economic aspects, ethical aspects, socio-cultural aspects and legal aspects in complex technologies). The outputs of step 3 are evidence reports and standardized evidence summaries for each assessment aspect (e.g. report on economics, report on ethical aspects, etc.). fi Step 4: In step 4, the assessment results of step 3 are structured using the logic model developed in step 2. Whereas the initial logic model in step 2 specifies what evidence is relevant, the extended logic model to assist decision-making in step 4 visualizes the assessment results as well as the interaction with respect to the HTA objectives. It also allows for the consideration of different scenarios depending on the variation in context, implementation and patient characteristics. 7 | fi Step 5: Step 5 involves a structured decision-making process and is not an integral part of the HTA in the narrow sense. Decision-making can be supported by applying quantitative e.g. MCDA- (Multi-criteria decision analysis) or qualitative decision support tools. Flexibility in the application of these tools by the decision committee is crucial, taking different decision settings and evidence needs into consideration. Conclusions In current HTA, different aspects are usually assessed and presented independent of each other. Context, implementation issues and patient characteristics are rarely considered. The INTEGRATE-HTA Model enables a coordinated assessment of all these aspects and addresses their interdependencies. The perspective of stakeholders such as patients and professionals with their values and preferences is integrated in the INTEGRATE-HTA Model to obtain HTA results that are meaningful for all relevant stakeholders. Finally, health policy makers obtain an integrated perspective of the assessment results to achieve fair and legitimate conclusions at the end of the HTA process. The application of the model will usually require more time and resources than traditional HTA. An initial assessment of the degree and the character of complexity of a technology might be helpful to decide whether or not the whole process or only specific elements will be applied

Stakeholder involvement throughout health technology assessment: an example from palliative care

Objectives: Internationally, funders require stakeholder involvement throughout health technology assessment (HTA). We report successes, challenges, and lessons learned from extensive stakeholder involvement throughout a palliative care case study that demonstrates new concepts and methods for HTA. Methods: A 5-step “INTEGRATE-HTA Model” developed within the INTEGRATE-HTA project guided the case study. Using convenience or purposive sampling or directly / indirectly identifying and approaching individuals / groups, stakeholders participated in qualitative research or consultation meetings. During scoping, 132 stakeholders, aged ≥ 18 years in seven countries (England, Italy, Germany, The Netherlands, Norway, Lithuania, and Poland), highlighted key issues in palliative care that assisted identification of the intervention and comparator. Subsequently stakeholders in four countries participated in face–face, telephone and / or video Skype meetings to inform evidence collection and / or review assessment results. An applicability assessment to identify contextual and implementation barriers and enablers for the case study findings involved twelve professionals in the three countries. Finally, thirteen stakeholders participated in a mock decision-making meeting in England. Results: Views about the best methods of stakeholder involvement vary internationally. Stakeholders make valuable contributions in all stages of HTA; assisting decision making about interventions, comparators, research questions; providing evidence and insights into findings, gap analyses and applicability assessments. Key challenges exist regarding inclusivity, time, and resource use. Conclusion: Stakeholder involvement is feasible and worthwhile throughout HTA, sometimes providing unique insights. Various methods can be used to include stakeholders, although challenges exist. Recognition of stakeholder expertise and further guidance about stakeholder consultation methods is needed

Symposion Forschungsdateninfrastruktur

Wie sollen qualitative sozialwissenschaftliche Daten gespeichert, archiviert und nachgenutzt werden? Das Symposion nimmt diese Frage auf und diskutiert sie anhand von Beiträgen aus Forschungsdatenzentren, die auf qualitative Daten spezialisiert sind. Neben der Vorstellung und Darstellung dieser Einrichtungen werden die mit der Archivierung einhergehenden Probleme und deren Lösungen angesprochen. How should qualitative social science data be stored, archived and re-used? The symposium takes up this question and discusses it on the basis of contributions from research data centres that specialise in qualitative data. In addition to the presentation of these institutions, the problems associated with archiving and their solutions will be discussed

Lay and professional stakeholder involvement in scoping palliative care issues: Methods used in seven European countries

BACKGROUND: Stakeholders are people with an interest in a topic. Internationally, stakeholder involvement in palliative care research and health technology assessment requires development. Stakeholder involvement adds value throughout research (from prioritising topics to disseminating findings). Philosophies and understandings about the best ways to involve stakeholders in research differ internationally. Stakeholder involvement took place in seven countries (England, Germany, Italy, Lithuania, the Netherlands, Norway and Poland). Findings informed a project that developed concepts and methods for health technology assessment and applied these to evaluate models of palliative care service delivery. AIMS: To report on stakeholder involvement in the INTEGRATE-HTA project and how issues identified informed project development. DESIGN: Using stakeholder consultation or a qualitative research design, as appropriate locally, stakeholders in seven countries acted as 'advisors' to aid researchers' decision making. Thematic analysis was used to identify key issues across countries. SETTING/PARTICIPANTS: A total of 132 stakeholders (82 professionals and 50 'lay' people) aged ⩾18 participated in individual face-to-face or telephone interviews, consultation meetings or focus groups. RESULTS: Different stakeholder involvement methods were used successfully to identify key issues in palliative care. A total of 23 issues common to three or more countries informed decisions about the intervention and comparator of interest, sub questions and specific assessments within the health technology assessment. CONCLUSION: Stakeholders, including patients and families undergoing palliative care, can inform project decision making using various involvement methods according to the local context. Researchers should consider local understandings about stakeholder involvement as views of appropriate and feasible methods vary. Methods for stakeholder involvement, especially consultation, need further development

Integrated assessment of home based palliative care with and without reinforced caregiver support: ‘A demonstration of INTEGRATE-HTA methodological guidances’

About this Executive Summary This comprehensive executive summary reports on a case study designed to demonstrate the application of a number of the key concepts and methods developed in the INTEGRATE-HTA project to the assessment of complex technologies. The case study focuses on models of home based palliative care with and without an additional element of caregiver support, known as reinforced and non-reinforced home based palliative care respectively. What is already known about the topic With changing disease patterns in Europe, increasingly complex health care technologies, such as palliative care, have gained importance. Current HTA methods rarely take account of wider legal, ethical and socio-cultural issues or context and implementation and are not adequately equipped to assess highly complex technologies, despite considerable progress in recent years. As a result, HTA is rarely applied to highly complex health technologies. What this case study report adds The INTEGRATE-HTA project developed concepts and methods for the assessment of complex technologies, taking into account legal, ethical and socio-cultural issues as well as context and implementation. This case study is designed to demonstrate the application of a number of the key concepts and methods developed in the INTEGRATE-HTA project to home based palliative care, with and without an additional element of caregiver support, as an example of a complex technology; known as “reinforced” and “non-reinforced” home based palliative care respectively. The INTEGRATE-HTA model, developed to enable integration of relevant assessment aspects, is used to structure this report. The case study reports on the application of some of the concepts and methods developed within the INTEGRATE-HTA project to the assessment of effectiveness as well as economic, sociocultural, ethical, and legal aspects; patient preferences and patient-specific moderators of treatment and context and implementation issues related to reinforced and non-reinforced home based palliative care. The highly complex nature of reinforced and non-reinforced home based palliative care is illustrated through an assessment of complexity characteristics. | 4 The case study also involved extensive lay and professional stakeholder involvement, using a variety of methods, to inform the HTA process at key stages throughout the project. Key messages are highlighted for both HTA research and for palliative care. The implications for research and practice The concepts and methods developed in the INTEGRATE-HTA project have been shown to be feasible and to have the potential to offer added value, but require further development and application in the assessment of other complex technologies. Reinforced and non-reinforced models palliative care are highly complex. The case study findings offer some insights into their effectiveness as well as economic, sociocultural, ethical, and legal issues; patient preferences and patient-specific moderators of treatment as well as context and implementation issues