Developing and using ontologies in behavioural science: addressing issues raised [version 1; peer review: awaiting peer review]

Ontologies are ways of representing aspects of the world in terms of uniquely defined classes of ‘entities’ and relationships between them. They are widely used in biological science, data science and commerce because they provide clarity, consistency, and the ability to link information and data from different sources. Ontologies offer great promise as representational systems in behavioural science and could revolutionise descriptions of studies and findings, and the expression of models and theories. This paper discusses issues that have been raised about using ontologies in behavioural science and how these can be addressed. The issues arise partly from the way that ontologies represent information, which can be perceived as reductionist or simplistic, and partly from issues to do with their implementation. However, despite the simplicity of their structure, ontologies can represent complex entities that change over time, as well as their inter-relationships and highly nuanced information about them. Nevertheless, ontologies are only one of many ways of representing information and it is important to recognise when other forms are more efficient. With regard to implementation, it is important to build ontologies with involvement from the communities who will be using them. Far from constraining intellectual creativity, ontologies that are broadly-based can facilitate expression of nuance, comparison of findings and integration of different approaches and theories. Maintaining and updating ontologies remain significant challenges but can be achieved through establishing and coordinating communities of practice

Predicting outcomes of smoking cessation interventions in novel scenarios using ontology-informed, interpretable machine learning

Background Systematic reviews of effectiveness estimate the relative average effects of interventions and comparators in a set of existing studies e.g., using rate ratios. However, policymakers, planners and practitioners require predictions about outcomes in novel scenarios where aspects of the interventions, populations or settings may differ. This study aimed to develop and evaluate an ontology-informed, interpretable machine learning algorithm to predict smoking cessation outcomes using detailed information about interventions, their contexts and evaluation study methods. This is the second of two linked papers on the use of machine learning in the Human Behaviour-Change Project. Methods The study used a corpus of 405 reports of randomised trials of smoking cessation interventions from the Cochrane Library database. These were annotated using the Behaviour Change Intervention Ontology to classify, for each of 971 study arms, 82 features representing details of intervention content and delivery, population, setting, outcome, and study methodology. The annotated data was used to train a novel machine learning algorithm based on a set of interpretable rules organised according to the ontology. The algorithm was evaluated for predictive accuracy by performance in five-fold 80:20 cross-validation, and compared with other approaches. Results The machine learning algorithm produced a mean absolute error in prediction percentage cessation rates of 9.15% in cross-validation, outperforming other approaches including an uninterpretable ‘black-box’ deep neural network (9.42%), a linear regression model (10.55%) and a decision tree-based approach (9.53%). The rules generated by the algorithm were synthesised into a consensus rule set to create a publicly available predictive tool to provide outcome predictions and explanations in the form of rules expressed in terms of predictive features and their combinations. Conclusions An ontologically-informed, interpretable machine learning algorithm, using information about intervention scenarios from reports of smoking cessation trials, can predict outcomes in new smoking cessation intervention scenarios with moderate accuracy.</ns3:p

Developing and using ontologies in behavioural science : addressing issues raised

Peer reviewe

The Behaviour Change Technique Ontology: Transforming the Behaviour Change Technique Taxonomy v1 [version 1; peer review: awaiting peer review]

BACKGROUND: The Behaviour Change Technique Taxonomy v1 (BCTTv1) specifies the potentially active content of behaviour change interventions. Evaluation of BCTTv1 showed the need to extend it into a formal ontology, improve its labels and definitions, add BCTs and subdivide existing BCTs. We aimed to develop a Behaviour Change Technique Ontology (BCTO) that would meet these needs. METHODS: The BCTO was developed by: (1) collating and synthesising feedback from multiple sources; (2) extracting information from published studies and classification systems; (3) multiple iterations of reviewing and refining entities, and their labels, definitions and relationships; (4) refining the ontology via expert stakeholder review of its comprehensiveness and clarity; (5) testing whether researchers could reliably apply the ontology to identify BCTs in intervention reports; and (6) making it available online and creating a machine-readable version. RESULTS: Initially there were 282 proposed changes to BCTTv1. Following first-round review, 19 BCTs were split into two or more BCTs, 27 new BCTs were added and 26 BCTs were moved into a different group, giving 161 BCTs hierarchically organised into 12 logically defined higher-level groups in up to five hierarchical levels. Following expert stakeholder review, the refined ontology had 247 BCTs hierarchically organised into 20 higher-level groups. Independent annotations of intervention evaluation reports by researchers familiar and unfamiliar with the ontology resulted in good levels of inter-rater reliability (0.82 and 0.79, respectively). Following revision informed by this exercise, 34 BCTs were added, resulting in a final version of the BCTO containing 281 BCTs organised into 20 higher-level groups over five hierarchical levels. DISCUSSION: The BCT Ontology provides a standard terminology and comprehensive classification system for the content of behaviour change interventions that can be reliably used to describe interventions

The Behaviour Change Technique Ontology: Transforming the Behaviour Change Technique Taxonomy v1

Background: The Behaviour Change Technique Taxonomy v1 (BCTTv1) specifies the potentially active content of behaviour change interventions. Evaluation of BCTTv1 showed the need to extend it into a formal ontology, improve its labels and definitions, add BCTs and subdivide existing BCTs. We aimed to develop a Behaviour Change Technique Ontology (BCTO) that would meet these needs. Methods: The BCTO was developed by: (1) collating and synthesising feedback from multiple sources; (2) extracting information from published studies and classification systems; (3) multiple iterations of reviewing and refining entities, and their labels, definitions and relationships; (4) refining the ontology via expert stakeholder review of its comprehensiveness and clarity; (5) testing whether researchers could reliably apply the ontology to identify BCTs in intervention reports; and (6) making it available online and creating a machinereadable version. Results: Initially there were 282 proposed changes to BCTTv1. Following first-round review, 19 BCTs were split into two or more BCTs, 27 new BCTs were added and 26 BCTs were moved into a different group, giving 161 BCTs hierarchically organised into 12 logically defined higher-level groups in up to five hierarchical levels. Following expert stakeholder review, the refined ontology had 247 BCTs hierarchically organised into 20 higher-level groups. Independent annotations of intervention evaluation reports by researchers familiar and unfamiliar with the ontology resulted in good levels of inter-rater reliability (0.82 and 0.79, respectively). Following revision informed by this exercise, 34 BCTs were added, resulting in a final version of the BCTO containing 281 BCTs organised into 20 higher-level groups over five hierarchical levels. Discussion: The BCT Ontology provides a standard terminology and comprehensive classification system for the content of behaviour change interventions that can be reliably used to describe interventions

An ontology of mechanisms of action in behaviour change interventions

BACKGROUND: Behaviour change interventions influence behaviour through causal processes called “mechanisms of action” (MoAs). Reports of such interventions and their evaluations often use inconsistent or ambiguous terminology, creating problems for searching, evidence synthesis and theory development. This inconsistency includes the reporting of MoAs. An ontology can help address these challenges by serving as a classification system that labels and defines MoAs and their relationships. The aim of this study was to develop an ontology of MoAs of behaviour change interventions. METHODS: To develop the MoA Ontology, we (1) defined the ontology’s scope; (2) identified, labelled and defined the ontology’s entities; (3) refined the ontology by annotating (i.e., coding) MoAs in intervention reports; (4) refined the ontology via stakeholder review of the ontology’s comprehensiveness and clarity; (5) tested whether researchers could reliably apply the ontology to annotate MoAs in intervention evaluation reports; (6) refined the relationships between entities; (7) reviewed the alignment of the MoA Ontology with other relevant ontologies, (8) reviewed the ontology’s alignment with the Theories and Techniques Tool; and (9) published a machine-readable version of the ontology. RESULTS: An MoA was defined as “a process that is causally active in the relationship between a behaviour change intervention scenario and its outcome behaviour”. We created an initial MoA Ontology with 261 entities through Steps 2-5. Inter-rater reliability for annotating study reports using these entities was α=0.68 (“acceptable”) for researchers familiar with the ontology and α=0.47 for researchers unfamiliar with it. As a result of additional revisions (Steps 6-8), 21 further entities were added to the ontology resulting in 282 entities organised in seven hierarchical levels. CONCLUSIONS: The MoA Ontology extensively captures MoAs of behaviour change interventions. The ontology can serve as a controlled vocabulary for MoAs to consistently describe and synthesise evidence about MoAs across diverse sources

Ontologies relevant to behaviour change interventions: a method for their development.

Background: Behaviour and behaviour change are integral to many aspects of wellbeing and sustainability. However, reporting behaviour change interventions accurately and synthesising evidence about effective interventions is hindered by lacking a shared, scientific terminology to describe intervention characteristics. Ontologies are standardised frameworks that provide controlled vocabularies to help unify and connect scientific fields. To date, there is no published guidance on the specific methods required to develop ontologies relevant to behaviour change. We report the creation and refinement of a method for developing ontologies that make up the Behaviour Change Intervention Ontology (BCIO). Aims: (1) To describe the development method of the BCIO and explain its rationale; (2) To provide guidance on implementing the activities within the development method. Method and results: The method for developing ontologies relevant to behaviour change interventions was constructed by considering principles of good practice in ontology development and identifying key activities required to follow those principles. The method's details were refined through application to developing two ontologies. The resulting ontology development method involved: (1) defining the ontology's scope; (2) identifying key entities; (3) refining the ontology through an iterative process of literature annotation, discussion and revision; (4) expert stakeholder review; (5) testing inter-rater reliability; (6) specifying relationships between entities, and; (7) disseminating and maintaining the ontology. Guidance is provided for conducting relevant activities for each step.  Conclusions: We have developed a detailed method for creating ontologies relevant to behaviour change interventions, together with practical guidance for each step, reflecting principles of good practice in ontology development. The most novel aspects of the method are the use of formal mechanisms for literature annotation and expert stakeholder review to develop and improve the ontology content. We suggest the mnemonic SELAR3, representing the method's first six steps as Scope, Entities, Literature Annotation, Review, Reliability, Relationships

Using machine learning to extract information and predict outcomes from reports of randomised trials of smoking cessation interventions in the Human Behaviour-Change Project

Peer reviewe

Development of an Intervention Setting Ontology for behaviour change: Specifying where interventions take place.

Background: Contextual factors such as an intervention's setting are key to understanding how interventions to change behaviour have their effects and patterns of generalisation across contexts. The intervention's setting is not consistently reported in published reports of evaluations. Using ontologies to specify and classify intervention setting characteristics enables clear and reproducible reporting, thus aiding replication, implementation and evidence synthesis. This paper reports the development of a Setting Ontology for behaviour change interventions as part of a Behaviour Change Intervention Ontology, currently being developed in the Wellcome Trust funded Human Behaviour-Change Project. Methods: The Intervention Setting Ontology was developed following methods for ontology development used in the Human Behaviour-Change Project: 1) Defining the ontology's scope, 2) Identifying key entities by reviewing existing classification systems (top-down) and 100 published behaviour change intervention reports (bottom-up), 3) Refining the preliminary ontology by literature annotation of 100 reports, 4) Stakeholder reviewing by 23 behavioural science and public health experts to refine the ontology, 5) Assessing inter-rater reliability of using the ontology by two annotators familiar with the ontology and two annotators unfamiliar with it, 6) Specifying ontological relationships between setting entities and 7) Making the Intervention Setting Ontology machine-readable using Web Ontology Language (OWL) and publishing online. Re sults: The Intervention Setting Ontology consists of 72 entities structured hierarchically with two upper-level classes: Physical setting including Geographic location, Attribute of location (including Area social and economic condition, Population and resource density sub-levels) and Intervention site (including Facility, Transportation and Outdoor environment sub-levels), as well as Social setting. Inter-rater reliability was found to be 0.73 (good) for those familiar with the ontology and 0.61 (acceptable) for those unfamiliar with it. Conclusion: The Intervention Setting Ontology can be used to code information from diverse sources, annotate the setting characteristics of existing intervention evaluation reports and guide future reporting