Looking both ways: A review of methods for assessing research impacts on policy and the policy utilisation of research

Background: Measuring the policy and practice impacts of research is becoming increasingly important. Policy impacts can be measured from two directions – tracing forward from research and tracing backwards from a policy outcome. In this review, we compare these approaches and document the characteristics of studies assessing research impacts on policy and the policy utilisation of research. Methods: Keyword searches of electronic databases were conducted in December 2016. Included studies were published between 1995 and 2016 in English and reported methods and findings of studies measuring policy impacts of specified health research, or research use in relation to a specified health policy outcome, and reviews reporting methods of research impact assessment. Using an iterative data extraction process, we developed a framework to define the key elements of empirical studies (assessment reason, assessment direction, assessment starting point, unit of analysis, assessment methods, assessment endpoint and outcomes assessed) and then documented the characteristics of included empirical studies according to this framework. Results: We identified 144 empirical studies and 19 literature reviews. Empirical studies were derived from two parallel streams of research of equal size, which we termed ‘research impact assessments’ and ‘research use assessments’. Both streams provided insights about the influence of research on policy and utilised similar assessment methods, but approached measurement from opposite directions. Research impact assessments predominantly utilized forward tracing approaches while the converse was true for research use assessments. Within each stream, assessments focussed on narrow or broader research/policy units of analysis as the starting point for assessment, each with associated strengths and limitations. The two streams differed in terms of their relative focus on the contributions made by specific research (research impact assessments) versus research more generally (research use assessments) and the emphasis placed on research and the activities of researchers in comparison to other factors and actors as influencers of change. Conclusions: The Framework presented in this paper provides a mechanism for comparing studies within this broad field of research enquiry. Forward and backward tracing approaches, and their different ways of ‘looking’, tell a different story of research-based policy change. Combining approaches may provide the best way forward in terms of linking outcomes to specific research, as well as providing a realistic picture of research influence

Parental influences on child physical activity and screen viewing time: a population based study

Background: Parents can influence their children’s physical activity participation and screen time.This study examined the relative significance of perceived parental barriers and self-efficacy in relation to children’s physical activity participation and screen time viewing. The associations between these factors and the behaviours were analysed. Methods: Cross-sectional population survey in New South Wales, Australia of parents of pre-school (N = 764), younger (Kindergarten, Grades 2 and 4; N = 1557) and older children (Grades 6, 8 and 10; N = 1665). Parents reported barriers and self-efficacy to influence their child’s physical activity and screen time behaviours in a range of circumstances. Differences were examined by child’s sex and age group, household income, maternal education and location of residence. The duration of physical activity and screen viewing was measured by parental report for pre-school and younger children and self-report for older children. Associations between parental factors and children’s organised, non-organised and total activity and screen time were analysed. Results: Cost, lack of opportunities for participation and transport problems were the barriers most often reported, particularly by low income parents and those in rural areas. The number of barriers was inversely related to children’s time spent in organised activity, but not their non-organised activity. Higher parental self-efficacy was positively associated with organised physical activity in the younger and older children’s groups and the non-organised activity of older children. School-age children (younger and older groups) were less likely to meet physical activity guidelines when parents reported ≥4 barriers (OR 3.76, 95% CI 1.25-11.34 and OR 3.72, 95% CI 1.71-8.11 respectively). Low parental self-efficacy was also associated with the likelihood of children exceeding screen time guidelines for each age group (pre-school OR 0.62, 95% CI 0.43-0.87; young children OR 0.56, 95% CI 0.39-0.80; and older children OR 0.57, 95% CI 0.43-0.74). Conclusion: Parental barriers are associated with the time that children spend in both active and sedentary pursuits. These findings highlight family, economic and environmental factors that should be addressed in programs to promote child physical activity and tackle sedentary behaviour

Harnessing the power of advertising to prevent childhood obesity

Background: Social marketing integrates communication campaigns with behavioural and environmental change strategies. Childhood obesity programs could benefit significantly from social marketing but communication campaigns on this issue tend to be stand-alone. Methods: A large-scale multi-setting child obesity prevention program was implemented in the Hunter New England (HNE) region of New South Wales (NSW), Australia from 2005-2010. The program included a series of communication campaigns promoting the program and its key messages: drinking water; getting physically active and; eating more vegetables and fruit. Pre-post telephone surveys (n = 9) were undertaken to evaluate awareness of the campaigns among parents of children aged 2-15 years using repeat cross-sections of randomly selected cohorts. A total of 1,367 parents (HNE = 748, NSW = 619) participated. Results: At each survey post baseline, HNE parents were significantly more likely to have seen, read or heard about the program and its messages in the media than parents in the remainder of the state (p < 0.001). Further, there was a significant increase in awareness of the program and each of its messages over time in HNE compared to no change over time in NSW (p < 0.001). Awareness was significantly higher (p < 0.05) in HNE compared to NSW after each specific campaign (except the vegetable one) and significantly higher awareness levels were sustained for each campaign until the end of the program. At the end of the program participants without a tertiary education were significantly more likely (p = 0.04) to be aware of the brand campaign (31%) than those with (20%) but there were no other statistically significant socio-demographic differences in awareness. Conclusions: The Good for Kids communication campaigns increased and maintained awareness of childhood obesity prevention messages. Moreover, messages were delivered equitably to diverse socio-demographic groups within the region

A mixed methods study of the factors that influence whether intervention research has policy and practice impacts: perceptions of Australian researchers

Objectives: To investigate researchers’ perceptions about the factors that influenced the policy and practice impacts (or lack of impact) of one of their own funded intervention research studies. Design: Mixed method, cross-sectional study. Setting: Intervention research conducted in Australia and funded by Australia’s National Health and Medical Research Council between 2003 and 2007. Participants: The chief investigators from 50 funded intervention research studies were interviewed to determine if their study had achieved policy and practice impacts, how and why these impacts had (or had not) occurred and the approach to dissemination they had employed. Results: We found that statistically significant intervention effects and publication of results influenced whether there were policy and practice impacts, along with factors related to the nature of the intervention itself, the researchers’ experience and connections, their dissemination and translation efforts, and the postresearch context. Conclusions: This study indicates that sophisticated approaches to intervention development, dissemination actions and translational efforts are actually widespread among experienced researches, and can achieve policy and practice impacts. However, it was the links between the intervention results, further dissemination actions by researchers and a variety of postresearch contextual factors that ultimately determined whether a study had policy and practice impacts. Given the complicated interplay between the various factors, there appears to be no simple formula for determining which intervention studies should be funded in order to achieve optimal policy and practice impacts

Tracking funded health intervention research

Objective: To describe the research publication outputs from intervention research funded by Australia’s National Health and Medical Research Council (NHMRC). Design and setting: Analysis of descriptive data and data on publication outputs collected between 23 July 2012 and 10 December 2013 relating to health intervention research project grants funded between 1 January 2003 and 31 December 2007. Main outcome measures: Stages of development of intervention studies (efficacy, effectiveness, replication, adaptation or dissemination of intervention); types of interventions studied; publication output per NHMRC grant; and whether interventions produced statistically significant changes in primary outcome variables. Results: Most of the identified studies tested intervention efficacy or effectiveness in clinical or community settings, with few testing the later stages of intervention development, such as replication, adaptation or dissemination. Studies focused largely on chronic disease treatment and management, and encompassed various medical and allied health disciplines. Equal numbers of studies had interventions that produced statistically significant results on primary outcomes, (27) and those that did not (27). The mean number of total published articles per grant was 3.3, with 2.0 articles per grant focusing on results, and the remainder covering descriptive, exploratory or methodological aspects of intervention research. Conclusions: Our study provides a benchmark for the publication outputs of NHMRC-funded health intervention research in Australia. Research productivity is particularly important for intervention research, where findings are likely to have more immediate and direct applicability to health policy and practice. Tracking research outputs in this way provides information on whether current research investment patterns match the need for evidence about health care interventions

Public health research outputs from efficacy to dissemination: a bibliometric analysis

<p>Abstract</p> <p>Background</p> <p>More intervention research is needed, particularly 'real world' intervention replication and dissemination studies, to optimize improvements in health. This study assessed the proportion and type of published public health intervention research papers over time in physical activity and falls prevention, both important contributors to preventable morbidity and mortality.</p> <p>Methods</p> <p>A keyword search was conducted, using Medline and PsycINFO to locate publications in 1988-1989, 1998-1999, and 2008-2009 for the two topic areas. In stage 1, a random sample of 1200 publications per time period for both topics were categorized as: non-public health, non-data-based public health, or data-based public health. In stage 2 data-based public health articles were further classified as measurement, descriptive, etiological or intervention research. Finally, intervention papers were categorized as: efficacy, intervention replication or dissemination studies. Inter-rater reliability of paper classification was 88%.</p> <p>Results</p> <p>Descriptive studies were the most common data-based papers across all time periods (1988-89; 1998-1999;2008-2009) for both issues (physical activity: 47%; 54%; 65% and falls 75%; 64%; 63%), increasing significantly over time for physical activity. The proportion of intervention publications did not increase over time for physical activity comprising 23% across all time periods and fluctuated for falls across the time periods (10%; 21%; 17%). The proportion of intervention articles that were replication studies increased over the three time periods for physical activity (0%; 2%; 11%) and for falls (0%; 22%; 35%). Dissemination studies first appeared in the literature in 2008-2009, making up only 3% of physical activity and 7% of falls intervention studies.</p> <p>Conclusions</p> <p>Intervention research studies remain only a modest proportion of all published studies in physical activity and falls prevention; the majority of the intervention studies, are efficacy studies although there is growing evidence of a move towards replication and dissemination studies, which may have greater potential for improving population health.</p

Policy and practice impacts of applied research: a case study analysis of the New South Wales Health Promotion Demonstration Research Grants Scheme 2000-2006

BackgroundIntervention research provides important information regarding feasible and effective interventions for health policy makers, but few empirical studies have explored the mechanisms by which these studies influence policy and practice. This study provides an exploratory case series analysis of the policy, practice and other related impacts of the 15 research projects funded through the New South Wales Health Promotion Demonstration Research Grants Scheme during the period 2000 to 2006, and explored the factors mediating impacts.MethodsData collection included semi-structured interviews with the chief investigators (n = 17) and end-users (n = 29) of each of the 15 projects to explore if, how and under what circumstances the findings had been used, as well as bibliometric analysis and verification using documentary evidence. Data analysis involved thematic coding of interview data and triangulation with other data sources to produce case summaries of impacts for each project. Case summaries were then individually assessed against four impact criteria and discussed at a verification panel meeting where final group assessments of the impact of research projects were made and key influences of research impact identified.ResultsFunded projects had variable impacts on policy and practice. Project findings were used for agenda setting (raising awareness of issues), identifying areas and target groups for interventions, informing new policies, and supporting and justifying existing policies and programs across sectors. Reported factors influencing the use of findings were: i) nature of the intervention; ii) leadership and champions; iii) research quality; iv) effective partnerships; v) dissemination strategies used; and, vi) contextual factors.ConclusionsThe case series analysis provides new insights into how and under what circumstances intervention research is used to influence real world policy and practice. The findings highlight that intervention research projects can achieve the greatest policy and practice impacts if they address proximal needs of the policy context by engaging end-users from the inception of projects and utilizing existing policy networks and structures, and using a range of strategies to disseminate findings that go beond traditional peer review publications.<br /