Evaluation of the Middle Years Reform Program

The Middle Years Reform Program (MYRP) was conducted in all Victorian government secondary and P-12 schools over the period 2001-2003. The program was designed to provide these schools with financial support to employ additional classroom teaching capacity to develop and implement initiatives in the areas of literacy, attendance and the ‘thinking curriculum’ in Years 7-9. Data for the evaluation were taken from: a. Literature and document review; b. Three preliminary consultations with representative groups of regional office personnel, school principals, middle years co-ordinators and other teachers familiar with middle years issues; c. A questionnaire that was distributed by e-mail to all schools with students in years 7-9 that achieved a response rate of just over 80%; d. Analysis of school-level aggregate data for the period 1998-2003 on Year 9 literacy, Years 7, 8 and 9 attendance, and retention to Year 11; e. Six brief case studies of purposefully selected schools with Year 7-9 students. The questionnaire data formed a key component of the evaluation of MYRP

Percentile ranks and benchmark estimates of change for the Health Education Impact Questionnaire: normative data from an Australian sample

Objective: Participant self-report data play an essential role in the evaluation of health education activities, programmes and policies. When questionnaire items do not have a clear mapping to a performance-based continuum, percentile norms are useful for communicating individual test results to users. Similarly, when assessing programme impact, the comparison of effect sizes for group differences or baseline to follow-up change with effect sizes observed in relevant normative data provides more directly useful information compared with statistical tests of mean differences and the evaluation of effect sizes for substantive significance using universal rule-of-thumb such as those for Cohen’s ‘d’. This article aims to assist managers, programme staff and clinicians of healthcare organisations who use the Health Education Impact Questionnaire interpret their results using percentile norms for individual baseline and follow-up scores together with group effect sizes for change across the duration of typical chronic disease self-management and support programme. Methods: Percentile norms for individual Health Education Impact Questionnaire scale scores and effect sizes for group change were calculated using freely available software for each of the eight Health Education Impact Questionnaire scales. Data used were archived responses of 2157 participants of chronic disease self-management programmes conducted by a wide range of organisations in Australia between July 2007 and March 2013. Results: Tables of percentile norms and three possible effect size benchmarks for baseline to follow-up change are provided together with two worked examples to assist interpretation. Conclusion: While the norms and benchmarks presented will be particularly relevant for Australian organisations and others using the English-language version of the Health Education Impact Questionnaire, they will also be useful for translated versions as a guide to the sensitivity of the scales and the extent of the changes that might be anticipated from attendance at a typical chronic disease self-management or health education programme

The grounded psychometric development and initial validation of the Health Literacy Questionnaire (HLQ)

BACKGROUND: Health literacy has become an increasingly important concept in public health. We sought to develop a comprehensive measure of health literacy capable of diagnosing health literacy needs across individuals and organisations by utilizing perspectives from the general population, patients, practitioners and policymakers. METHODS: Using a validity-driven approach we undertook grounded consultations (workshops and interviews) to identify broad conceptually distinct domains. Questionnaire items were developed directly from the consultation data following a strict process aiming to capture the full range of experiences of people currently engaged in healthcare through to people in the general population. Psychometric analyses included confirmatory factor analysis (CFA) and item response theory. Cognitive interviews were used to ensure questions were understood as intended. Items were initially tested in a calibration sample from community health, home care and hospital settings (N=634) and then in a replication sample (N=405) comprising recent emergency department attendees. RESULTS: Initially 91 items were generated across 6 scales with agree/disagree response options and 5 scales with difficulty in undertaking tasks response options. Cognitive testing revealed that most items were well understood and only some minor re-wording was required. Psychometric testing of the calibration sample identified 34 poorly performing or conceptually redundant items and they were removed resulting in 10 scales. These were then tested in a replication sample and refined to yield 9 final scales comprising 44 items. A 9-factor CFA model was fitted to these items with no cross-loadings or correlated residuals allowed. Given the very restricted nature of the model, the fit was quite satisfactory: χ(2)(WLSMV)(866 d.f.) = 2927, p<0.000, CFI = 0.936, TLI = 0.930, RMSEA = 0.076, and WRMR = 1.698. Final scales included: Feeling understood and supported by healthcare providers; Having sufficient information to manage my health; Actively managing my health; Social support for health; Appraisal of health information; Ability to actively engage with healthcare providers; Navigating the healthcare system; Ability to find good health information; and Understand health information well enough to know what to do. CONCLUSIONS: The HLQ covers 9 conceptually distinct areas of health literacy to assess the needs and challenges of a wide range of people and organisations. Given the validity-driven approach, the HLQ is likely to be useful in surveys, intervention evaluation, and studies of the needs and capabilities of individuals

Measuring health literacy in community agencies: a Bayesian study of the factor structure and measurement invariance of the Health Literacy Questionnaire (HLQ)

Background The development of the Health Literacy Questionnaire (HLQ), reported in 2013, attracted widespread international interest. While the original study samples were drawn from clinical and home-based aged-care settings, the HLQ was designed for the full range of healthcare contexts including community-based health promotion and support services. We report a follow-up study of the psychometric properties of the HLQ with respondents from a diverse range of community-based organisations with the principal goal of contributing to the development of a soundly validated evidence base for its use in community health settings.Methods Data were provided by 813 clients of 8 community agencies in Victoria, Australia who were administered the HLQ during the needs assessment stage of the Ophelia project, a health literacy-based intervention. Most analyses were conducted using Bayesian structural equation modelling that enables rigorous analysis of data but with some relaxation of the restrictive requirements for zero cross-loadings and residual correlations of &lsquo;classical&rsquo; confirmatory factor analysis. Scale homogeneity was investigated with one-factor models that allowed for the presence of small item residual correlations while discriminant validity was studied using the inter-factor correlations and factor loadings from a full 9-factor model with similar allowance for small residual correlations and cross-loadings. Measurement invariance was investigated scale-by-scale using a model that required strict invariance of item factor loadings, thresholds, residual variances and co-variances.Results All HLQ scales were found to be homogenous with composite reliability ranging from 0.80 to 0.89. The factor structure of the HLQ was replicated and 6 of the 9 scales were found to exhibit clear-cut discriminant validity. With a small number of exceptions involving non-invariance of factor loadings, strict measurement invariance was established across the participating organisations and the gender, language background, age and educational level of respondents.Conclusions The HLQ is highly reliable, even with only 4 to 6 items per scale. It provides unbiased mean estimates of group differences across key demographic indicators. While measuring relatively narrow constructs, the 9 dimensions are clearly separate and therefore provide fine-grained data on the multidimensional area of health literacy. These analyses provide researchers, program managers and policymakers with a range of robust evidence by which they can make judgements about the appropriate use of the HLQ for their community-based setting

Cultural adaptation and validation of the Health Literacy Questionnaire (HLQ): robust nine-dimension Danish language confirmatory factor model

Health literacy is an important construct in population health and healthcare requiring rigorous measurement. The Health Literacy Questionnaire (HLQ), with nine scales, measures a broad perception of health literacy. This study aimed to adapt the HLQ to the Danish setting, and to examine the factor structure, homogeneity, reliability and discriminant validity. The HLQ was adapted using forward-backward translation, consensus conference and cognitive interviews (n&nbsp;=&nbsp;15). Psychometric properties were examined based on data collected by face-to-face interview (n&nbsp;=&nbsp;481). Tests included difficulty level, composite scale reliability and confirmatory factor analysis (CFA). Cognitive testing revealed that only minor re-wording was required. The easiest scale to respond to positively was \u27Social support for health\u27, and the hardest were \u27Navigating the healthcare system\u27 and \u27Appraisal of health information\u27. CFA of the individual scales showed acceptably high loadings (range 0.49-0.93). CFA fit statistics after including correlated residuals were good for seven scales, acceptable for one. Composite reliability and Cronbach\u27s &alpha; were &gt;0.8 for all but one scale. A nine-factor CFA model was fitted to items with no cross-loadings or correlated residuals allowed. Given this restricted model, the fit was satisfactory. The HLQ appears robust for its intended application of assessing health literacy in a range of settings. Further work is required to demonstrate sensitivity to measure changes

Absence of social desirability bias in the evaluation of chronic disease self-management interventions

BACKGROUND: Bias due to social desirability has long been of concern to evaluators relying on self-report data. It is conceivable that health program evaluation is particularly susceptible to social desirability bias as individuals may be inclined to present themselves or certain health behaviors in a more positive light and/or appease the course leader. Thus, the influence of social desirability bias on self-report outcomes was explored in the present study. METHODS: Data were collected from 331 participants of group-based chronic disease self-management interventions using the highly robust eight-scale Health Education Impact Questionnaire (heiQ) and the 13-item short form Marlowe-Crowne Social Desirability Scale (MC-C). The majority of self-management courses were run by community-based organizations across Australia between February 2005 and December 2006 where 6 to 12 individuals have the opportunity to develop considerable rapport with course leaders and each other over about six weeks. Pre-test data were collected on the first day of courses, while post-test and social desirability scores were assessed at the end of courses. A model of partial mediation within the framework of structural equation modeling was developed with social desirability as the mediating variable between pre-test and post-test. RESULTS: The \u27Defensiveness\u27 factor of the MC-C showed clear association with heiQ pre-test data, a prerequisite for investigating mediation; however, when investigating the eight full pre-test/post-test models \u27Defensiveness\u27 was only associated with one heiQ scale. This effect was small, explaining 8% of the variance in the model. No other meditational effects through social desirability were observed. CONCLUSIONS: The overall lack of association of social desirability with heiQ outcomes was surprising as it had been expected that it would explain at least some of the variance observed between pre-test and post-test. With the assumption that the MC-C captures the propensity for an individual to provide socially desirable answers, this study concludes that change scores in chronic disease self-management program evaluation are not biased by social desirability

Construction of the descriptive system for the assessment of quality of life AQoL-6D utility instrument

BackgroundMulti attribute utility (MAU) instruments are used to include the health related quality of life (HRQoL) in economic evaluations of health programs. Comparative studies suggest different MAU instruments measure related but different constructs. The objective of this paper is to describe the methods employed to achieve content validity in the descriptive system of the Assessment of Quality of Life (AQoL)-6D, MAU instrument.MethodsThe AQoL program introduced the use of psychometric methods in the construction of health related MAU instruments. To develop the AQoL-6D we selected 112 items from previous research, focus groups and expert judgment and administered them to 316 members of the public and 302 hospital patients. The search for content validity across a broad spectrum of health states required both formative and reflective modelling. We employed Exploratory Factor Analysis and Structural Equation Modelling (SEM) to meet these dual requirements.Results and DiscussionThe resulting instrument employs 20 items in a multi-tier descriptive system. Latent dimension variables achieve sensitive descriptions of 6 dimensions which, in turn, combine to form a single latent QoL variable. Diagnostic statistics from the SEM analysis are exceptionally good and confirm the hypothesised structure of the model.ConclusionsThe AQoL-6D descriptive system has good psychometric properties. They imply that the instrument has achieved construct validity and provides a sensitive description of HRQoL. This means that it may be used with confidence for measuring health related quality of life and that it is a suitable basis for modelling utilities for inclusion in the economic evaluation of health programs.<br /

Development of the Multidimensional Readiness and Enablement Index for Health Technology (READHY) Tool to Measure Individuals' Health Technology Readiness:Initial Testing in a Cancer Rehabilitation Setting

BACKGROUND: The increasing digitization of health care services with enhanced access to fast internet connections, along with wide use of smartphones, offers the opportunity to get health advice or treatment remotely. For service providers, it is important to consider how consumers can take full advantage of available services and how this can create an enabling environment. However, it is important to consider the digital context and the attributes of current and future users, such as their readiness (ie, knowledge, skills, and attitudes, including trust and motivation). OBJECTIVE: The objective of this study was to evaluate how the eHealth Literacy Questionnaire (eHLQ) combined with selected dimensions from the Health Education Impact Questionnaire (heiQ) and the Health Literacy Questionnaire (HLQ) can be used together as an instrument to characterize an individual\u27s level of health technology readiness and explore how the generated data can be used to create health technology readiness profiles of potential users of health technologies and digital health services. METHODS: We administered the instrument and sociodemographic questions to a population of 305 patients with a recent cancer diagnosis referred to rehabilitation in a setting that plans to introduce various technologies to assist the individuals. We evaluated properties of the Readiness and Enablement Index for Health Technology (READHY) instrument using confirmatory factor analysis, convergent and discriminant validity analysis, and exploratory factor analysis. To identify different health technology readiness profiles in the population, we further analyzed the data using hierarchical and k-means cluster analysis. RESULTS: The confirmatory factor analysis found a suitable fit for the 13 factors with only 1 cross-loading of 1 item between 2 dimensions. The convergent and discriminant validity analysis revealed many factor correlations, suggesting that, in this population, a more parsimonious model might be achieved. Exploratory factor analysis pointed to 5 to 6 constructs based on aggregates of the existing dimensions. The results were not satisfactory, so we performed an 8-factor confirmatory factor analysis, resulting in a good fit with only 1 item cross-loading between 2 dimensions. Cluster analysis showed that data from the READHY instrument can be clustered to create meaningful health technology readiness profiles of users. CONCLUSIONS: The 13 dimensions from heiQ, HLQ, and eHLQ can be used in combination to describe a user\u27s health technology readiness level and degree of enablement. Further studies in other populations are needed to understand whether the associations between dimensions are consistent and the number of dimensions can be reduced

The Health Literacy Questionnaire (HLQ) at the patient-clinician interface: a qualitative study of what patients and clinicians mean by their HLQ scores

BackgroundThe Health Literacy Questionnaire (HLQ) has nine scales that each measure an aspect of the multidimensional construct of health literacy. All scales have good psychometric properties. However, it is the interpretations of data within contexts that must be proven valid, not just the psychometric properties of a measurement instrument. The purpose of this study was to establish the extent of concordance and discordance between individual patient and clinician interpretations of HLQ data in the context of complex case management.MethodsSixteen patients with complex needs completed the HLQ and were interviewed to discuss the reasons for their answers. Also, the clinicians of each of these patients completed the HLQ about their patient, and were interviewed to discuss the reasons for their answers. Thematic analysis of HLQ scores and interview data determined the extent of concordance between patient and clinician HLQ responses, and the reasons for discordance.ResultsHighest concordance (80%) between patient and clinician item-response pairs was seen in Scale 1 and highest discordance (56%) was seen in Scale 6. Four themes were identified to explain discordance: 1) Technical or literal meaning of specific words; 2) Patients&rsquo; changing or evolving circumstances; 3) Different expectations and criteria for assigning HLQ scores; and 4) Different perspectives about a patient&rsquo;s reliance on healthcare providers.ConclusionThis study shows that the HLQ can act as an adjunct to clinical practice to help clinicians understand a patient&rsquo;s health literacy challenges and strengths early in a clinical encounter. Importantly, clinicians can use the HLQ to detect differences between their own perspectives about a patient&rsquo;s health literacy and the patient&rsquo;s perspective, and to initiate discussion to explore this. Provision of training to better detect these differences may assist clinicians to provide improved care.The outcomes of this study contribute to the growing body of international validation evidence about the use of the HLQ in different contexts. More specifically, this study has shown that the HLQ has measurement veracity at the patient and clinician level and may support clinicians to understand patients&rsquo; health literacy and enable a deeper engagement with healthcare services.<br /