Structural equation model testing and the quality of natural killer cell activity measurements

BACKGROUND: Browne et al. [Browne, MacCallum, Kim, Andersen, Glaser: When fit indices and residuals are incompatible. Psychol Methods 2002] employed a structural equation model of measurements of target cell lysing by natural killer cells as an example purportedly demonstrating that small but statistically significant ill model fit can be dismissed as "negligible from a practical point of view". METHODS: Reanalysis of the natural killer cell data reveals that the supposedly negligible ill fit obscured important, systematic, and substantial causal misspecifications. RESULTS: A clean-fitting structural equation model indicates that measurements employing higher natural-killer-cell to target-cell ratios are more strongly influenced by a progressively intrusive factor, whether or not the natural killer cell activity is activated by recombinant interferon γ (rIFN γ). The progressive influence may reflect independent rate limiting steps in cell recognition and attachment, spatial competition for cell attachment points, or the simultaneous lysings of single target cells by multiple natural killer cells. CONCLUSIONS: If the progressively influential factor is ultimately identified as a mere procedural impediment, the substantive conclusion will be that measurements of natural killer cell activity made at lower effector to target ratios are more valid. Alternatively, if the individual variations in the progressively influential factor are modifiable, this may presage a new therapeutic route to enhancing natural killer cell activity. The methodological conclusion is that, when using structural equation models, researchers should attend to significant model ill fit even if the degree of covariance ill fit is small, because small covariance residuals do not imply that the underlying model misspecifications are correspondingly small or inconsequential

Reliability and validity of the Alberta context tool (ACT) with professional nurses: Findings from a multi-study analysis

Although organizational context is central to evidence-based practice, underdeveloped measurement hindersitsassessment. The Alberta Context Tool, comprised of 59 items that tap10 modifiable contextual concepts, was developed to address this gap. The purpose of this study to examine the reliability and validity of scores obtained when the Alberta Context Tool is completed by professional nurses across different healthcare settings. Five separate studies (N = 2361 nurses across different care settings) comprised the study sample. Reliability and validity were assessed. Cronbach\u27s alpha exceeded 0.70 for9/10 Alberta Context Tool concepts. Item-total correlations exceeded acceptable standards for 56/59items. Confirmatory Factor Analysescoordinated acceptably with the Alberta Context Tool\u27s proposed latent structure. The mean values for each Alberta Context Tool concept increased from low to high levels of research utilization(as hypothesized) further supporting its validity. This study provides robust evidence forreliability and validity of scores obtained with the Alberta Context Tool when administered to professional nurses

A protocol for advanced psychometric assessment of surveys

Background and Purpose. In this paper, we present a protocol for advanced psychometric assessments of surveys based on the Standards for Educational and Psychological Testing. We use the Alberta Context Tool (ACT) as an exemplar survey to which this protocol can be applied. Methods. Data mapping, acceptability, reliability, and validity are addressed. Acceptability is assessed with missing data frequencies and the time required to complete the survey. Reliability is assessed with internal consistency coefficients and information functions. A unitary approach to validity consisting of accumulating evidence based on instrument content, response processes, internal structure, and relations to other variables is taken. We also address assessing performance of survey data when aggregated to higher levels (e.g., nursing unit). Discussion. In this paper we present a protocol for advanced psychometric assessment of survey data using the Alberta Context Tool (ACT) as an exemplar survey; application of the protocol to the ACT survey is underway. Psychometric assessment of any survey is essential to obtaining reliable and valid research findings. This protocol can be adapted for use with any nursing survey.<br /

The changing causal foundations of cancer-related symptom clustering during the final month of palliative care: A longitudinal study

<p>Abstract</p> <p>Background</p> <p>Symptoms tend to occur in what have been called symptom clusters. Early symptom cluster research was imprecise regarding the causal foundations of the coordinations between specific symptoms, and was silent on whether the relationships between symptoms remained stable over time. This study develops a causal model of the relationships between symptoms in cancer palliative care patients as they approach death, and investigates the changing associations among the symptoms and between those symptoms and well-being.</p> <p>Methods</p> <p>Complete symptom assessment scores were obtained for 82 individuals from an existing palliative care database. The data included assessments of pain, anxiety, nausea, shortness of breath, drowsiness, loss of appetite, tiredness, depression and well-being, all collected using the Edmonton Symptom Assessment System (ESAS). Relationships between the symptoms and well-being were investigated using a structural equation model.</p> <p>Results</p> <p>The model fit acceptably and explained between 26% and 83% of the variation in appetite, tiredness, depression, and well-being. Drowsiness displayed consistent effects on appetite, tiredness and well-being. In contrast, anxiety's effect on well-being shifted importantly, with a direct effect and an indirect effect through tiredness at one month, being replaced by an effect working exclusively through depression at one week.</p> <p>Conclusion</p> <p>Some of the causal forces explaining the variations in, and relationships among, palliative care patients' symptoms changed over the final month of life. This illustrates how investigating the causal foundations of symptom correlation or clustering can provide more detailed understandings that may contribute to improved control of patient comfort, quality of life, and quality of death.</p

Advancing the argument for validity of the Alberta Context Tool with healthcare aides in residential long-term care

<p>Abstract</p> <p>Background</p> <p>Organizational context has the potential to influence the use of new knowledge. However, despite advances in understanding the theoretical base of organizational context, its measurement has not been adequately addressed, limiting our ability to quantify and assess context in healthcare settings and thus, advance development of contextual interventions to improve patient care. We developed the Alberta Context Tool (the ACT) to address this concern. It consists of 58 items representing 10 modifiable contextual concepts. We reported the initial validation of the ACT in 2009. This paper presents the second stage of the psychometric validation of the ACT.</p> <p>Methods</p> <p>We used the <it>Standards for Educational and Psychological Testing </it>to frame our validity assessment. Data from 645 English speaking healthcare aides from 25 urban residential long-term care facilities (nursing homes) in the three Canadian Prairie Provinces were used for this stage of validation. In this stage we focused on: (1) advanced aspects of internal structure (e.g., confirmatory factor analysis) and (2) relations with other variables validity evidence. To assess reliability and validity of scores obtained using the ACT we conducted: Cronbach's alpha, confirmatory factor analysis, analysis of variance, and tests of association. We also assessed the performance of the ACT when individual responses were aggregated to the care unit level, because the instrument was developed to obtain unit-level scores of context.</p> <p>Results</p> <p>Item-total correlations exceeded acceptable standards (> 0.3) for the majority of items (51 of 58). We ran three confirmatory factor models. Model 1 (all ACT items) displayed unacceptable fit overall and for five specific items (1 item on <it>adequate space for resident care </it>in the Organizational Slack-Space ACT concept and 4 items on use of electronic resources in the Structural and Electronic Resources ACT concept). This prompted specification of two additional models. Model 2 used the 7 scaled ACT concepts while Model 3 used the 3 count-based ACT concepts. Both models displayed substantially improved fit in comparison to Model 1. Cronbach's alpha for the 10 ACT concepts ranged from 0.37 to 0.92 with 2 concepts performing below the commonly accepted standard of 0.70. Bivariate associations between the ACT concepts and instrumental research utilization levels (which the ACT should predict) were statistically significant at the 5% level for 8 of the 10 ACT concepts. The majority (8/10) of the ACT concepts also showed a statistically significant trend of increasing mean scores when arrayed across the lowest to the highest levels of instrumental research use.</p> <p>Conclusions</p> <p>The validation process in this study demonstrated additional empirical support for construct validity of the ACT, when completed by healthcare aides in nursing homes. The overall pattern of the data was consistent with the structure hypothesized in the development of the ACT and supports the ACT as an appropriate measure for assessing organizational context in nursing homes. Caution should be applied in using the one space and four electronic resource items that displayed misfit in this study with healthcare aides until further assessments are made.</p

Improving measurement-invariance assessments: correcting entrenched testing deficiencies

Abstract Background Factor analysis historically focused on measurement while path analysis employed observed variables as though they were error-free. When factor- and path-analysis merged as structural equation modeling, factor analytic notions dominated measurement discussions – including assessments of measurement invariance across groups. The factor analytic tradition fostered disregard of model testing and consequently entrenched this deficiency in measurement invariance assessments. Discussion Applying contemporary model testing requirements to the so-called configural model initiating invariance assessments will improve future assessments but a substantial backlog of deficient assessments remain to be overcome. This article summarizes the issues, demonstrates the problem using a recent example, illustrates a superior model assessment strategy, and documents disciplinary entrenchment of inadequate testing as exemplified by the journal Organizational Research Methods. Summary Employing the few methodologically and theoretically best, rather than precariously-multiple, indicators of latent variables increases the likelihood of achieving properly causally specified structural equation models capable of displaying measurement invariance. Just as evidence of invalidity trumps reliability, evidence of configural model misspecification trumps invariant estimates of misspecified coefficients

Should researchers use single indicators, best indicators, or multiple indicators in structural equation models?

<p>Abstract</p> <p>Background</p> <p>Structural equation modeling developed as a statistical melding of path analysis and factor analysis that obscured a fundamental tension between a factor preference for multiple indicators and path modeling’s openness to fewer indicators.</p> <p>Discussion</p> <p>Multiple indicators hamper theory by unnecessarily restricting the number of modeled latents. Using the few best indicators – possibly even the single best indicator of each latent – encourages development of theoretically sophisticated models. Additional latent variables permit stronger statistical control of potential confounders, and encourage detailed investigation of mediating causal mechanisms.</p> <p>Summary</p> <p>We recommend the use of the few best indicators. One or two indicators are often sufficient, but three indicators may occasionally be helpful. More than three indicators are rarely warranted because additional redundant indicators provide less research benefit than single indicators of additional latent variables. Scales created from multiple indicators can introduce additional problems, and are prone to being less desirable than either single or multiple indicators.</p