Context: Measuring the impact of long-term Care (LTC) is essential to ensure effective allocation of limited resources.
Objectives: We explored the feasibility and validity of a pragmatic approach to evaluation, known as the counterfactual self-estimation of programme participants (CSEPP). CSEPP forms part of the Adult Social Care Outcomes Toolkit (ASCOT), and is referred to as the ‘expected’ method since participants estimate their expected quality of life in the absence of services.
Methods: We used survey data from interviews with 748 LTC users in 22 English local authorities. Questions on self- and interviewer-assessed understanding of the ‘expected’ questions were used to assess feasibility. Construct validity was assessed by examining hypothesised associations between the expected score and individual characteristics. Bias was assessed by comparing the expected impact estimate to one produced using Forder et al.’s (2014) instrumental variables approach on the same dataset.
Findings: We found evidence that the expected method was feasible and the self-estimated counterfactual outcome score valid. There were indications that the method is less appropriate for some groups and it may slightly overestimate the impact of LTC.
Limitations: Due to the opportunistic design, exploration of the method’s appropriateness for people with mental health problems was limited. The assumption of the between-methods comparison that the instrumental variables estimate of the effect of LTC is unbiased is unlikely to be true.
Implications: The expected method is a promising tool for the LTC context, but more research is needed to understand potential sources of bias and its feasibility with certain groups
