Risk of bias in routine mental health outcome data: the case of Health of the Nation Outcome Scales

Abstract

Background Routine outcome data in secondary mental health services have significant potential for service planning, evaluation and research. Expanding the collection and use of these data is an ongoing priority in the National Health Service (NHS), but inconsistent use threatens their validity and utility. If recording is more likely among certain patient groups or at specific stages of treatment, measured outcomes may be biased and unreliable. Objective The objective is to assess the scale, determinants and implications of incomplete routine outcome measurement in a secondary mental health provider, using the example of the widely collected Health of the Nation Outcome Scores (HoNOS). Methods A retrospective cohort study was conducted using routine HoNOS assessments and episodes of care for patients receiving secondary mental healthcare from an NHS Trust in Southeast England between 2016 and 2022 (n=30 341). Associations among demographic, clinical and service factors, and rates and timings of HoNOS assessments were explored with logistic regressions. Relationships between total HoNOS scores and related mental health outcomes (costs, relapse and improvement between assessments) were estimated after adjusting for the likelihood of assessment. Findings 66% of patients (n=22 288) had a recorded HoNOS assessment. Of the distinct episodes of care for these patients (n=65 439), 43% (n=28 170) were linked to any assessment, 25% (n=16 131) were linked to an initial baseline assessment, while 4.7% (n=3 094) were linked to multiple HoNOS assessments, allowing for evaluation of clinical progress. Likelihood and timing of assessment were significantly associated with a range of factors, including service type, diagnosis, ethnicity, age and gender. After adjusting for observed factors determining the likelihood of assessment, the strength of association between HoNOS scores and overall costs was significantly reduced. Conclusion Most of the activity observed in this study cannot be evaluated with HoNOS. HoNOS assessments are highly unlikely to be missing at random. Without approaches to correct for substantial gaps in routine outcome data, evaluations based on these may be systematically biased, limiting their usefulness for service-level decision-making. Clinical implications Routine outcome collection must increase significantly to successfully implement proposed strategies for outcome assessment in community mental healthcare without inconsistent records undermining the use of resulting data.