Behind the statistics: our new approach to analysing trends
Picker’s Chief Statistician, Steve Sizmur, has almost 30 years’ experience in social research. In this blog, he talks about the Inpatients Survey 2017, and Picker’s new approach to analysing trends with process-control charts.
Today’s publication of the 2017 results of the Adult Inpatient Survey marks its 15th iteration. There has been an annual inpatient survey since 2004, and consequently there is a rich source of historic data on the performance of NHS hospitals covering a period that has seen major changes in the NHS: organisational changes and responses to events such as the outbreaks of hospital-acquired infections in 2004-2006 and those leading to the public inquiry led by Sir Robert Francis.
Is the quality of hospital care improving? Is there evidence that lessons have been learned from previous problems? Is the NHS becoming increasingly patient centred? The inpatient survey data can provide answers if interrogated carefully.
Gathering increasing quantities of data has its problems when it comes to making sense of it. Up until this year, national patient survey reports have made comparisons between the most recent set of results and specific previous time-points (for 2016, the data were compared back to 2015, 2011 and 2006). The problem with this is twofold: the large numbers contributing to these results render very small differences ‘statistically significant’, and fluctuations over time can reverse these differences in the next wave of data: this year’s significant increase may turn out to be next year’s significant decrease. In some situations, small changes may accumulate over time but be missed by the tests applied.
This year’s publication makes use of a new approach to analysing trends. ‘P-charts’ or ‘process-control charts’ will be familiar to those versed in the ways of Six-Sigma, an established suite of quality-control methods that has seen use elsewhere in the NHS. The charts tackle the above two issues by taking a strongly-visual approach to interpreting the results and by differentiating ‘natural’ (ie expected) variation from changes that reflect probable differences in experience. The data go back to 2009; previous years are excluded because the weighting used to account for non-response cannot be applied to earlier surveys. The results are revealing.
Many questions in the survey show little change from earlier years. A few show steady, long-term improvement. Amongst these is the question on ward cleanliness (Q16 in the 2017 survey), those on food quality and choice (Q19 & Q20), on privacy (Q39 & Q40), and several questions on the quality of interactions between staff and patients (the ‘Doctors’ and ‘Nurses’ sections of the questionnaire and a number of questions) and on patients’ involvement in their care. Two key summary items, the Overall rating of care and whether patients felt they were treated with respect and dignity, have also shown steady improvement (Figure 1).
Figure 1: Improvement in overall rating of experience
In very few cases was there a sustained decline in reported experience. This was, however, evident for the length of delays when being discharged. There are also signs that some of the improvements may be levelling off. The proportion of people indicating that their ward was very clean has remained flat over the last three surveys (Figure 2), while there has been a downturn since 2015 in the proportion reporting that they were definitely involved in decisions about their care. Both of these results followed an earlier run of continuous improvement, and similar patterns are visible in ratings of privacy.
Figure 2: Is progress in cleanliness levelling off?
The data also reveal trends in patterns of care as well as patient demographics. There has, for example, been an increase in the proportion of respondents who were admitted in an emergency (in line with other data sources) and in those who spent time in a critical care unit, while the proportion of patients reporting that they experienced pain, or indicating that they had an operation, declined. The proportion of respondents in the older age bands has increased markedly since 2009.
So yes, the NHS does seem to have become more patient-focused, and there have been improvements in some key areas, including those related to the problems of the previous decade. There are also signs that progress may be slowing or even, in some cases, reversing. Elsewhere, we have found relationships between workforce factors, system capacity and patient experience ratings. The latest findings from the inpatient survey may signal that strain within the system is placing limits on what can be achieved. These results will need revisiting in future years for confirmation.