Steve Sizmur

Are patient-reported experience measures of psychometric evaluation valid?

iPad with charts

Picker’s clients and partners – quite naturally – want to know that they can measure service-user experience with ‘validated’ instruments. Picker takes validation seriously and gathers evidence to support the use of its questionnaires for evaluating patient care and for quality improvement.

As with our mission in general, we centre our validation work on service users themselves, drawing on their experience in focus groups and interviews to determine the scope and content of the questions, to test our interpretation of their answers to those questions and to ensure that people can respond in ways that properly convey their experience of care. We use statistical methods too, to detect where respondents may be finding it difficult to answer or where questions are not providing the level of information we need to evaluate care.

Psychometricians have, over many years, developed a suite of statistical methods to assess the validity and reliability of questionnaire data. Their origins lie in the fields of educational and psychological testing. These techniques cannot automatically be applied to the development of patient-reported experience measurement because the nature of what is being measured is quite different from psychological traits or abilities. However, this is not always recognised by researchers or by journal reviewers. In an article published in the International Journal for Quality in Health Care, Steve Sizmur and Chris Graham from Picker and Nanne Bos from the Netherlands Institute for Health Services Research, argue that statistical validation methods need to be matched to the measurement model appropriate for the instrument under development.

For the psychometric measurement model to be valid, there must be an underlying influence that affects how someone will respond to a set of questions, in the way that (for example) arithmetic ability underlies how you will respond to an arithmetic test. Your ability precedes you taking the test and it determines how many questions you will get right. In a healthcare setting, there are multiple different causes behind how someone experiences their care, including individual members of staff, the design of healthcare facilities and who provides the catering. Someone who has a positive experience of their hospital food is not necessarily more likely to be treated with respect by clinicians and may not have a quieter, more comfortable room.

It makes more sense to say that patient experience is an evaluative response to these diverse events rather than their cause. This means that the neat mathematical properties of the psychometric model – which depend on the way that responses are correlated – do not hold. Psychometric tools, therefore, need to be chosen with care. Applying the wrong statistical criteria in survey development can lead to distortions in the questionnaire, which might not then fully reflect service users’ priorities.

The Picker Principles of Person Centred Care provide a conceptual blueprint for measuring patient experience. When developing a questionnaire or instrument to evaluate person-centred care, we select the most appropriate validation techniques and strive to develop and continually improve methods that are suited to this end. This may well include psychometric techniques, but only those that are sympathetic to this conceptual model.

Tags: PREM, Psychometric, Questionnaire, survey design.

See Also:

Could monthly tracking improve NHS staff morale?

Reports have claimed that the government wants to launch a new regular survey of staff morale as part of efforts to make the NHS the best place to work. But is morale tracking all it’s cracked up to be? Chris Graham argues that monthly tracking may not be as straightforward…

8 January, 2020

Trust and confidence in survey methods

How important is the wording and order of survey questions in determining how people respond? Any textbook will tell you that the answer is “very important indeed”, but real life examples of problems caused by question order and wording remain surprisingly easy to come across. This week, the Telegraph published…

14 August, 2019

Behind the statistics: our new approach to analysing trends

Picker’s Chief Statistician, Steve Sizmur, has almost 30 years’ experience in social research. In this blog, he talks about the Inpatients Survey 2017, and Picker’s new approach to analysing trends with process-control charts. Today’s publication of the 2017 results of the Adult Inpatient Survey marks its 15th iteration. There has…

13 June, 2018

Leave a comment

Talk to us about

Person centred care

Request a callback