Covid-19 Update, Autumn 2021: Picker continues to operate as usual. Learn more


Chris Graham

Trust and confidence in survey methods

How important is the wording and order of survey questions in determining how people respond? Any textbook will tell you that the answer is “very important indeed”, but real life examples of problems caused by question order and wording remain surprisingly easy to come across.

This week, the Telegraph published a story [paywall] including some striking statements based on a ComRes poll. The story lead with a claim that “54 per cent of British adults think Parliament should be prorogued to prevent MPs stopping a no-deal Brexit” – but this reporting has been criticised by a variety of commentators because:

Both the Telegraph and ComRes are respected organisations and the purpose of this blog is neither to second guess their motives nor to comment on public attitudes to Brexit. But it does seem highly likely that the order and wording of questions included in the poll, coupled with the way the results were reported, will have overstated public support for proroguing of parliament – and this demonstrates the importance of balance in surveying.

This type of error in survey design is far from new – indeed the importance of wording, order, and context in surveys is so well known as to have been satirised in a 1986 episode of Yes, Prime Minister:

Most of the time, context effects in surveys are more likely the result of oversight rather than the kind of calculated manipulation demonstrated by Sir Humphrey above. Without experience of how different questions can interact and how the precise wording of statements can influence people’s responses, it’s easy to overlook issues that can have a profound impact on the validity of survey findings. The moral of the story is that designing good, balanced surveys takes care, skill, and expertise.

At Picker, we follow a robust process to ensure that the questions we ask in surveys are balanced and fair and that they are reported accurately and meaningfully. This includes careful design by experienced researchers; systematic testing with the groups of people who will respond to the survey; and a commitment to precision in the way we describe our findings. These steps are important to us because they are part of how we live two of our core values: excellence and integrity. By using best-practice methods we can produce evidence that we are confident in and that our users can trust.

If you’d like our help with a survey, or if you want to find out more about our approach, please don’t hesitate to get in contact with us at We’d love to hear from you and we’d be delighted to help.

Tags: integrity, survey, survey design, survey methods.

See Also:

Are NHS complaints really ‘the tip of the iceberg’?

“It is difficult to know how many people want to complain, but maybe we should be asking whether complaints are the right measure of quality…." It’s been widely reported that complaints about the NHS are on the rise – and, worse, that the complaints that are received may reflect only…

22 September, 2014

Leave a comment

Talk to us about

Person centred care

Request a callback