Trust and confidence in survey methods
How important is the wording and order of survey questions in determining how people respond? Any textbook will tell you that the answer is “very important indeed”, but real life examples of problems caused by question order and wording remain surprisingly easy to come across.
This week, the Telegraph published a story [paywall] including some striking statements based on a ComRes poll. The story lead with a claim that “54 per cent of British adults think Parliament should be prorogued to prevent MPs stopping a no-deal Brexit” – but this reporting has been criticised by a variety of commentators because:
- The order of the questions and adjacent items may have put “people in an anti-parliament, anti-political elite mode” (specifically, prior questions asked respondents whether they agreed that ‘parliament is out of touch with the British public’ and that ‘most MPs seem to ignore the wishes of voters and push their own agendas’);
- The wording of the question respondents answered didn’t mention proroguing parliament, but rather “[delivering] Brexit by any means, including suspending parliament if necessary”; and
- The reporting of the result excludes 19% of respondents who said they “don’t know”: with these included, only 44% (a minority) answered in agreement.
Both the Telegraph and ComRes are respected organisations and the purpose of this blog is neither to second guess their motives nor to comment on public attitudes to Brexit. But it does seem highly likely that the order and wording of questions included in the poll, coupled with the way the results were reported, will have overstated public support for proroguing of parliament – and this demonstrates the importance of balance in surveying.
This type of error in survey design is far from new – indeed the importance of wording, order, and context in surveys is so well known as to have been satirised in a 1986 episode of Yes, Prime Minister:
Most of the time, context effects in surveys are more likely the result of oversight rather than the kind of calculated manipulation demonstrated by Sir Humphrey above. Without experience of how different questions can interact and how the precise wording of statements can influence people’s responses, it’s easy to overlook issues that can have a profound impact on the validity of survey findings. The moral of the story is that designing good, balanced surveys takes care, skill, and expertise.
At Picker, we follow a robust process to ensure that the questions we ask in surveys are balanced and fair and that they are reported accurately and meaningfully. This includes careful design by experienced researchers; systematic testing with the groups of people who will respond to the survey; and a commitment to precision in the way we describe our findings. These steps are important to us because they are part of how we live two of our core values: excellence and integrity. By using best-practice methods we can produce evidence that we are confident in and that our users can trust.
If you’d like our help with a survey, or if you want to find out more about our approach, please don’t hesitate to get in contact with us at firstname.lastname@example.org. We’d love to hear from you and we’d be delighted to help.