Why you can't trust surveys: The Response Bias(es)

In my experience, I do not put much trust into the responses of surveys or polls.

Especially when it comes to asking people what they think or feel about a topic, which so often happens in new product development and innovation.

And there is a simple reason for this:

What people say they would do, and what people actually do, are two completely different things.

Let us look at a classic example: The United States 2016 election.

With Hillary Clinton going up against Donald Trump before the election, even on the day of the election, people were being asked in polls around the country how they would vote.

With thousands of people asked, it seemed almost certain that Hillary would win, based on who people said they would vote for. The polls seemed certain in their prediction.

And yet, overall Donald Trump won the election.

Another classic example is that while newspapers were still the population’s most trusted source of news and information, when asked which newspapers people read, people would list “respectable” broadsheet newspapers like The Times or The Guardian. Yet throughout history, the best selling newspapers were always the tabloids like the News of the World.

And even in experimental settings, people put under pressure to not stand out within a group will give answers to tests they know are wrong, just because other people around them gave these answers.

So why do people not always tell the factual truth when they are asked what they think?

Psychologists believe it is due to a series of biases we have called Response Biases.

Response biases are a tendency for people to respond falsely or inaccurately to questions, due to a number of different reasons. They might do this because they feel pressure to conform to social desirability, trying to please the people asking, but also when they know that there will be little to no impact of hiding what they really think. After all, if nobody except you knows what you really think and asks you about a hypothetical situation, there is essentially no impact on you by lying.

This can make the data based on self-reports or interviews highly inaccurate, and is a big problem when studying opinions, psychology or predicting behaviour.

In fact, there may be multiple other biases why people don’t say or report what they really think when asked, including:

Acquiescence Bias: Also known as “Yes-Saying”, this is the tendency of people to prefer agreeing with the positive potential answers, such as by answering “I agree / True” more than “I disagree / False”. This may result in people giving contradicting answers across multiple questions, such as saying “I strongly agree” to a question asking if they prefer spending time with people, and the same answer to a later question asking if they prefer spending time alone. This trait seems to be especially strong in cultures that value collectivism and prefer to avoid uncertainty.
Courtesy Bias: This is the tendency for people to give the answer they think the questioner wants to hear. They might do this in order to be polite or not offend anyone, or to not appear as a burden when in fact they need help. This bias can be especially damaging for innovation projects or when asking people face to face for their opinions on new ideas or products. People being asked what they think might see that the person asking is invested emotionally in the idea, and therefore give positive feedback to please the questioner even if they would never actually use or buy the idea themselves.
Social Desirability Bias: Perhaps the hardest one to manage, this is the tendency when people are asked about themselves to give answers which they believe fit in best with what is socially desirable. When asked, people give socially desirable responses instead of choosing responses that are reflective of their true feelings. This is especially true when people are asked about taboo topics such as sexual activities, illegal behaviour such as social fraud, or unsocial attitudes such as racism, where the responders are likely to under-report times when they did or agreed with those issues. Like many biases, what further complicates things is that while some responses are to deceive the survey, sometimes the person may be deceiving themselves by thinking they would do something socially desirable, when in reality they would not. Researchers have however found that one way of reducing the impact of this bias is to ask about a topic in an indirect way, instead of asking what a person would personally do.

All of these biases mean it can be incredibly difficult to get a true understanding of what people think about a new idea.

People may say they like an idea. They might even say they would definitely buy it and pay money for it when it is released. But when it comes to asking if they would pay money for it there and then, they will find an excuse why not.

That is why instead of surveys, polls or focus groups to get feedback on new innovations or ideas, I almost always suggest a simple alternative: behavioral validation tests.

Put an option in front of people to guage whether they want to find out more about an idea and are interested in it. Then observe what they do next and how they behave.

This could be as simple as putting a short description of your idea on a website with a button that says “find out more” and seeing how many people click it, to putting together a fake checkout page for the product, having people think they are entering their credit card information and at the end displaying a message that “sorry, the item is now out of stock. You have not been charged. Enter your email and we will let you know when it is available again“.

In these types of experiments, you are validating how people actually behave, instead of what they say they might theoretically do.

And this is a much more accurate way to validate interest.

Bio
Latest Posts

Nick Skillicorn

Chief Editor, Founder & CEO at Improvides Innovation Consulting

Creativity & Innovation expert: I help individuals and companies build their creativity and innovation capabilities, so you can develop the next breakthrough idea which customers love. Chief Editor of Ideatovalue.com and Founder / CEO of Improvides Innovation Consulting. Coach / Speaker / Author / TEDx Speaker / Voted as one of the most influential innovation bloggers.