Why the long survey is dead

Why the long survey is dead

The long market research survey (anything that takes over 20 minutes to finish) is a dead man walking. Just like a colleague you all know is going to be fired, long surveys are headed towards the insight scrap heap.

Most customers today are not prepared to spend over 20 minutes filling in a survey, and studies have shown that the people that answer long surveys are significantly different from the typical customer. They are heavy users of the internet, and are more likely to buy and browse online. They don’t represent your customers or the population.

Surveys have also become less suitable to the tasks of understanding customers as markets have become more complex. Today’s business landscape involves more brands, more product lines and more channels to reach customers. When companies use surveys to understand their complicated market, the surveys get longer and more tedious. Sending a survey about 20 competitors, as a typical company needs to do to understand its market, will only frustrate your customers.

Most importantly, long surveys are not suitable for mobile devices. About 30 percent of all online surveys are now completed from a mobile device, and this is expected to increase to over 50 percent within two years.

If you are still using long market research surveys, you are wasting money, collecting inaccurate information, annoying customers and missing out on the opportunity to gather impactful business insight.

The Evidence

The following points highlight some of the evidence that long surveys are past their sell-by date.

  • Declining popularity. The most recent ESOMAR Global Market Research report (2014) showed a decline in the use of surveys, as a proportion of all money spent, and this is predicted to fall further.
  • Declining response rates. The gold standard for surveys is the Pew Research Center and they report that response rates have fallen from 36 percent in 1996 to nine percent in 2012. They are expected to keep falling.
  • Satisficing is when a respondent takes the quickest route through a survey to gain prizes and other incentives, for example by claiming to buy fewer products than they really buy, and by straightlining their way through studies. Longer surveys have been shown to be more likely to suffer from satisficing. For instance, a 2011 study of 100,000 surveys showed that as surveys get longer, people spend less time per question.
  • People aren’t grids! Over the past ten to 15 years, there has been more research that confirms that people are essentially emotional and that they employ time-saving patterns to their decision making. For example, Nobel Prize winner Daniel Kahneman, author of the best seller Thinking, Fast and Slow, has shown that about 95 percent of decisions are made by an automatic process, based on habit, emotions and patterns—a process he refers to as System 1. He shows that System 2, what we would consider to be rational thinking, is only used for a relatively small number of decisions. However, a long survey is a System 2 sort of exercise because it asks research participant to reveal the multiple factors that contributed to their decision (for example, by completing a grid of questions about brands and attributes).

The Alternatives

Ten years ago, there were two main options for finding out what customers wanted: the survey and the focus group. Now there is a long list of alternatives, and the list is growing ever longer. Here are three themes that underpin the replacements for long surveys (and increasingly the alternatives for focus groups):

  1. Ongoing discussions. Rather than trying to ask people every question in one go, research is becoming more like real life. I ask a question, you say something, and I listen. Then maybe I ask another question, based on what you have said, or perhaps you ask a question. This theme underpins approaches such as insight communities and customer engagement. The learning is continuous, respectful and focused on what can be achieved. This process tends to be a mixture of discussions and forums.
  2. In the moment. In the moment is about asking people questions at the time that something is happening, for example when they visit a store or when they use a product, while the experience is still fresh. I see this theme manifest itself in many mobile research approaches, especially those where participants gather ethnographic information in the form of photos, diary entries, and video.
  3. Observational data. Observation is used at large and small scale. At large scale we often refer to it as big data (for example looking at media consumption, transactions, or social media). At the smaller scale it is often related to creating experiments, to measure what happens when A is changed for B.

The Big Picture

Many people still use long market research surveys. However, these projects continue only because companies have done them that way for a long time. Many of the people supplying long surveys are on their way out.

Long market research surveys do not work well, are not delivering value, are not delivering insight, and are certainly not delivering impact.

Where should you start? If you have any surveys over 20 minutes, they need to go. Don’t just try to trim them. Consider other alternatives and look to provide better experiences for your customers. Think mobile, think engagement, and think about developing long-term relationships with customers.

No Spam Surveys - an ebook from Vision Critical

  • Its hard to disagree with anything in this blog but you’ve ignored the impact of good questionnaire design on the quality of data produced by surveys. A badly designed survey is a bad survey irrespective of length. Unfortunately, the badly designed survey is not rare (including those produced by the highest profile market research agencies) and is probably a major cause of falling response rates.

    Against this background of consumers being less willing to complete surveys, or to complete them properly, its more important than ever that surveys are designed to be engaging and simple to complete irrespective of platform. There’s still life in the old dog yet but more care than ever is needed to keep them off the life support machine.

    • RayPoynter

      I agree with the need for good design, and that the need has always been there. My point is not that short surveys are always good (there are too many bad ones), but rather that even well-designed long surveys are not an option, and need to be converted to short, well-designed surveys, or to a non-survey option.

  • Ray 20 minutes on a mobile is too long. Well designed short surveys for mobile first will become the norm. The smaller the screen the shorter the survey. Just because the survey works on your phone doesn’t mean it works on the masses. This is especially true in the emerging markets where the devices are very different from those in the US and Europe.

    • RayPoynter

      The problem is that some people will do the survey, event though they may be atypical and produce misleading results. Good clients and trusted advisors need to stop doing long surveys not because they are impossible, but because they are not delivering what is needed.

  • Charles Vaneker

    It depends on the engagement of the participants with the used research methodes, the survey topic and the offered incentive. We test tv pilots online and I noticed people engage in the research and are willing to watch a 30 min tv pilot and fill in a survey. We pay participants a (fundamental) cash incentive and people love to participate in the research. But yes, If we schedule the umpteenth survey on purchase intentions I would say 20 minutes is max.

    • RayPoynter

      Hi Charles, I think that it is an interesting question whether watching a 30 minute pilot show and then doing a 10 minute survey is 40 minute survey or a 10 minute survey. We have clients who get members to play games and then do surveys about the games, they are engaged and popular, and we typically do not count the game time as survey time (I am not talking about gamified surveys her, but actual games).

  • It should make sense for long surveys to be a thing of the past: the most important reason being that fatigue changes the data, starting at about the 20 minute mark. Our research over 11 years keeps showing this. Yet we see survey length increasing year after year, with the average now well over 20 minutes. And mobile adds to the problem since, even if optimized for the device, the same survey just takes longer to complete on mobile. Last week 1460 researchers signed up for our webinar on this very topic, so there’s clearly interest and concern in the industry; and some of our clients are working to modularize
    surveys and make other changes. But widespread industry-change could be slow to come on this.

    • RayPoynter

      Hi Jackie, yes! This has been known for years, but clients and researchers keep doing it. We keep asking for people to adopt evidence-based decision making, but many in our industry seem to prefer to wait until they are obliged to change, rather than choosing to do what is right.

  • Well said. It is always best to begin a search for answers with secondary data (Big Data, purchase transactions, etc.) before launching expensive primary research, which we now know needs to be more finely tuned with new methodologies.

  • Baljit Ghai

    Great article. Could not agree more. This is the premise for our business model.

Subscribe to the Vision Critical blog

Get free customer intelligence tips and resources delivered weekly to your inbox.

By completing this form you consent to receive emails from Vision Critical. You can unsubscribe at any time. Learn more in our privacy policy.