Have you ever hired a new university grad? The process can be long and rigorous, usually involving evaluating the person carefully to determine if he or she is the right fit. When talking to a new grad, you're usually looking for potential. You want to know if the person has what it takes to get the job done.
Evaluating a voice of market panel solution is a lot like hiring a fresh graduate. It involves evaluating many alternatives and doing your homework. And just like hiring an awesome new employee, getting the right voice of market solution can be a huge win for your organization. Listening to the market can give you a broader view outside of your own insight community.
But for a market panel to give you accurate insights, you need to look under the hood and make sure you're employing the right solution. Basing your decision on price alone might lead to bad quality data and wrong conclusions, which may then result to costly business decisions. A high quality national market panel will give you information closely reflecting the thoughts and feelings of the general population.
Here are 3 questions you absolutely need to investigate before employing a voice of market solution:
- How closely does it match your requirements?
You wouldn't hire a person unless you've gone through their professional experience. Internships, previous jobs, and maybe even their volunteer experience are some things to consider before hiring. The person's experience needs to closely match your business needs.
Similarly, a voice of market solution needs to match your research needs: it needs to be representative of the general public in its composition so you can confidently draw insights from your studies. For a market panel to be useful, sample pulls should closely mirror census data.
When evaluating a voice of market solution, ask the company for the market panel's demographics. Compare age, gender, language, employment and household income distribution with data from the national census. You can pull a comparative sample, so the market panel doesn't have to be exactly identical with the census. However, the market panel needs to closely mirror national data to be considered representative. Also, pulling a comparative sample is easier when the market panel's composition already closely matches census data. Once you employ a market panel, you can control census comparative sampling on the front end and weighting on the back end.
- How was it built?
A person's professional background matter, but you also want to know how they built their experience. It's not enough someone did an internship. What kind of internship did the person go through? What has she learned from the experience?
Likewise, when evaluating a voice of market solution, you want to know its history and composition. Check for things such as the following:
- Recruitment process. How did the company recruit people into the market panel? The insights you get from the voice of the market solution are only as good as the people participating. Learn if the company's recruitment strategy gathers from a diverse number of sources and engaged non-incentivized cross sections of the population. Recruitment should focus on quality over quantity. A bigger panel doesn't necessarily equate with quality or ability to deliver participants
- Partners. There are more market panels these days being built with partners (eg: airlines, retailers). While this isn't an unusual practice, you also want to ensure that the market panel is made up a multitude of sources. Having both partner and non-partner sample as sources helps create a market panel that is more representative of the population. It also helps reduce bias by ensuring one source isn't dominating the market panel and potentially introducing an unintended bias into the study. For example, a retail study running on a voice of market solution built mainly through a retail partner's shopper database could have a major effect on respondent bias.
- Blending of samples source/river source. As more voice of market solutions become enabled, so does the proliferation of blending multiple sample sources. Ask questions such as:
- What are the sources you'll be employing?
- When you say river, what does that mean?
- How will you control for maintaining consistent blends across multiple cells?
- How do you control for duplicate respondents?
- Ongoing quality control. To continuously get the highest quality data, the voice of market provider needs to constantly refine its management practices. The company needs rigorous quality control, making sure people are engaged during and in between surveys without overwhelming them with emails and invitations. In addition to making sure people are engaged, the company needs to be employing a lot of quality control procedures to catch satisficing behaviors such as straight lining. Ongoing engagement and quality control help ensure the data you get back from the panel remains reliable and useful.
- Does it measure up?
Generally speaking, hiring managers check a person's educational background before hiring or they check references to assess their previous performance. Grades aren't make-or-break factors, but they are a good indication of a person's smarts and work ethic.
When evaluating different market panels, there are also several metrics you can check. For example, a response rate outperforming the industry is usually an indication of health. We recommend using a voice of the market solution with at least a 25 percent response rate - anything lower and you may be dealing with inherent bias in your data. To maintain a high response rate, the voice of market solution provider needs to engage its members, provide intrinsic incentives, and respect people's time and attention. It involves cultivating a long-term relationship with people.
A market panel that is representative of the general population is an important tool in consumer insights. But just like hiring a new person for your team, you need to dig deeper and ask good questions if you don't want to make the wrong choice when evaluating voice of market solutions.
What steps do you usually take to evaluate a market panel? And what other metrics besides response rates do researchers need to look into? Let us know in the comments.