Vision Critical Podcasts - Transcripts

Maximizing Respondent Engagement Presented at ESOMAR Congress 2007

Caroline Today we’re talking to Jen Reid and Monique Morden about their conference paper on maximizing respondent engagement through the use of rich media. This paper was presented at the ESOMAR conference in Berlin, and it’s a pleasure to have Jen and Monique here to present their findings with us today. Jennifer Morning. My colleague Monique and I are very pleased today to present our paper on maximizing respondent engagement. Before I get into the meat of the presentation, I’d like to take a minute to talk about why we wrote this paper in the first place. The market research industry is facing a very significant challenge with respect to respondent engagement. There are fewer and fewer people willing to take our surveys. We believe very strongly that the survey experience itself is one of the factors that will increase respondent experience. In my day to day job I have a lot of my panel clients that ask me about respondent retention and how to keep people interested in doing their surveys and they want to jump more into talking about incentives and newsletters and other kinds of retention devices, but I always like to stop them at the beginning and say the best retention policy that you can come up with is going to be to write engaging and surveys. If the survey experience itself is a pleasant one, people are going to be more willing to come back and answer more studies. And we believe that the visualization of question types is one of the factors that can increase this respondent engagement, and that’s what we’re going to present today. Allow me for a moment to use the film industry as an analogy. This was the first movie that was ever produced, it’s called A Trip to the Moon, it was produced in 1902. And as you can see from this little bit of the film that I’m showing you right now what they basically did when they were first brought to the technology and being able to put pictures up on the screen was to take a single camera and film a stage play. We would argue that we’re basically doing the same thing in online research today. What we’ve done is we’ve taken paper surveys and cut and pasted them onto the screen. We haven’t actually had the opportunity to take advantage of any of the special effects, sound, and those sorts of things that the film industry uses now in order to make the experience more round and more engaging. A couple of definitions too before we get into some of the data. We’re going to use the word flat to mean traditional online surveys, those would be the surveys that we typically see most of the time. We use the word fusion to mean those surveys that we program using flash and other rich media techniques. A couple of examples just so you can see specifically what I’m talking about here are some screen shots of the flat survey that we used. Looks not too bad, but it’s pretty much radio buttons, grids, checkboxes, and does look a lot like what a paper survey would. Here you can see some examples from the fusion survey that we used. The first one here is the card sort, this was replacing the big grids. Slider scales that we can move back and forth in order to give the respondents an opportunity to give us their answer. Multi choice questions, laid out a little bit differently than the traditional horizontal radio buttons or check boxes that you would normally see. We also have a couple of visual question types that really move away completely from the kinds of questions that you can traditionally ask. The map is one of them and Monique is going to talk about that more specifically when she talks about the data. This is an example of using scales to mimic what we’re currently doing with grids. And this last one also Monique’s going to talk about some more is an example of an exercise that maybe we would traditionally use during focus groups and actually recruit people to do that you can’t actually right now do them online. Alright, methodology. We used a split sample source, all of the data was collected online. In Canada we used our own panel, the Angus Reid forum, and in the U.S. UK, France, and Australia we used Research Now’s panel. Samples were polled to be matched identically, and so we require that everyone, whether or not they were going to use the flat survey, or see the fusion survey had to have the flash application so we could get rid of that bias. It is a fairly robust sampling design of 3605 interviews, and that basically worked out to 500 interviews of flat and fusion each in Canada and the US, and 200 each for UK, France and Australia. Being Canadian we felt comfortable with both French and English so the survey was deployed in those 2 languages in the countries where that was appropriate. When we sat down to talk about what we were going to testing we came up with basically 5 hypotheses. The first one, the main vein of this presentation, was that the visual question types would in fact elevate respondent engagement which we measured using some self reported questions, and I’m going to present those in a second. We also assume that we would get higher response rates as a result of that engagement, that we would see better data quality for scale questions, but consistent data for behavioural questions , so the idea being if you give somebody a different way to answer the questions they’re going to be a little bit more variable in their data but that their behaviour is going to be the same if they bought 2 tubes of toothpaste last week, it doesn’t matter how we ask that question. And the 5th hypothesis is that we’d be able to collect some data in completely new ways. So here are the measures we used for survey engagement. We asked at the end of the survey to everyone whether the survey was easy to complete, fun to complete, and more enjoyable than most. I will say that this study was on global warming. Because that’s a fairly hot topic around the world right now, we do see lifts in all of these that are just because of the subject matter, but the subject was the same for both. That’s why you can see that there was a big lift in the fun to complete elements of it. Easy to complete is the first measure that we look at, and we’re happy to see that the mean scores are basically the same. The concern was that because the fusion questions, a lot of people wouldn’t have seen them before, that maybe they would get confused about how they would have to answer the questions. So the fact that they’re both perceived as being equivalently easy to complete is good news to us. In fun to complete and more enjoyable than most we do see across the board a very significant increase in the fusion survey versus the flat. We felt when we looked at this data that we had certainly achieved that engaging piece to it. This was really our big measure of respondent engagement and this was the question that was “if more surveys were like this, I would be” and they were given a choice of “more likely to increase participation, no change in my current participation, or less likely.” And you can see what we’re calling the engagement lift here is about 18 points on fusion over the flat. And that’s what we’re really trying to do with these questions, we’re trying to get people to feel like they want to come back and do more surveys, it’s really about long term engagement, not just how they feel about a specific survey. I’m sure you’re wondering how this looks in terms of different countries, age, and gender, and you can see that we do have a universal engagement lift on all countries, age, and gender. There are certainly some areas where the lift is greater than most. I want to point out that one of the questions that we’ve been asked about this data is how older people specifically feel about the visual question types, and we find it very interesting that both the middle aged group and the 55+ both showed a larger engagement lift as a result of fusion than the 18-34 year olds. We think that’s because the 18-34 year olds are starting to really expect something that’s a little bit more engaging and are less surprised by what we can do with this technology, whereas the older age groups were a little more pleasantly surprised and they’ll probably see more of a bigger lift. We also see a bigger lift with women to men and we think that’s because women tend to be a little more visual. A second hypothesis if you recall was that we’re going to see an increase in completion rates. This is where the tempering news of this data exists. We in fact did not see a lift in response rates across all of the countries. The countries where the broadband internet penetration is higher, we see that the difference is much smaller, for example in Canada which has one of the highest broadband penetrations, but when we compare the loss in response rate to the lift we saw in engagement, they do sort of work themselves out a little bit. So I think we do have to conclude that technology plays an important part in the using of these tools and that we have to be careful when we are using them because we will see a bit of a drop. We also think that the consideration for why that response rate is smaller is more than broadband though. And this is what we found on some of the other fusion surveys that we’ve done. It’s actually the result of a very complex combination of your connection speed, the platform that you’re using, the version of your web browser, the version of flash, the age of your computer; so we’re working harder to do a better job of sniffing out those things, because we found that when we looked at the drops in response rates it was all a result of people dropping after the first or second question. And we think that’s because the load time is simply too heavy. So I think just to conclude the engagement portion of this presentation before Monique talks about the data quality aspects of it. We are really heartened to see that we do see an engagement lift and we are left challenged to get over the technological hurdles that exist. Monique Thank you so much Jennifer. What I’d like to talk about now is to shift gears and look at the data that actually comes out of these fusionized surveys that we do. To start with I want to focus on scales. We often use scales to collect a lot of the attitudinal information from the panelists and respondents that we talk to. And if you can think back to earlier in this presentation where we viewed the traditional way, or the flat way, of having grid questions and having the fusionized way. You can see that it’s a very different experience for our respondents. In fact, what we find when we examine the data on the back end of this process, is that respondents and panelists are actually paying more attention to these questions as they go through and give their attitudinal responses to statements or concepts or policy options, whatever they might have in front of them. So we have fewer people who are in the neutral category when you have a 5 point scale and you get more people who are using the lower end of the scale. I would like to say that this is a nuance thing, you won’t have dramatically different results from your flat survey vs. your fusionized survey, which is a good thing because sometimes it’s necessary to use a mixed methodology in order to cover your survey population. But what we do find is that you’re getting more thoughtful responses and more nuanced responses in terms of the data that comes out of there. The other thing that we looked at beyond just the use of the scale, which was also evidenced by examining standard deviations and f-scores to get into more detailed statistics, is looking at the percentage of people who go through the survey and do what we call flat-lining. So when we have a grid question with a lot of attitudinal statements and the 5 point scale that they’re responding to, there are unfortunately some respondents and panelists who go through and will give the exact same answer category for each of the statements or options that you put there, which in some cases may be totally valid, but in most cases is evidence that they are not really paying attention to the statements that are there and aren’t really putting thought towards how they’re answering the questions. So when we examine the percentage of people who do this flat lining on the traditional surveys vs. the fusionized surveys, we actually find there are six times as many panelists who do the flat lining on the traditional surveys vs. the fusionized. So again, definite evidence that panelists and respondents are paying attention to the surveys, and doing a better job of answering the questions. Alright, next I’d like to show you a chart which shows the use of the different points in the 5 point scale. We’re looking at the top box, the mid box, and the low box. You can see for the mid box, which shows in blue, that the respondents average anywhere from 2 points, to a high of 6 points, used the mid points less, and generally what they do is they move their responses in to the low box, either strongly agree, strongly disagree, or somewhat disagree. What you will see, in general, is that the top box scores do not change. Generally when researchers are looking at the results it is the top box scores that they focus on. I’d like to now move on to the other type of data that was collected in this survey which involved behavioural data. This is looking at actual behaviors of panelists and respondents and having a flat and fusionized version of these surveys. We had a variety of different behavioural questions, everything from which of the following actions have you done in the past 6 months, to how far do you travel by car each year, so a variety of different timelines and topics that we’re covering. Some related to actions, some actually related to a measurement of an actual magnitude in terms of distance or hours. What we found in comparing the data from the flat and the fusionized survey, is that in general there was no differences in responses for this behavioural data. There were a few rare exceptions, in some countries. And, what really became evident to us is that you do have to be very careful in your questionnaire design, as you do in any survey, but especially when you’re using a fusionized question to make sure that what maybe made sense to ask in a flat version, or over the telephone, is translated into a visual format. The one example we have relates to the slider question which pertains to the number of kilometers you travel by car each year. In this case, you can see in this example that the points along the slider scale do not include the actual amounts that are listed there, so it’s less than 5000 kilometers, less than 10 000 kilometers, less than 15 000 kilometers etc. So that actual point doesn’t include that numeric point 5000. So what we did find is that for some points where there is a sort of a mass of people that would congregate for instance around 10 000 kilometers, is that people interpreted that 10 000 kilometers was at that point. That’s just something to keep in mind in terms of questionnaire design moving forward when you’re designing these questions, that there are some slight nuances that need to be considered that might not have been so when you had a flat survey. The next slide which shows a map of the world was a very interesting comparison of question types, so this is taking the fusionized survey and pushing it even farther from complex grid questions, into card sorts and our slider questions. We’ve actually taken a question that had to do with per capita green house emissions, and taken that information and put it into a very visual question that shows a map of the world, in this case you’re actually seeing the results which wouldn’t have been the case. For the respondents they would have seen the map of the world and clicked on the region that they thought had the most per capita green house emissions. We found that there were dramatically different for 3 of the countries that we included in this research project from the flat version to the fusionized version. When we look at the UK, France, and Australia, their perceptions of their own continent in per capita green house gas emissions was dramatically different from the flat and the fusionized versions. This again indicates that, while this was a really interesting way of presenting the data, it does provide a different context for panelists and respondents as they’re answering these questions. By visually showing them a map of the world, you’ve made them much more aware of the context of the question, the size of the different regions that are there, and have taken them out of their own little world which they’re more familiar with, and made them more aware of the other regions there. France for instance, which on the flat version 24% thought that Europe had the highest per capita greenhouse emissions when they see the actual map, it goes down to 4 %, and they tend to shift most of their responses over the North America. Something to keep in mind, very engaging ways to ask questions, but it can change your data. You need a good research design as you go into this process. Another thing that we would like to show you is that these fusionized surveys not only give you more engaging ways to ask questions, they also give you ways to collect data in ways that weren’t possible before. The one example that we have here is what we call a virtual highlighter. In this example, this is a purely text example we used in this particular survey, you can include images, or you can include combinations of images and text, whatever happens to apply to your particular situation in what you’re trying to research. You can present text or visuals to people and it’s a very user friendly and intuitive way for them to go in and highlight certain sections of the text, or sections of the image. It depends on how you frame the question, so you can frame it in terms of what do you like about this, what do you dislike about it, what’s important to you, what don’t you understand, whatever your research objectives are in terms of trying to gain feedback on the text or image that you have in front of them. Respondents love this, it’s very user friendly. They can highlight sections, they can use an eraser to erase something if they’ve made a mistake, and it’s very quick and easy for them to use. What this does on the back end, what you can see is a heat map, which is basically the output of the data that comes out of this. The heat map uses a scale from blue to green to yellow to orange to red in terms of intensity, and the intensity represents the percentage of people who highlighted that section of text or image. It gives you a very quick way to see in this case where the important pieces of information on this particular article related to global warming. You can easily go and look at the red sections of text and see what were the points that were really salient with people and important down to orange and yellow and then see the sections of text that weren’t highlighted at all. Previously what you would do is show people this particular article, and ask in an open ended question, which sections of this text in this article are important to you and they would type it in. That was very useful in the old world. But you did find that 30% of the people who respond to your question don’t actually respond directly to your question, so they end up going off on tangents, or elaborating on certain issues and it doesn’t get directly to what you’re trying to capture. We’ve also used this technique in concert with open ends, so not only do you get people to highlight the important parts to them, then you ask a follow-up question that says why was that important to you. You get really rich, more targeted open ended responses than you do when you have purely open ended response where they spend most of their time just reiterating what as in the article and not why it’s important to them. It’s a really valuable tool that takes what used to be a qualitative approach and now makes it a quantitative tool. So in conclusion and to sort of bring together what Jennifer talked about earlier, and the data side that I talked about, what we found from this research project was that indeed you do get higher respondent engagement and that there are definite positives to this higher engagement that a lot of clients and people who are managing panels can leverage and use to their benefit. There is the caution in terms of the technology influence, so who you’re conducting your research with, are they technologically linked in, do they have broadband, the very basic question should be do they have internet access and how much of your population does have internet access, do they have broadband, are there in certain regions or certain populations complexities in their browser and operating system that could perhaps affect your response rate. But there are advantages to using this beyond just respondent engagement. You will get better variability in your scale data, so more thoughtful responses from your respondents, and there are ways to collect data that you couldn’t previously before, which really opens up new avenues for researchers, and gives us a way to innovate in terms of moving market research forward. Also, just as there was a caution with technology influence, in terms of who you’re conducting your research with, as with any survey you need to have really stringent questionnaire design and really think through your questions and how they’re phrased, and really think through what the answer categories are when you’re fusionizing them that it’s rock solid in terms of the information you’re collecting. And if anyone is interested in learning more about these fusionized surveys and fusionized techniques, there are 2 ways that you can do it. If you’re Canadian, you can go to the Angus Reid Forum and join our forum and experience one of these fusionized surveys, you can go to http://www.angusreidforum.com, And if you’re not Canadian you can go to the Vision Critical website and they have all sorts of demos there where you can experience fusion surveys, and you can take a look at other techniques for collecting data from virtual reality shopping, to shelf set designs, and also highlighter tool. Caroline Any listeners that have comments or questions on the presentation about maximizing respondent engagement send us an email! Our email is podcast@visioncritical.com. I’m Caroline Hickton and thanks for listening to around the block.