I have noticed a pattern about many of the market research reports I have seen recently (and unfortunately some I have written). The trend seems to be to move away from the Total picture as quickly as possible to look for interesting stories within sub-groups, usually sub-groups of people, but sometimes sub-groups of the features or attributes.
This process, moving quickly from the total picture to sub-groups, seems to fit well with the plea from clients to look deeper and to find the hidden insights. However, this process runs the risk of reducing the impact of the main message in the data in pursuit of smaller interesting issues.
The Titanic Example
Perhaps I can illustrate what I mean about the market research shift of focus by using the story of the sinking of the Titanic as an example. The big story is that the Titanic sank on its maiden voyage, after hitting an ice-berg, with 1514 people being killed. The next level of information might include: 710 people survived, it was travelling from London to New York, that there were only enough lifeboats for 1178 people, that it happened at night, and that the inquiry suggested excessive speed in an area with icebergs was a factor in the sinking.
This information represents most of the important facts, the ones that most people would want to know, and they all relate to the total picture. However, if a typical market research report were produced we might see a comparison between percentage survival/drowning rates by type of passenger.
A market researcher might summarise the sinking of the Titanic in the following way: Overall, 32% of those on board survived. Three groups fared worse than the average traveller. These three groups were Male Crew (22% surviving), Men in Third Class (16% surviving), and Men in Second Class (just 8% surviving). By contrast, the three groups with the highest survival rate were Female Crew 87%, Women in First Class 97%, and Children Travelling in Second Class 100%.
This detailed picture will be interesting to a few, particular those looking at the historical sociology of accidents, but it obscures the big picture if it takes precedence in the reporting.
Why Might Researchers be Focusing on the Chaff?
In discussing the problem of the tendency of market researchers to dive too quickly into reporting detail one of my muses suggested that one of the reasons related to the type of software we use today. In the 'old days' the first thing a researcher did was to run frequencies on the total data to check that the previous processes had been completed correctly. For example, did the N match the number of questionnaires, did the hole count show any concerns, was the spread of responses what was expected? This 'process driven' procedure led fairly naturally to the researcher thinking about the data at a total level, before starting to look for additional stories.
By contrast with the 'old days', much modern data is collected via online data collection, which means there is no data transcribing, punching, cleaning etc. The first time the researcher sees the data it is likely to already be formatted as tables, with a Total column and often 20 columns of breaks, with statistical differences between the columns being highlighted. Visually, the researcher is drawn towards the differences (which are often small) and away from the Total column. Indeed the Total column is often used as a reference point for the rest of the data, rather than as the source of the most important information.
The Infographics Exception
One exception to focusing on the smaller differences, rather than the bigger picture, is provided by most informative and engaging infographics. If you flip through David McCandless's book Information is Beautiful you will notice that most of the pages look at data at the total level. The reason is simple, infographics are very wasteful of space, the amount of data per square centimetre is very low, which means the infographic has to concentrate on a high level view, which in turn is often the most important aspect of the information.
A Recommendation for Stressing the Total Picture
My recommendation, and something I have started putting into place, is to start any write up with one of the two following positions.
1. The total picture revealed by the data is XYZ and this is true of most of the data, although there are some interesting and potentially useful variations in the data. I then provide a clear, attractive, and impactful description of what the total picture is.
2. The total picture of the data is XYZ, but that picture hides some important differences which need to be assessed as top level concerns, and there are of course some interesting and potentially useful variations in the rest of the data. I then provide the big picture and the reasons why the big picture is misleading as the first part of the analysis/report, only then moving onto the nuances.
Here are two examples that help explain my two cases drawn from recent AskVC* studies that I have been involved in, one about global warming and one about smartphone choices.
Is global warming caused by mankind? (see at SlideShare) - an example of where to the total data provide the big story.
In total, 73% of the UK sample said global warming was mostly or partly caused by mankind. That is the big story. Further, just 5% thought that global warming was not happening. This data contrasts strongly with the picture portrayed by a large number of journalists, some politicians, and a vanishingly small percentage of scientists from relevant disciplines.
The data did contain statistically significant differences, which is not surprising since the sample size was just over 2000. For example, 78% of 18-34 years said the cause was mostly or partly mankind whereas 68% of those aged 55+ said the cause was mostly or partly mankind. Depending on what the researcher was looking for, that difference may or may not be interesting, but it does not change the story that a comfortable majority of every age, region, social class, and gender group say that climate change is happening and is at least partly caused by mankind.
If money were no object, which phone would you prefer? (see at Slideshare) - an example of where the big story is partly the total data and partly drawn from deeper analysis.
In the total data, 49% of people said they would choose an iPhone 'if money were no object', with Android, Blackberry, and Nokia all being preferred by similar, but small, percentage (15%, 13%, and 12% respectively).
However, that big picture hides an important message revealed by deeper analysis. The data suggest that the iPhone and Android tend to draw their support more from younger people than older people. Blackberry and Nokia tend to draw their support more from older people than younger people. If one subscribes to the view that mobile phone trends tend to migrate from younger people to older people, this would suggest that the iPhone and Android are positioned well for the near future, whereas Blackberry and Nokia have a more challenging support pattern.
* AskVC is a free service from Vision Critical. Potential questions are shouted out on Twitter using the hashtag #AskVC . The questions are reviewed and the winners are asked on Vision Critical's omnibus in the UK with a sample size of about 2000. Check the latest news on Twitter.