Big data—the massive quantities of information collected from sources like CRM and social media—can be tremendously powerful. Big data helps leading retailers like Amazon drive sales, transport authorities improve services and financial organizations assess risk and promote products. The relative newness and apparent power of big data has led many to favor it over more traditional sources of information, such as surveys.
However, as the interest in big data has grown, so too has a recognition of its limitations: it measures ‘what’ not ‘why,’ is a rear-view mirror looking at past behavior, only measures part of the story and only a small percentage of data is ever usefully analyzed.
Experts are also starting to raise questions about big data. In a New York Times article, Gary Marcus, a professor of psychology at New York University, lists nine problems with big data, including how it’s good at identifying large numbers of correlations but weak at determining which are meaningful. In The Signal and the Noise, the big data guru Nate Silver examines how the very act of measuring data tends to change human behavior. And, back in 2013 I reported on the successes and limitations of big data, including the problems caused by noise growing faster than signal, making the messages in the data harder to find as the amount of data becomes bigger.
How do we make the most of big data and deal with the negatives? More and more, researchers are recognizing the value small data brings to big data. By “small data,” I’m referring to human-scale information, for example discussions, personal observations, customer feedback and crowdsourcing.
Here are five examples of using small data to make big data better:
- Talk to people
Sometimes big data shows a pattern, but the underlying ‘why?’ might not be obvious. Often the answer can be found by simply asking the people involved. For example, at the 2013 MRS Conference in London, Lucien Bowater, the director of strategy and insight at the telecommunications company BskyB, talked about a situation where the company’s customers were unsubscribing, but they could not work out why from the data. Bowater had his team contact customers who were unsubscribing to ask them why. Armed with this small data, the team were able to correctly interpret the big data messages and implement solutions.
- Collaborate with customers
A growing number of companies are using insight communities to engage their customers, through ongoing conversations and collaboration. Many organizations are adding transactional and other big data elements to their insight communities in order to gain a 360-degree view of customers, creating customer intelligence platforms. It’s worth noting that most of the companies that thrive on big data (for example Google, Yahoo!, Twitter and telcos such as SingTel and Vodafone) utilize one or more communities with one or more suppliers.
- Get out and observe
Sometimes the data can tell you what is happening, but often you need to get out and look at the situation on the ground. Ben Wellington, the creator of the public data blog Quant NY, demonstrated a fascinating example of in-person observation in a TED talk about using open data. Wellington used public data, an example of big data, to create a map of parking fines in Manhattan. One location was generating a much larger amount of money than any other, but there was nothing about the data that suggested why. Wellington visited the location, an example of small data, and saw that it was caused by ambiguous street markings. The city fixed the problem after he blogged about it.
- Experience the customer’s life
It’s often said that you can’t really understand another person’s experience until you’ve walked a mile in their shoes. Martin Lindstrom, the branding consultant and author of the immensely influential book, Buyology, has turned this approach into a passion, spending some 300 nights a year in other people’s homes, walking in other people’s shoes and closely observing what’s happening in their daily lives. In his new book, Small Data, Lindstrom shows how he uses personal observation to unpick problems on behalf of blue-chip clients such as Disney and McDonald’s. These organizations all have petabytes of big data, but Lindstrom adds small data through his research to provide insight that generates impact. For example, his observations of an 11 year-old German skateboarder produced insight that helped Lego change direction and rediscover its purpose, which in turn revived the brand’s fortunes.
- Use behavioral economics
Behavioral economists like Nobel prize winner Daniel Kahneman and Dan Ariely, author of the bestseller Predictably Irrational, have shown that small experiments can uncover how the unconscious mind is determining behavior. Big data typically has too many variables to really understand why; in behavioral economics, a single element is isolated for experimentation, to determine causality (as opposed to mere correlation). For example, in one study (North, Hargreaves & McKendrick, 1997) different music was played in a supermarket wine section over a period of two weeks to measure impact on sales. When French music was played in the background, the sales of French wine increased greatly, while German music produced a noticeable uplift in the sale of German wine. When customers were asked about the music, most said they hadn’t noticed any.
Find out more about Small Data
If this post has whetted your appetite, you’ll be interested in Martin Lindstrom’s new book Small Data: The Tiny Clues That Uncover Huge Trends. He writes about how he utilizes detailed observations of life to unlock big messages. On February 25, 2016, in a webinar hosted by Vision Critical, Lindstrom will discuss the limitations of big data and the solutions presented by small data.