There's been lots of talk about big data over the past few years. The growth of digital data is staggering, and the prospect of being able to harness all that data is exciting. Google, as the most obvious example, has become a global giant by capitalizing on search data. But can everyone be like Google?
The whole big data conversation started back to 2008. That was when Chris Anderson, editor of Wired magazine, made the bold statement that the scientific method was obsolete (and presumably so was survey research). He claimed that we no longer need to test hypotheses about behavior - that we now have enough hard data to tell us all we need to know about how people act and how to shape those actions.
But recently, the excitement around 'big data' has faded a bit. There are the IT challenges involved with managing and accessing petabytes and exabytes of big data. And, as Facebook and others are now finding out, there are also privacy concerns. Perhaps the biggest issue is what to do with the data once you have it. Do you drop your line into this bottomless pool of data, or are you better off having some idea of where the fishing is best? How do you distill all this data to generate actionable insights? How do you put a human face on the data so you can better understand not only what just happened but what might happen when market conditions change?
One of the unlikely big data skeptics is Nate Silver, the baseball statistician turned poll guru who, on his fivethirtyeight.com blog, successfully predicted the outcome of the Presidential election in 50 out of 50 states. (In his post-mortem of election polling, Nate also credited our own Angus Reid Public Opinion polls for providing among the most accurate predictions of the election outcome.)
In his book, The Signal and the Noise: Why Most Predictions Fail - But Some Don't, Silver puts the promise of big data in perspective by noting that, as powerful it can be, big data needs to be filtered and interpreted through the lens of whoever is analyzing the results: "Numbers have no way of speaking for themselvesÛ_ Data-driven predictions can succeed - and they can fail. It is when we deny our role in the process that the odds of failure rise."
Being able to know where to look in the data and how to pull insights from it will continue to be valued in the big data era. Formal survey research may play a smaller role in the research mix as behavior becomes more easily tracked. But the data we get from behavior tracking will still leave huge gaps in understanding -- gaps that can be filled by the kind of cost effective conversations possible through qualitative and quantitative studies conducted on community panels.