As a researcher, I work with data all the time. I find data nerdishly fascinating, and I’m excited to see data come together in one dazzling story.
But things don’t always work out this way. Sometimes I’m going on a wild goose chase for that story – slicing and dicing the data in all ways I can imagine hoping to uncover somethingÛ_anything. And other times, my data is story-less and boring.
For a researcher dealing with data, there’s always an implicit fear that: (a) your hypothesis is incorrect and does not yield significant results, (b) the data is counterintuitive and unexplainable given the context, or (c) your tracking data doesn’t track well – meaning changes that occur in the data are not necessarily based on external market variables but are rather attributed to methodological factors, such as consistency in sampling.
The last time I dealt with data that qualified to be one of the above, it was for a high profile project and we had a bit of a freak out. Our team originally had a sample size that we thought was sufficient (at the very least adequate for directional results vs. significant results), so we drew conclusions based on this data knowing we would run the study again using a much larger sample. But when we used a larger sample size the next time around, what came back was a different story.
Maybe I’m being dramatic, but I think all who were involved in the project grieved. Our sexy story disappeared and what we were left with was less attractive. If you’re a researcher, these 5 steps of grieving for bad data probably sound familiar:
Step 1: Denial – ”This is not happening.”
The natural knee jerk reaction is to deny that it’s happening. You see the data is inconsistent, but perhaps the issue is caused by a variation in other things like the sample composition (age, gender, region), the way the question was asked, or the field window or timing of the study. Or perhaps you’re hoping that your results still fall within the margin of error. Surely there must be a quick fix?
Step 2: Anger – ”How can this happen to me? Who is to blame?”
Getting angry and placing blame on something – or sometimes, someone – is often part of the process. Why did we change the wording of the question? Whose bright idea was that anyway? (It was mine – I changed it for clarity and specificity). Thankfully, this step is often short-lived because the next step arrives quickly.
Step 3: Bargaining – ”I’ll do anything to make it better!”
Researchers are problem solvers, so quitting almost isn’t an option. Maybe I’ll weight the data? I could run it in field again? Maybe I’ll revert back to the older data or change the question back! This phase of grieving is both creative and crazy.
Step 4: Depression – ”I’m so sad, the data isn’t tracking. My story has changed. This sucks. How am I ever going to explain this to Client X?”
Bargaining naturally produces ideas that will not work – ones that will only delay the inevitable – and realizing this can be a sobering thought. Perhaps we should just give up. We could have been wrong from the get go. Should we pull out of this project or just pitch a new story? If you’re too invested in the project – and really, is there a researcher who isn’t? – sadness becomes part of the process.
Step 5: Acceptance – ”It’s going to be OK.”
This is my favorite part, for obvious reasons. This is where researchers realize that we don’t have the original glam story but instead, we have a different story that is still a great one. We will re-work it and it will still be awesome. High-fives all around.
In our project’s case, we found that although our findings in the smaller sample size did not hold as strongly with the larger sample size, we were still well within the margin of error. The result isn’t what we had in mind originally, but the data still had an insightful story to tell.
I would rather not work with data that do not track well – or any type of bad data, for that matter – but as every researcher knows, it is part of the process. It keeps our jobs interesting and rewarding. And sometimes, the story is there – we just need to look at it through a different lens.
How do you deal with story-less data? Let us know.