Friday, Oct 24, 2014

Don’t count your chickens

New York Times | Posted: April 9, 2014 12:37 am | Updated: April 8, 2014 11:46 pm
Big data programs for grading essays rely on sentence length and word sophistication, found to correlate well with scores. But once students figure out how such programs work, they start writing  long sentences and using obscure words. Reuters Big data programs for grading essays rely on sentence length and word sophistication, found to correlate well with scores. But once students figure out how such programs work, they start writing
long sentences and using obscure words. Reuters

BY: Gary Marcus and Ernest Davis

Big data is not all it’s made out to be.

Big data is suddenly everywhere. Everyone seems to be collecting it, analysing it, making money from it and celebrating (or fearing) its powers. Whether we’re talking about analysing zillions of Google search queries to predict flu outbreaks, or zillions of phone records to detect signs of terrorist activity, or zillions of airline stats to find the best time to buy plane tickets, big data is on the case. By combining the power of modern computing with the plentiful data of the digital era, it promises to solve virtually any problem — crime, public health, the evolution of grammar, the perils of dating — just by crunching the numbers.

Or so its champions allege. “In the next two decades,” the journalist Patrick Tucker writes in the latest big data manifesto, The Naked Future, “we will be able to predict huge areas of the future with far greater accuracy than ever before in human history, including events long thought to be beyond the realm of human inference.”

Is big data really all it’s cracked up to be? The first thing to note is that although big data is very good at detecting correlations, especially subtle correlations that an analysis of smaller data sets might miss, it never tells us which correlations are meaningful. A big data analysis might reveal, for instance, that from 2006 to 2011 the United States murder rate was well correlated with the market share of Internet Explorer: both went down sharply. But it’s hard to imagine there is any causal relationship between the two.

Second, big data can work well as an adjunct to scientific inquiry but rarely succeeds as a wholesale replacement. Molecular biologists, for example, would very much like to be able to infer the three-dimensional structure of proteins from their underlying DNA sequence, and scientists working on the problem use big data as one tool among many. But no scientist thinks you can solve this problem by crunching data alone, no matter how powerful the statistical analysis; you will always need to start with an analysis that relies on an understanding of physics and biochemistry.

Third, many tools that are based on big data can be easily gamed. For example, big data programs for grading student essays often rely on measures like sentence length and word sophistication, which are found to correlate well with the scores given by human graders. But once students figure out how such a program continued…

comments powered by Disqus