Tags:
Journalism of Courage
JOHN MARKOFF
More than 60 years ago,in his Foundation series,the science fiction novelist Isaac Asimov invented a new sciencepsychohistorythat combined mathematics and psychology to predict the future. Now social scientists are trying to mine the vast resources of the InternetWeb searches and Twitter messages,Facebook and blog posts,the digital location trails generated by billions of cellphonesto do the same thing.
This is a significant step forward, said Thomas Malone,the director of the Center for Collective Intelligence at the Massachusetts Institute of Technology. We have vastly more detailed and richer kinds of data available as well as predictive algorithms to use,and that makes possible a kind of prediction that would have never been possible before.
This summer a little-known intelligence agency began seeking ideas from academic social scientists and corporations for ways to automatically scan the Internet in 21 Latin American countries for big data. The three-year experiment,to begin in April,is being financed by the Intelligence Advanced Research Projects Activity,or Iarpa pronounced eye-AR-puh,part of the office of the US director of national intelligence. The automated data collection system is to focus on patterns of communication,consumption and movement of populations. It will use publicly accessible data,including Web search queries,blog entries,Internet traffic flow,financial market indicators,traffic webcams and changes in Wikipedia entries.
It is intended to be an entirely automated system. The research would not be limited to political and economic events,but would also explore the ability to predict pandemics and other types of widespread contagion,something that has been pursued independently by civilian researchers and by companies like Google.
A similar project by a military sister organisation,the Defence Advanced Research Projects Agency,or Darpa,aims to automatically identify insurgent social networks in Afghanistan. In its most recent budget proposal,the defence agency argues that its analysis can expose terrorist cells and other stateless groups by tracking their meetings,rehearsals,and sharing of material and money transfers.
So far there have been only scattered examples of the potential of mining social media. Last year HP Labs researchers used Twitter data to accurately predict box-office revenues of Hollywood movies. In August,the US National Science Foundation approved funds for research in using social media like Twitter and Facebook to assess earthquake damage in real time. The accessibility and computerisation of huge databases has already begun to spur the development of new statistical techniques and new software to manage data sets with trillions of entries or more. Big data allows one to move beyond inference and statistical significance and move toward meaningful and accurate analyses, said Norman Nie,a political scientist who was a pioneering developer of statistical tools for social scientists and who recently formed a new company,Revolution Analytics,to develop software for the analysis of immense data sets.
Im hard pressed to say that we are witnessing a revolution, said Prabhakar Raghavan,the director of Yahoo Labs,who is an information retrieval specialist. He noted that much had been written about predicting flu epidemics by looking at Web searches for flu but noted that the predictions did not improve significantly on what could already be found in data from the Centers for Disease Control and Prevention. You can look at search queries and divine that flu is about to break out, he said,but what our research has highlighted is that many of these new methods dont add a huge lift.
Other researchers are far more optimistic. There is a huge amount of predictive power in this data, said Albert-Laszlo Barabasi,a physicist at Notre Dame who specialises in network science. If I have hourly information about your location,with about 93 per cent accuracy I can predict where you are going to be an hour or a day later.
Still,the ease of acquiring and manipulating huge data sets charting Internet behaviour causes many researchers to warn that the data mining technologies may be quickly outrunning the ability of scientists to think through questions of privacy and ethics.
