The Volume Game: Book review of Big Data– Does Size Matter and Small Data

The short answer to the question posed on the cover of comedian and math proselytiser Timandra Harkness’ book is: of course, size doesn’t matter. It never matters, except in the King Kong and Godzilla movies. In all other things, it’s the approach which matters. Big Data approaches are defined by vastly distributed and parallelised storage and processing strategies. The size of the data set is secondary, but things really begin to buzz when volumes are overclocked.

Harkness, who has a delightfully light touch, points out that the volume of data out there – which she visualises as terabyte hard disks packed in suitcases on an airport baggage trolley — is growing so fast that figures are out of date by the time they are published. Anything which is logically connected to anything else is generating data and the more cash-sparse an economy is, the richer these connections are. If a travel card is recharged by an electronic wallet connected to a bank account and a mobile number (which constantly reports call and location data), enough data is generate to profile the owner.

The curve is going to get steeper as the Internet of Things kicks in. Internet-connected cars, refrigerators, baggage tags, shipping pallets and the like are going to generate data like foam, and algorithms are being written to mine it. Herein lies the main operational difference between traditional statistical methods and big data approaches: the latter depend completely on artificial intelligence which learns as it goes along. You first instruct it in the rudiments of pattern recognition and as it improves itself, it should theoretically reach a point where it plows through the data sea seeking patterns which you did not expect to find, but would be interested in.

Interestingly, the human intelligence of branding guru Martin Lindstrom seeks out rather similar patterns. Since they are culled from rather small sample sets, his conclusions may be treated as “anecdotal” or as “insight”, depending on their utility. In the first category is his extraordinary observation that suited and booted business travellers in airports tends to have their boarding cards facing downwards in their pockets. Because they want to conceal the fact that they are flying economy. This has excellent entertainment value, but no utility. That honour goes to Lindstrom’s observation that apparently turned around Lego, which was losing ground to the instant gratification offered by digital gaming. He asked an 11-year-old Lego fan what his most prized possession was. It turned out to be a pair of old sneakers scuffed at precisely the right angle to announce to the world that their owner was a champion skateboarder. From gratification, Lego turned its focus to visible proof of achievement, like the expensive models of the Millennium Falcon that even grown-ups play with.

Insightfully, Lindstrom distrusts Big Data. Because one of these days, AI will replace human gurus. Pattern recognition is much more reliable than insight. It is an innate human skill which machines are now learning. Harkness alludes to the pathbreaking work of the British epidemiologist John Snow, who used statistical techniques to trace the 1854 cholera outbreak in London to a single handpump in Soho. Today, AIs crunch Google searches for medical advice, from which they map wavefronts of approaching epidemics, giving health services strategic depth. Big Data is already being used by corporates and governments and will influence our world in ways yet unimaginable. If it is kind, room for human insight will remain.