You can't let the data defeat you in this digital world

technology

Justin Thomas

The cursor blinks in a vain attempt to attract my attention; my mind has taken flight, giving wing to the wind of imagination. The laptop screen has become a window; I'm gazing across a digital vastness. This unending cyber universe, like its physical counterpart, is expanding with imperceptible rapidity. Global data growth is currently at 60 per cent per annum.

The rapid proliferation of digital devices, combined with ever greater storage capacity, and increasingly sophisticated data processing techniques have all coalesced to make "big data" super relevant. The implications of big data are widespread, and over the coming decades they will increasingly shape the way we live our lives, operate our businesses, and govern our societies. Just as our ancestors pondered the celestial bodies trying to predict portentous events, so too our own awed contemplation of big data is slowly morphing into innovative applications aimed at predicting and explaining human behaviour.

Social psychologists argue that predicting and explaining other people's behaviour is just something we humans do, both naturally and routinely. Like amateur scientists we look at the observable data in other peoples' actions and start to make all kinds of inferences. Many of us will recall eyeing other people's purchases at the checkout, and watching our thoughts run to speculations about what their lives might be like: Hello magazine, beep, three tins of cat food, beep, a 12 pack of red-bull, beep, low-fat vegetarian lasagna, beep. For decades supermarkets have systematically used electronic point of sale data to understand and predict consumer behaviour. Retail giant Walmart knows that a hurricane forecast will increase the sale of "pop tarts", along with other more obvious survival items such as, flashlights and batteries.

But big data is bigger than a single organisation deriving intelligence from it's own monolithic data sources. According to Google executive Megan Smith, big data is increasingly about "adjacency", the idea that we can now combine previously disparate data sources traversing geographic and temporal boundaries. Such combinations can help us look for predictive patterns and relationships in new ways. Imagine having combined access to Amazon account data, and medical health records. Could we predict mortality and morbidity by book choice? Customers who bought the following books also bought book X, and were 25 per cent more likely to suffer from a mental illness. More realistically you might have an online supermarket that could recommend healthier product options based on the individual's health data. Over the long haul we might identify products or product combinations associated with an elevated prevalence for particular illnesses.

Another major development in big data's rise is the recent advances in unstructured data analysis techniques. Such techniques now make it possible to trawl through unstructured data such as text messages, e-mails and even images, to extract meaning and actionable intelligence. This is probably one of the reasons Google's chief economist, Hal Varian suggests; "The sexy job in the next ten years will be statisticians… The ability to take data—to be able to understand it, to process it, to extract value from it, to visualise it, to communicate it."

We can get a foretaste of big data through some of Google's power-user services like Google trends and zeitgeist. These services allow us to see which search terms people are using most frequently. We can also enter our own random terms and see how frequently they've been searched for. The output is ranked by nation, and the trend data for the last seven years is displayed. For example, if we explore the use of "Abu Dhabi" as a search term, we get a list of nations ranked by frequency of search term use. Number one is of course the UAE, followed by Qatar, India, and then Egypt, all easily explicable. The current number five however, is Ireland, a slightly more insightful surprise. The biggest spike in UAE related search volume happened early in 2010 when the world's largest building was opened, and no prizes for guessing when Qatar experienced its largest ever Google search volume spike. For a short time Qatar was actually as popular as Dubai in terms of search volume.

Like it or not, big data is here to stay. This overly helpful guest may occasionally infringe on your privacy, but he always knows exactly what gifts you would like.

Justin Thomas is an assistant professor at Zayed University in Abu Dhabi