by Mark Patron, CEO of RedEye International.

23 Aug 2011: Big data is exploding. According to a new report by the McKinsey Global Institute, "Big data: the next frontier for innovation, competition and productivity" as more and more information is collected and stored daily, we need to prepare. If not we're going to be completely overwhelmed. The report states that by 2018 there will be a shortage in the US of 140,000 to 190,000 people with the deep analytical skills needed and with 70% of the world’s data stored in Europe and North America we will probably see a similar shortfall in Europe.

The numbers are quite extraordinary.

• Over 5 billion mobiles phones in use today.
• $600 buys you a disc drive that can store all the world’s music.
• 40% growth in data each year compared with 5% more spend on IT.
• 90% of the data in the World today has been created in the last two years.
• Thank goodness for Moore’s Law which states that for the same money computing power doubles every two years.

Opportunities

The McKinsey report makes some big claims for big dat. By using big data for better segmenting and targeting, plus offering other benefits, retailers can increase operating margins by 60%. Tesco and Amazon are held up as leaders in capitalizing on big data. At one point 30% of Amazon sales were reported to be due to its product recommendation engine.

I believe some of the report’s potential efficiency gains will not materialise. For example a €250 billion efficiency saving in European public administration and $300 billion in US healthcare seems like a stretch. At the very least the benefits are likely to be hidden.

Not surprisingly the biggest beneficiaries will be information businesses such as internet companies that can collect huge amounts of online behavioural data. Interestingly, McKinsey graded these information businesses as one of the lowest in terms of a data- driven mind-set and this may well be their greatest bottleneck in capitalizing on data. Internet companies are only just waking up to the goldmine of data they are sitting on.

Challenges

The McKinsey report highlights that regulation will be a challenge. This was well demonstrated recently by the poor implementation of new cookie legislation. Data security is another major issue. Data breeches at Sony and two email service providers have been hard lessons in the importance of security of data.

For high data intensity sectors such as financial services, a nagging question I have, is will big data be like IT was in 1990’s - a zero sum game? On the advice of leading consultants the UK’s leading financial institutions simultaneously embarked on a mission to make IT a strategic advantage over their rivals. Except for a few exceptions like First Direct and Capital One that strategic advantage did not materialize.

Wikipedia tellingly defines big data “as datasets that grow so large that they become awkward to work with using on-hand database management tools. Difficulties include capture, storage, search, sharing, analytics, and visualizing”. Good data access tools are clearly important. As a wise colleague once explained to me “you need the answer to your question while you still remember why you asked it”.

The report has a good description of the various data analysis techniques and data management technology. However the report was clearly written by consultants rather than practitioners so it misses some fundamental data matters. While the report quantifies the gap in analytical skills in numbers of people it does not really describe what those skills are. The whole area of skills required for good data management is only now emerging. For example the IDM’s new Award in Data Management is the first professional qualification of its kind in the UK.

Sadly the impression is often given that all you need is the right bit of technology, but people with the right skills is crucial. Recently I read a piece about Groupon where the spokesman had the job title of Chief Data Officer which I thought was quite telling about Groupon’s view of the importance of data. Time and time again research finds that for data mining around 65% of the challenge is data preparation rather than using the various analysis techniques or technologies. This has to be a human decision, how many outliers to drop etc. We’re all information workers now.