Big data: big opportunities, big challenges
by Mark Patron, CEO of RedEye International.
23 Aug 2011: Big data is exploding. According to a new report by the McKinsey Global Institute, "Big data: the next frontier for innovation, competition and productivity" as more and more information is collected and stored daily we need to prepare. If not we're going to be completely overwhelmed. The report states that by 2018 there will be a shortage in the US of 140,000 to 190,000 people with the deep analytical skills needed and with 70% of the world’s data stored in Europe and North America we will probably see a similar shortfall in Europe.
The numbers are quite extraordinary.
- Over 5 billion mobiles phones in use today.
- $600 buys you a disc drive that can store all the world’s music.
- 40% growth in data each year compared with 5% more spend on IT.
- 90% of the data in the World today has been created in the last two years.
- Thank goodness for Moore’s Law which states that for the same money computing power doubles every two years.
The McKinsey report makes some big claims for big dat. By using big data for better segmenting and targeting, plus offering other benefits, retailers can increase operating margins by 60%. Tesco and Amazon are held up as leaders in capitalizing on big data. At one point 30% of Amazon sales were reported to be due to its product recommendation engine.
I believe some of the report’s potential efficiency gains will not materialise. For example a €250 billion efficiency saving in European public administration and $300 billion in US healthcare seems like a stretch. At the very least the benefits are likely to be hidden.
Not surprisingly the biggest beneficiaries will be information businesses such as internet companies that can collect huge amounts of online behavioural data. Interestingly McKinsey graded these information businesses as one of the lowest in terms of a data- driven mind-set and this may well be their greatest bottleneck in capitalizing on data. Internet companies are only just waking up to the goldmine of data they are sitting on.
The McKinsey report highlights that regulation will be a challenge. This was well demonstrated recently by the poor implementation of new cookie legislation. Data security is another major issue. Data breeches at Sony and two email service providers have been hard lessons in the importance of security of data.
For high data intensity sectors such as financial services a nagging question I have is will big data be like IT was in 1990’s - a zero sum game? On the advice of leading consultants the UK’s leading financial institutions simultaneously embarked on a mission to make IT a strategic advantage over their rivals. Except for a few exceptions like First Direct and Capital One that strategic advantage did not materialize.
Wikipedia tellingly defines big data “as datasets that grow so large that they become awkward to work with using on-hand database management tools. Difficulties include capture, storage, search, sharing, analytics, and visualizing”. Good data access tools are clearly important. As a wise colleague once explained to me “you need the answer to your question while you still remember why you asked it”.
The report has a good description of the various data analysis techniques and data management technology. However the report was clearly written by consultants rather than practitioners so it misses some fundamental data matters. While the report quantifies the gap in analytical skills in numbers of people it does not really describe what those skills are. The whole area of skills required for good data management is only now emerging. For example the IDM’s new Award in Data Management is the first professional qualification of its kind in the UK.
Sadly the impression is often given that all you need is the right bit of technology, but people with the right skills is crucial. Recently I read a piece about Groupon where the spokesman had the job title of Chief Data Officer which I thought was quite telling about Groupon’s view of the importance of data. Time and time again research finds that for data mining around 65% of the challenge is data preparation rather than using the various analysis techniques or technologies. This has to a human decision, how many outliers to drop etc. We’re all information workers now.
14 June 2013: Cloud-based marketing software platform provider Marketo has announced the availability of the Marketo Customer Engagement engine, which it claims is the first marketing solution that intelligently and automatically manages the timing and distribution of the right content, to the right person at the right time.
13 Jun 2013: Kognitio and Celebrus Technologies marke the anniversary of a strategic collaboration that provides one of the largest global manufacturers in the world with an innovative solution for digital marketing analytics.
12 Jun 2013: The Institute of Direct and Digital Marketing has revealed that student teams from the University of Northumbria and University of Central Lancashire have made it to the top three in the prestigious IDM Student Marketing Competition 2013
7 Jun 2013: Salesforce.com, the world’s biggest CRM platform, has announced an agreement to acquire ExactTarget, the leading cloud marketing platform for around $2.5 bn.
7 Jun 2013: Whisbi, the lead management technology provider, has launches a new product designed to monetise Facebook.
6 Jun 2013: Database Marketing Award-winning customer engagement agency Indicia has expanded with the opening of a new London office.
5 Jun 2013: The London School of Marketing has announced a formal partnership with the Digital Marketing Institute.
1 Jun 2013: Experian insight reveals the location and characteristics of the UK’s growing number of single person households
22 May 2013: – D&B, the world's leading source of commercial information and insight on businesses, had made D&B360 available for Oracle CRM On Demand, Oracle's Siebel CRM, Microsoft Dynamics CRM and SAP CRM.
20 May 2013: In the third of our special series of articles profiling the database marketing industry, we catch up with Hopewiser, Alchemetrics, Callcredit and greenstone data solutions.
22 May 2013: Insurance provider Ageas has appointed Mark Hanson as Marketing Director for its two over 50s insurance brands, RIAS and Castle Cover.
20 May 2013: Comet Global Consulting, the customer interaction expert, has secured an additional two-year deal with Sky, to further develop and support the company’s customer relationship management and customer interaction systems and operations.
21 May 2013: — Experian has launched a new online credit risk and marketing portal putting a wealth of vital business information at the fingertips of UK SMEs.
8 May 2013: Hector Vass, former Systems Architect in the Customer Lifecycle and Optimisation team at the Lloyds Banking Group, has joined marketing analytics agency Metametrics as Chief Technology Officer.
8 May 2013: The Institute of Fundraising’s Special Interest Group for Insight in Fundraising has announced the shortlist of finalists for the Insight in Fundraising Awards 2013.
8 May 2013: ghd, the UK hair-straightening brand, has seen conversion rates soar as a result of improving the shopping experience on mobile devices using data gathered through Celebrus Technologies’ software.