Rise Of The Machines – Why Big Data Is Getting Bigger Every Day

Hundreds of terabytes. That’s the amount of Big Data being generated by multi-national corporations, every day. While terrifying for some, others see it as an opportunity to fundamentally transform how their organisations operate and make decisions.

According to Gartner, Big Data is forecast to drive $34 billion of IT spending in 2013. But what most people think of when they hear the term ‘Big Data’ almost certainly isn’t what I’m talking about. What many organisations still regard as Big Data – unstructured information contained in emails, electronic documents, social media interactions etc – is just a thin layer in the vast strata of data available to them.

In our mobile, time-precious world, the new frontier in Big Data analytics is the real-time analysis of machine data. Every interaction with a ‘machine’ – whether it’s a website, mobile device, application server, corporate network, sensor or electronic tag, and whether it’s automatically generated or a manual transaction – leaves a trail and a record. And it’s within this layer of data that things really start to get interesting.

Correlating that machine data – data at rest, dark data in sensors and human unstructured data – could help you to see opportunities and threats earlier than the competition. In other words, competing on timeliness, by seeing patterns emerging from the data locked inside the machines that drive your website, protect your property, create your products, and service your customers. Machine data is now the fastest growing element of Big Data and is set to represent 90% of ALL data.

The ability to analyse and derive insight from all this machine data is where the big prize is in Big Data, but up until now, its value has been largely ignored by businesses. This is a major oversight, as the information generated by machines can paint a richer and far more detailed picture of the operational health of an organisation than any other measure.

But machine data doesn’t just apply to large IT systems – anytime someone switches on a phone, turns on a laptop or moves a mouse, machine data is generated. Commercial aircrafts, for example, can create enough machine data during a flight to fill a mid-sized data centre.

Imagine what an airline could do if it were to mine this data. They’d be able to analyse wind shear, engine efficiency, power output, fuel consumption and a whole host of other factors that would enable them to streamline and enhance the operational efficiencies of the aircraft and save money. Apply this principle to a multi-national corporation and the potential is very exciting.

However, some organisations are waking up to the value of machine data, integrating it into the Big Data mix to get a fuller, more holistic overview of how their company works. And this goes way beyond traditional notions of ‘business intelligence’, looking in the rear-view mirror of historical data to try to understand what you should do next: this is about ‘operational intelligence,’ making decisions based on a constantly updated real-time snapshot of the organisation.

Machine data is changing the Big Data landscape irrevocably, becoming increasingly pervasive as technology underpins everything we do. For organisations worldwide, it opens up the potential to genuinely revolutionise the way that they work – the challenge now is to manage and control their machine data without becoming overwhelmed by it.

James Murray, VP EMEA, Splunk