We’re fast approaching a time when it becomes possible to connect everyone to everyone.    Affordable, ubiquitous, on-when-you-want compute power, communications capability and storage will soon be in everyone’s hands. Literally. And not just in their hands. On their wrists. In front of their eyes. As part of their clothes. Everywhere. In everything.

We’re fast approaching a time when everything’s connected. The Internet of Everything. This goes well beyond people and devices and documents and machines. It’s about a time when every car park in your town tweets the number of spaces available. A time when you can subscribe to the article you’re reading, and be alerted if there are changes or additions.

A time when tweets start to tweet. When a tweet tells you “Hey it’s been three hours and you still haven’t read me, even though four of your friends have asked you to”.

When everything is a node on the network, everything can share information about its state. Everyone and everything will be able to “publish” and to “subscribe”.

That’s a lot of noise from which to extract any sort of signal. So, as Clay Shirky reminds us, it becomes important to have the right filters.

So our friends, the people who know us well, will operate as filters, ensuring that we get to see and hear what they know to be important or valuable to us. Social becomes a filter.

Our mobile devices and cloud services already come with ways to manage our notifications. We decide when, where and how we receive messages and alerts; in that sense, notification centres are filtration points.

Many of these alerts and messages are meaningful only when they are presented with context. The metadata of time, location, source, identity of person, identity of device all become important, and we can use that metadata to filter further. Additionally, the status messages themselves contain valuable information, which can be compared against predetermined thresholds.

This process of looking at the data and metadata, comparing them against lists and thresholds before agreeing disposition, this is what tools like IFTTT set out to do. We are going to see much more of this, as notification exchanges and clearinghouses emerge around platforms. People will build services that use platform APIs to test levels and contexts via the data, metadata and thresholds. The ecosystems that emerge around the platforms will also form part of the filtering process.

The social networks we belong to, the communities represented, the devices used, the platforms and ecosystems they form part of, each of these is a revolution in its own right. And all these are necessary but not sufficient to deal with yet another revolution, the explosion of Big Data.

All revolutions. All necessary. And all not sufficient.

The most important revolution taking place is about the customer, in terms of transfer of power, level of participation and proactive nature. And this is a revolution of trust, and about the framework of trust that is identity.

Soon everything will be connected, everyone will be connected. Soon everything will be a node on the network, able to publish, able to subscribe.

We can now “quantify” ourselves: the services and devices available now allow us to measure much about ourselves as individuals. We are able to measure our health and well being in ways we could never do before.

It’s not just about quantified selves, we can apply the same techniques at work, the quantified firm. We’re able to see correlations and patterns we just could not see earlier, and to do something with those insights. Insights about our products and processes, and more importantly, insights about our customers, what they want, what they like, what they don’t like.

It goes beyond that. We can gather collective intelligence, start seeing patterns at much larger levels, in communities, geographies, even globally. Every one of us becomes a sensor and a participant in that process, collecting information both actively as well as passively, and making that information available for aggregation and analysis.

All this is brought about by a series of revolutions, revolutions that help us separate the signal from the noise, filter the facts from the firehose. Revolutions in how we connect, who we connect with, when and where we connect, what we connect to. Revolutions in how we connect.

We may have an Internet of everything, changing all that we can see and hear and experience.  But some things don’t change. Everything we do is about people; about relationships between people. And about the trust that binds us together.

The Internet of everything is not a technological revolution. It’s a trust revolution.

But first, a disclaimer: although the official partner city was Kuala Lumpur, it would be more accurate to say that the partner city was Kuala Lumpur and her surrounding areas (to be precise we had 5 events in Kuala Lumpur, 2 events in Cyberjaya, and 1 event each in Petaling Jaya and Subang Jaya – for a total of 9 events). Maybe that was bending the rules a bit, but that’s just how things are here – in our defence, even Kuala Lumpur International Airport isn’t technically in Kuala Lumpur. So there.

Setting the stage

Since we were so eager to go, we kicked off a bit early with TRANSIT: Distillation on April 19th (i.e. the Friday just before Big Data Week 2013 officially began). Hosted by VLT, and co-sponsored by Experian and iTrain, this was our opening party, which we spiced up with a data visualisation showcase. This event had people coming and going, but at it’s peak it had a crowd of about 80 people: a good mix of creative professionals, software engineers, entrepreneurs, investors, and more.

Off to a good start, the very next morning the local Python User Group organised an all-day data analytics workshop, where 30 hackers got together to work on some sample challenges from Kaggle, using Python of course.

In recognition of the clear relevance of Big Data to their industry, Telekom Malaysia Research & Development hosted a meetup on Tuesday afternoon catering primarily to their staff but open to the public. The crowd of 70 telco professionals benefitted from an entry-level introduction to Big Data from the perspective of 4 major industry players.

We did not intentionally plan a single “signature event” as part of Big Data Week, but as we saw the registrations come in for Big Data: Strategy, Practice, and Research it became clear that this was going to be slightly bigger than the others, so MDec generously upped their already major sponsorship for this whole-day event. Ultimately on Wednesday we checked-in 222 attendees. The participants comprised a mix of engineers, executives, managers, and academics from across a number of different verticals including oil & gas, telecommunications, banking, publishing, marketing, and many more. It is our hope that this event significantly expanded awareness of Big Data opportunities to mainstream corporate Malaysia, as well as government policy-makers.

The Performance Management Delivery Unit (PEMANDU), an agency under the Prime Ministers Department, recognises the transformative potential of Big Data to Malaysia’s economic development. To contribute towards awareness-building, they put together the following clip featuring a couple of our panelists (these interviews were recorded on-site at our “Big Data: Strategy, Practice, and Research” event, therefore this clip is another direct contribution of Big Data Week):

Later that Wednesday, Netapp hosted a highly technical Storage Networking Industry Association meetup, which included high-quality presentations on NFSv4, object storage, and data protection. Despite a manic thunderstorm that flooded parts of KL and even took out parts of the cellular phone network, this event managed to bring together about 10 experienced infrastructure professionals from various vendors, for a highly interactive and frank discussion on the realities of enterprise technology use vs. the ideals of Big Data.

nVidia’s GPU Technology Workshop was already scheduled to happen on Thursday – but when Big Data Week rolled into town, they decided to add a Big Data track. This was fortuitous, as of the 3 concurrent tracks (the others being Computational Finance and Media & Entertainment), the Big Data track proved to be the most popular – yet more evidence that this technology concept has captured mainstream interest in this market. The total combined crowd of about 180 benefitted from technical talks from academics and industry specialists.

To many, Big Data and The Cloud are practically inseparable. AWS presented their take on it on Thursday afternoon, with a seminar on the use of some of their platform offerings to accelerate development of data solutions. Their talk was followed by over an hour of interactive discussion amongst the 25 or so participants, with an AWS architect providing expert advice.

Thursday evening saw the first ever NoSQL Asia event, sponsored by Praxis BT. NoSQL Asia is a new movement that’s working towards development of a regional conference, and this event was essentially a “beta test” – and a superbly successful one at that, proving that this region is ready for a dedicated NoSQL conference. This event brought together a mostly deeply technical audience of about 60 people to discuss in breadth and some depth “the technology behind Big Data Week”, as they appropriately billed it.

Data impacts Malaysia

Malaysia’s final contribution to Big Data Week was Friday afternoon’s panel discussion on Data Journalism. The timing of this event was especially relevant: Malaysia was just a week away from what turned out to be her closest and most bitterly fought General Election in history, the aftermath of which has seen a mushrooming of people analysing and reporting on what few reliable data sources are available.

Taylor’s University hosted this event of about 100 participants – a crowd which consisted primarily of journalism and communication students and practitioners, though there were plenty of techs and others in attendance as well. Participants were treated to a great overview of what data journalism is, what it can do that traditional forms of journalism cannot, as well as a frank and open discussion of what’s missing in the Malaysian context. As an added bonus, participants got a fascinating live glimpse of what a sophisticated DoS attack on a major local news portal looks like, real-time on Google Analytics.

Going forward

Most people seem satisfied with the scale of what we achieved – in fact more events might have been too much – but it’s encouraging that we actually came close to holding 3 additional events: “Big Data in Healthcare & Life Sciences”, “Big Data in Banking & Finance”, and a hackathon to go with the Big Data Week Challenge. These events didn’t happen only because of issues around logistics, and they’re likely to happen in some other form in the near future.

Another encouraging sign is subtly communicated in all the above photos. Close inspection will reveal that Big Data Week in Malaysia attracted all flavours of participants across the various events – from hackers and creatives, to decision makers in suits. This bodes well for future initiatives, as success will require investment, business sponsors, and people with the skills to deliver.

It may be true that Malaysia currently lags behind in Big Data capabilities, but as we’ve demonstrated through our Big Data Week contribution, the commitment to closing the gap is significant. After all, it takes significant commitment, material investment, genuine expertise, willingness to collaborate, and high effort from various organisations, partner groups, and individuals working above and beyond the call of duty to make a festival like this happen. Through our combined efforts, we’ve proven that this stuff matters to us, we know a little bit about it, and we’re going to keep learning.

If you ask me, history is repeating itself. I remember the birth of CRM and how the industry was split between IT companies and their solve-it-all multi-million pound databases, and the pointy heads claiming it was their domain. CRM turned out to be data driven marketing using IT as a tool, and eventually became the industry buzz-word of the ’90s.

With this in mind, it is important that we monitor how the conversation around Big Data evolves and monitor how it is impacting on our industry. There are, however, some cautionary steps that can be taken. Some organisations are rushing to buy costly machines to capture what is in many cases vast amounts of ephemeral data without giving thought to how it will be of value to them in the long run. Others are burying their heads in the sand as they find Big Data to be too overwhelming and are still stuck with figuring out whether or not Social Media is worth it (it is!).

One can throw enormous volumes of data into a mega machine costing a fortune and it will come out with all sorts of interesting figures, but are these details useful or (more importantly) even usable? Is a machine really better at understanding how human data can be used?

The most effective way to approach Big Data is to combine the entirety of a company’s data collateral and then decide where to look within the Social Media plane. The data is not thrown into some black box, but analysed by humans using IT tools and software. It’s the creative interpretation of the data which will do most to drive the bottom line.

Some marketers may struggle with the sheer volume of data, but it is important to have the relevant skill sets and understand the technology being used. Many brands may have the right data analysis software which can pool data from various touchpoints, but without having the personnel that can properly analyse the incoming data, the insight is wasted. Brands should look at how their teams can make the most of customer information and deliver comprehensive insight based on this. Furthermore, those brands that look most likely to succeed are those that can translate this personal data into tangible strategy and activity.

This is doable, scalable and most importantly actionable. But the most important question to ask an organisation before embarking on any ‘Big Data’ project is what they want to get out of it!

Essentially, despite the hype and fanatical strategies surrounding Big Data, it’s not difficult or confusing; it’s just about asking the right questions. 

Ultimately, those brands that can accurately analyse customer data and apply it to segmented marketing campaigns are more likely to deliver successful campaigns. Being able to correctly identify target audiences with data-driven insight and having the right team to analyse this information will give marketers the tools to produce campaigns that are not only cost effective, but achieve strong ROI.

The marketing industry certainly faces another challenging year and marketers will have to demonstrate a deep understanding of their customers to succeed. A fully-rounded picture of a customer is the key to delivering the required levels of understanding and big data offers many opportunities to build upon and enrich your view of the customer.

Big data seems to mean many things to many people. For example is big data only data greater than 1 petabyte in size? Or should big data be defined by the number of days it takes days to ingest to a data store? Or perhaps big data is only big if it needs to be physically moved on an external device? Of course the data movement issue is an issue in itself and can be a reflection on cost rather than data movement performance. Ultimately it seems that big data is subjective and only big in the eye of the beholder.

The panel also discussed the use and re-use of data analytical capabilities / tools, which were either open source or have been produced for other purposes. For some, especially those working in research, this could be an economically viable route to be able to either ‘dip in’ to big data or ‘attempt to harness the monster.’

Legacy data was also discussed, especially around how to deal with public data, how long should this be held for, when does data no longer serve a useful purpose and when can it be deleted. What is clear is that for most, legacy data (including jurisdictional issues, restrictions and requirements) are not well understood and we are all still learning. More fuel for Big Data Week next year perhaps?

Some companies claim to have big data collection and analysis in their DNA and have built a separate strategy around this. Others say they recognise that big data is part of a larger total data management function and are looking to incorporate relevant tools and practices throughout the business to manage big data as well as enterprise and discrete data. Yet the question remains – to what extent has the focus of the conversation moved from the IT department to the boardroom?

Talend recently conducted a big data adoption survey with IT data professionals in 231 businesses, covering business objectives, adoption challenges and benefits as well as technologies being used.

Is big data starting to make an impact on C-Level decisions and in the boardroom? The signs are positive. 41% of companies reported having a strategy for dealing with big data and 62% are achieving benefits, including business process optimisation and marketing and sales improvements.

Highlighting why the approach should be of as much interest to the executive as the data manager, two of the top three business drivers for big data were linked directly to the bottom line. 51% indicated revenue optimisation was a key driver and 48% new revenue generation, underlining companies’ drive to do more in-depth analysis to maximise wallet and market share.

Driving Competitive Edge

Ultimately, to be a source of competitive advantage for business, big data solutions need to be able to detect, create and apply new types of processing to existing or new data sets, creating new value – often through the creation of innovative business models, products or services. Today, to the obvious interest of the C-Level director, it is being used in a broad range of different ways to drive revenue generation and optimisation and ultimately business advantage.

Businesses are awash with complex data that has the potential to lead to better understanding of customer demand, behaviour and preferences. By using infinitely scalable open source big data solutions, organisations can efficiently manage and process this data and interpret it accurately to gain an invaluable insight into their customers. In addition, the fact that open source typically brings higher value is broadening its appeal to encompass smaller businesses rather than just large enterprises.

From laser-sharp targeted marketing, to fraud detection to trading optimisation — the examples of how all sizes and types of businesses are today leveraging big data to optimise revenues are endless. Some industries specialise in leveraging massive amounts of data: banks and credit card companies focus on analysing transactions to detect fraud and retail chains are fine-tuning their inventories based on historical trends.

In line with this, the number one business driver highlighted by 68% of respondents with a big data strategy was to increase the accuracy and depth of analytics or the ability to analyse current and historical data to make predictions. In other words, businesses today from the largest enterprises to the smallest SMEs are increasingly seeing the value of big data as a strategic corporate asset and as an approach that can drive competitive advantage.

Open source can also play a key role here in delivering this edge. Its scalability enables businesses to gain fast access to intelligence and therefore helps them to analyse it and use it to make strategic decisions more quickly.

The language of analytics typically appeals to C-Level executives and its potential use to drive productivity and business edge is a major selling point to the board. A recent survey conducted by technology research firm Economist Intelligence Unit found that top business executives want big data access for more of their employees in hope that more people using analytics tools will lead to a more productive organisation. 77% of global executives surveyed agreed that more employees should access big data while 66% said big data is ‘instrumental in seizing market opportunities’.

Yet, despite acknowledging these benefits, 47% of respondents said they don’t expect to increase investments in big data over the next three years, with 37% referencing financial constraints as a barrier to such investment.

Opening up the Technology

Fortunately, there are big data solutions available on the market today that can address these kinds of concerns. At Talend, we believe that open source technology provides the most effective way to address the large-scale problem that big data presents and get the job done faster and more accurately, at a fraction of the price of alternative solutions. Open source effectively ‘shrinks the technology cliff’ by letting any company adopt and deploy technology, regardless of budget and level of expertise.

Today, big data, supported by open source technology, is coming more and more into the mainstream. Growing numbers of organisations, and the CEOs and other C-Level directors that lead them, are realising that information can be the key differentiator that gives them an advantage over their rivals by enabling them to reduce customer churn, more effectively target new markets, tackle fraud, or simply make faster, better-informed business decisions.

According to Gartner, Big Data is forecast to drive $34 billion of IT spending in 2013. But what most people think of when they hear the term ‘Big Data’ almost certainly isn’t what I’m talking about. What many organisations still regard as Big Data – unstructured information contained in emails, electronic documents, social media interactions etc – is just a thin layer in the vast strata of data available to them.

In our mobile, time-precious world, the new frontier in Big Data analytics is the real-time analysis of machine data. Every interaction with a ‘machine’ – whether it’s a website, mobile device, application server, corporate network, sensor or electronic tag, and whether it’s automatically generated or a manual transaction – leaves a trail and a record. And it’s within this layer of data that things really start to get interesting.

Correlating that machine data – data at rest, dark data in sensors and human unstructured data – could help you to see opportunities and threats earlier than the competition. In other words, competing on timeliness, by seeing patterns emerging from the data locked inside the machines that drive your website, protect your property, create your products, and service your customers. Machine data is now the fastest growing element of Big Data and is set to represent 90% of ALL data.

The ability to analyse and derive insight from all this machine data is where the big prize is in Big Data, but up until now, its value has been largely ignored by businesses. This is a major oversight, as the information generated by machines can paint a richer and far more detailed picture of the operational health of an organisation than any other measure.

But machine data doesn’t just apply to large IT systems – anytime someone switches on a phone, turns on a laptop or moves a mouse, machine data is generated. Commercial aircrafts, for example, can create enough machine data during a flight to fill a mid-sized data centre.

Imagine what an airline could do if it were to mine this data. They’d be able to analyse wind shear, engine efficiency, power output, fuel consumption and a whole host of other factors that would enable them to streamline and enhance the operational efficiencies of the aircraft and save money. Apply this principle to a multi-national corporation and the potential is very exciting.

However, some organisations are waking up to the value of machine data, integrating it into the Big Data mix to get a fuller, more holistic overview of how their company works. And this goes way beyond traditional notions of ‘business intelligence’, looking in the rear-view mirror of historical data to try to understand what you should do next: this is about ‘operational intelligence,’ making decisions based on a constantly updated real-time snapshot of the organisation.

Machine data is changing the Big Data landscape irrevocably, becoming increasingly pervasive as technology underpins everything we do. For organisations worldwide, it opens up the potential to genuinely revolutionise the way that they work – the challenge now is to manage and control their machine data without becoming overwhelmed by it.

James Murray, VP EMEA, Splunk

Although this information is easy to store in these repositories, the challenge lies in getting this information out quickly and easily. Extracting value from the information held in these repositories is usually very hard because it wasn’t stored in any uniform way and wasn’t categorised accurately. This is particularly pertinent as unstructured data contains a myriad of untapped and potentially valuable insight. But, unstructured data left spread across disparate repositories, unmanaged, inaccessible and un-analysed is worthless and represents a cost to the business in terms of storage. The ability to extract meaning from data is where its true value lies.

Unstructured data by its very nature is uncategorised and lacks the metadata that allows for easy identification and organisation. This can potentially leave data in a state of chaos, exposing the business to legal and compliance risks – especially those in regulated industries. This can end up costing businesses vast sums of money as they attempt to find and analyse information on a purely reactive basis. As such, organising, managing, and analysing this chaotic data proactively is a more cost effective approach, and can provide businesses with regulatory peace of mind and valuable insight through the evaluation of unstructured data. If businesses rely on structured information alone, they are potentially overlooking very important evidence and missing out on key information spread across their disparate systems within, as well as, conversations taking place between their customers and the business beyond its four walls.

Businesses need solutions that can help employees effectively access this key information. Integral to this is the ability to identify, understand and categorise the information amongst the vast amounts of unorganised and unstructured data. However, manual identification and categorisation simply doesn’t scale; therefore automation needs to be incorporated into the process to categorise these sorts of volumes.

By using software that accurately categorises any unstructured data in context, through the use of supervised learning and automatic categorisation technology, organisations can identify the relationships between entities, such as people, titles, instances, dates and departments. This software also provides a broad overview of the kind of information that resides on a company’s systems. This technology avoids the need to build a team of people to go through the information that businesses have stored on their repositories, map it out and find the relationships between entities. Instead you can use the experts within your company to teach the technology to focus on what matters to your business.

Businesses must embrace technology that can help them identify, categorise and manage unstructured data. By doing so, they can take advantage of the data explosion, rather than hiding from what could end up to be a Big Data minefield.

I’m a believer in the potential of big data.  At Lavastorm Analytics our data analytics platform has allowed me to see the benefits first hand, such as an organization that pulled together 60 different data sources into a single application to identify and eliminate fraudulent activity and another financial services organization that analyzed terabytes of data daily to improve the effectiveness of their trade execution.  I expect its ability to improve customer relationships, change business models, and improve overall decision making will be borne out in large scale over the next 5 years.

But while big data has high potential, it would be a mistake to think that simply add a new data source to your mix would immediately reveal the secrets to the world’s longest lived mysteries.  I believe any skepticism at this point has to be routed in poor processes applied to big data, not the big data itself.  It isn’t obvious in every industry which data is the most valuable.  You “could” analyze thousands of data relationships, but not all will bear fruit and if your peers haven’t already done something you can just emulate, then we have to recognize that the first task is identifying which data and relationships are the most valuable.  That cries out for an exploration and discovery approach – one that allows you to initially cast a wide net at minimal cost so that you can simply identify a handful of the most likely candidates that are worthy of greater investment.  Selecting a low cost, low risk infrastructure or toolset to enable that exploration and discovery of those targets is a major key to success.

Analytics aside, without the management basics in place, it would never be possible for companies to use big data in a meaningful way. It’s clear that dealing with huge volumes of structured and unstructured data has introduced a new set of challenges for the IT department, most of which have been answered with cloud services.

We’re living in an economy which is increasingly based around service provision and from a security point of view throws up several concerns. Usually in reference to the geographically dispersed architecture of services provided and information stored, security implications are – I would argue – the biggest barriers to businesses getting the most from their data. But my main concern for organisations is identity and access management (IAM), or lack thereof.

Quite simply, insightful data can only be used effectively if it can be accessed by the right people, on the device that is most convenient to them. The challenge is managing access across a wide variety of demands from the business. What IT departments, in conjunction with the business, need to do is ensure that the right information can be accessed by the right individuals (be that customers, employees of varying levels, suppliers or partners) at any given time and location in order for them to interact with the business accordingly. Getting this wrong affects the ability for businesses to use valuable insight derived from big data. IAM can help organisations manage the level of access for different types of users across the business. A recent survey conducted across eight European countries by independent analyst house Quocirca found that the top IT management challenge eased by IAM is the enforcement and management of access policy. Those surveyed also cited that IAM improves the user experience by providing easy federated access to multiple applications. So, we can see that access is a key issue for IT management teams and that by allowing the appropriate level of access to data companies can enable their employees to increase their productivity and use the insight from big data to provide value back to the business.

The importance of using data to derive greater understanding and improve efficiency is obvious and there are plenty of experts and technologies which enable businesses of all size to do just that. But before big data can be processed, analysed or used, it should first be considered how it will be managed and, equally as important, accessed.

The role of Big Data around the world to gauge happiness is growing by leaps and bounds. Just two weeks ago, Michael Porter from the Harvard Business School launched the Social Progress Index which focuses on measurements associated with well-being. Apps like Happier and Moodscope let people measure and record their mood or well-being on an ongoing basis. What this trend points out is that the availability of new and emerging technologies that help reveal aspects of our character we haven’t studied before is apparently feeding a big need in us. We want to know what makes us happy and why.

It’s a pretty common question to ask, “what is the measure of a man (or woman)?” in the sense of defining their character by their actions. Now people can put that question to the test with these tools and trends to learn more about that answer for themselves. If you’d like to get a sense of your “Personal Happiness Index” score from H(app)athon, there’s a free survey you can take online or via your iPhone at www.happathon.com/survey. We’d welcome your help/collaboration in helping us define a crowdsourced vision of happiness and well-being.

But one way or the other – let me ask you: what is the measure of your life?  It’s worth keeping track of, and Big Data can help.

Submitted by John C. Havens, Founder of The H(app)athon Project and contributing writer for Mashable.