By 2018, 50,000 gigabytes of data will be created per second. A significant amount of that data will be stored in corporate server farms. A report from IDG found that “[m]anaging unstructured data is growing as a challenge – rising from 31 percent in 2015 to 45 percent in 2016.” IDC’s Digital Universe report found that “the amount of data stored in the world’s IT systems” doubles every two years. On one hand, many companies are eager to start confronting the challenge of big data. But according to an IDG Enterprise report, 90% of the companies surveyed reported running into major problems when implementing or developing their big data initiatives. No wonder an October 2016 report from Gartner found that most companies who attempted to implement a big data project were mostly stuck in the pilot stage. The challenge of what do to with big data is daunting for many companies. However, machine learning has an important role to play in “solving” big data.
According to the Economist, only 5% of all data created is structured, meaning that “it comes in a standard format of words or numbers that can be read by computers”. The rest is unstructured and requires significant amounts of processing in order to be useful. That is, the data needs to be organized in such a way that it facilitates analysis by machine learning models in order to harvest insights. During a recent panel held by machine learning consultants Digamma.ai on the future of AI, founder Prateek Joshi explained “the way you have to pre-process the data [for machine learning initiatives] is so different for each use case that a lot of data scientists spend 80% of their time considering how to convert what they have into something usable. The last 20% is building…models.”
Machine learning consultants Digamma.ai are amongst a group of firms including Lexalytics and Crimson Hexagon that offer machine-learning-as-a-service (MLaaS). They play a key role in unleashing the significant amount of value hidden in unstructured big data. A Crimson Hexagon report states that “the value of unstructured data comes from the patterns and the meanings that can be derived from it; this includes identifying issues, market trends, or overall customer sentiment towards a brand.” Machine learning approaches such as Natural Language Processing (“NLP”) allow companies to “analyze multiple variables simultaneously” and discover how these variables form patterns that can lead to deeper insights.
Sentiment analysis is one of the clearest examples of technology that lies at the intersection of big data and machine learning. It has been shown to delivers serious, valuable insights for the firms who employ it. Sentiment analyses are often the building blocks of machine learning-driven recommendation systems, such as the one created by Digamma.ai for Silicon Valley-based Social Inertia. The system was integrated into an app called ActOn that allows users to choose a cause they care about and actively participate in it. Using techniques such as textual content extraction and topic extraction, the system is able to deliver content-based recommendations to its users that suggest causes that they are likely to support.
To truly understand the business case behind sentiment analysis, look no further than the online streaming giant Netflix. A recent article by Datamation illustrated the incredible value of Netflix’s sentiment-driven recommendation system:
“Netflix saves $1 billion a year by using AI to recommend videos customized to each viewer. Netflix discovered that viewers are 4-5 more times more likely to click on a video that appears on their Netflix recommendations screen. Netflix uses their AI system to track individual watching patterns, then recommends less well-known videos to the viewer. This system optimizes Netflix’ $6 billion a year content spend, since most of these videos are less expensive for Netflix to buy than blockbuster premium content.”
It’s no wonder that “solving” the big data problem is crucial for firms who wish to stay competitive. Luckily, there is a clear path forward: introducing and integrating machine learning into your business model.
With over a decade of AI experience, Digamma.ai’s team are your trusted machine learning consultants, partners, and engineers.