Finding an Application of Analytics to ‘Big Data’ in your own backyard

Back in January, I said that the use of sophisticated analytics as a business and competitive tool would become widespread. Since then, the number of articles, blogs and announcements relating to analytics has increased dramatically:  an internet search for the term ‘Business Analytics’ using Bing yields over 47 million hits. Smart Analytics (an IBM term) shrinks that number to approximately 12.3 million hits. If we change the search term to ‘Applied Analytics,’ the number decreases to a little less than 7 million hits.

Analytics has certainly captured the attention of government [1], business [2], the industry press and management. The question, though, is whether it’s being put to use in the trenches. Are CIOs and IT staff searching out, acquiring and applying these tools to address their problems? After all, it isn’t enough to have access to analytic tools and services; you have to understand how to use and apply them to real problems. How many users are actually prepared to move forward into the big world of applied analytics to solve pressing business problems? Where does one go to begin to use these tools? Are analytics only of use for working with ‘Big Data’? What is ‘Big Data’?

There are a lot of questions there; too many to exhaustively address in this blog, and some that can’t be resolved without some detailed research. We’ll provide answers based on our own experiences in interacting with clients, research and informed opinion.

First, let’s agree on a few definitions. It often seems ‘Big Data’ is defined in as many ways as there are vendors offering solutions.  For our purpose, we’ll use a fairly loose definition that is based on the Volume, Velocity, Variety and Veracity of the data. Big Data comes in large enough volumes that it requires special software and hardware to process in a reasonable time (terabytes, petabytes and beyond!). At least some of the data, and perhaps all of it is ‘in motion’, coming in and moving out and changing very quickly. The source and form of the data is highly variable; it comes in different varieties, data types, structures and formats – audio, visual, media, structured unstructured, different sources, etc. The fourth characteristic is the question of data veracity i.e. uncertainty over the accuracy of data including questions of confidence in the source.

Second, analytics can cover a lot of ground, from manual number crunching to giant, specialty processors designed specifically to do real-time analysis in exploration for oil deposits. What we’re interested in, however, is the application of software-based analytics to collect, analyze and report on data collected in our IT and business environment.

Big Data and analytics are frequently paired; however, the relationship is far from exclusive. Analytics can be profitably applied to smaller data sets. The benefit comes from using the analytics to gain actionable information and insight across multiple business functions. This can be application of an investment analysis program to determine the potential profitability of a product development project by tracking development, packaging, marketing, delivery costs versus forecasts of revenue expected from sales and support under alternative market growth patterns. But, it can also be correlating event log data on application access, network traffic, file access and data routing of confidential files and initiating action to prevent those files from being published around the world.

It can also take the form of a Manager of Software Development recognizing his department has the potential to directly impact revenue. He has an idea that a regularly used, revenue-generating asset is not being used in a way that optimizes its potential for revenue generation. He’s convinced that if it were scheduled and managed more effectively this could be done. He knows the data to prove this is collected in logs and data files, but he needs to pull it all together. With some work, he can bootstrap a basic analysis from available tools to make his case to management for more detailed, integrated tools.

Those are typical examples of analytics in action today, and it is being done within the budgets of mid- to large-scale enterprises and without mathematical wizards on the payroll. Our discussions and experiences uncovered a lot more talk about Big Data and Analytics going on in the executive suite and among business and IT staff than previously. There is lot more planning and speculating about use going on among potential users and in enterprises and business of all sizes. But all too often, this isn’t translating into action.

The path to more effective use and application of analytics begins by using what you have today to its maximum advantage. Most businesses have a log management solution with at least some analytic capabilities. Start using the analytics if you aren’t already. Push your boundaries and use your imagination to identify new ways to use the analytics. Look at adding new data that can be correlated or plotted together to uncover new relationships. Extend the data view to adjacent, interacting and interdependent functions. The Software Dev manager spoken of earlier looked into the relationship of revenue generated with usage and scheduling to identify potentially profitable idle time. Look for a potential application and develop the case by using what you have to get to where you want to be.

Don’t be afraid to see what vendors are doing and offering to promote their own analytic solutions. You can get ideas about where to look and what problems are being solved by understanding what others have done. Look at vendor announcements to see how analytics are being promoted, then look for the opportunity in your own environment.

[1] Big Data Big Deal

[2] Big Data The Next Frontier for Innovation