What is happening to log files? The Internet of Things, Big Data, Analytics, Security, Visualization – OH MY!

Over the past year, enterprise IT has had more than a few things emerge to frustrate and challenge it. High on the list has to be limited budget growth in the face of increasing demand for and expectations of new services. In addition, there has been an explosion in the list of technologies and concerns that appear to be particularly intended to complicate the task of maintaining smooth running operations and service delivery.

Whether it is security, Big Data, analytics, Cloud, BYOD, data center consolidation, or infrastructure refresh – IT infrastructure and operations are changing, expanding, becoming smarter and, definitely increasingly more chatty. The amount of data generated from operating and maintaining the infrastructure to run workloads and deliver services continues to increase at an accelerating pace. The successful delivery of IT-dependent services requires data to be properly correlated, analyzed and the results presented in a clear, concise and rapidly consumable manner.

The Internet of Things refers to the proliferation of smart devices that connect to, communicate over and exchange data across the internet. It is rapidly becoming the Internet of Everything [1] as the number and variety of networked devices and services continues to explode. In fact, it is growing at a pace that challenges the capabilities and capacities of existing infrastructure to create, support and maintain effective, reliable services. The lagging pace of infrastructure evolution both complicates and drives innovation in the how, what and format of data collection, normalization, analysis and presentation.

Monitoring, managing and controlling the devices and services involve the creation, collection and consumption of data. Big Data barely describes the volume of data and information that must be consumed and analyzed to provide information and knowledge for management and control. Much of which ends up in log files.

Whether residing in log files or consumed as data services, it must be collected, filtered, integrated and analyzed more quickly to yield easily consumable, actionable information to drive corrective or ameliorative action. Data analysis and modeling, even sophisticated analysis has been around and used for centuries – but it has only been more recently that a growing community of non-experts have had the ability to access and use very sophisticated data manipulation and processing techniques.

A continuing stream of stories call attention to the risk of exposure and malicious access to the increasing amount of data, both personal and business, private and public that is collected, exchanged and accessible on today’s network. Such stories have little apparent effect on the oftentimes reckless willingness of consumers and customers to neglect efforts to protect the security and assure the integrity of data and information they all too casually and willingly provide, exchange and store.

Today’s market and political environments are unforgiving and woefully unsecured. It isn’t only malicious attacks that result in access to data and information that should be both private and well-protected. Only the extremely foolish or incurably reckless will fail to make a proactive investment necessary to secure and protect the integrity and privacy of business, enterprise, consumer and customer data. Recent events and actions are driving IT and business communities to move toward a greater focus and sensitivity to security issues.

The demand is escalating for improvement in the ability to communicate complex and critical information quickly and accurately. Increasingly sophisticated consumers must absorb and understand the significance and criticality of information to promptly and appropriately respond. Advanced analytics and manipulation smooth the analysis of data and information from multiple sources to obtain detailed information and insight as a result. There are applications that can combine data from multiple sources [2] into a single report and even send the data itself to a smartphone or tablet. Visualization is recognized and, with increasing frequency used as the fastest, most effective path to understanding what is happening and what must be done.

So, what does this mean for us? The widespread availability of data from multiple, disparate sources in the enterprise greatly expand what is available for analysis. It enhances the role, impact and visibility of analyst and IT as they directly contribute to enterprise success. Benefiting from this opportunity requires IT staff to proactively move to expand the scope of their analysis as they work more closely with partners in enterprise operations. Perceptive providers of analysis tools and solutions are working hard to include extended capabilities and functions that make this task easier, more effective and powerful.

Finally, there remains the need for a user interface specifically designed to easily manipulate multiple documents and data sets simultaneously by using a touch screen without a keyboard. The fast acceptance and increasing popularity of tablets, phablets and smartphones have alerted vendors to the inadequacy of existing interfaces. The forces described above along with competitive market pressures are driving interest and activity to deliver a new generation of user interfaces specifically designed for creating working documents for these devices. Such an interface will allow users to advance far beyond today’s content-only consumption patterns. Developing the new interface means rethinking office productivity applications completely – something nobody has really done since Xerox PARC designed its Star Office system. Now that is something to look forward to.

[1] An apparently endlessly growing list of internet connected ‘things’ that started with computers and has been adding networked devices ever since to now include monitoring devices (medical, automobile, equipment, buildings, home, etc.), financial transaction services, security, communication formats that include voice, analog, digital, video, etc., etc..

[2] For example – DB2, Hive/Apache Hadoop, Teradata, MySQL, Amazon Redshift, PostgreSQL, Microsoft SQL and SAP.