Log Management in virtualized environments
Back in the early/mid-90s I was in charge of the global network for a software company. We had a single connection to the Internet and had set up an old Sun box as the gatekeeper between our internal network and the ‘net. My “log management” process consisted of keeping a terminal window open on my desktop where I streamed the Sun’s system logs (or “tailed the syslog”) in real time. Since we were using hardcoded IP addresses for the internal desktops, I could tell, just by looking at the log information, which person or device, inside the company, was doing what out on the Internet. If someone outside the company was performing a ping sweep, I saw the evidence in the log file and could respond immediately. This system worked fine for a couple of months. Then we installed a firewall, and a new mail server, and distribution servers in the DMZ, and, well, you get the idea. There was more log information than a single human could parse, not to mention the fact that while I worked a 50 hour week, the log files were on a 168 hour/week schedule.
While my example may seem almost laughably archaic to many, we’re seeing a similar data overload phenomenon occurring in today’s data centers and network operations centers (NOCs). Log management systems that were installed a few years ago to handle 100 servers and applications that can’t scale to today’s needs. What started out as a few gigabytes of log information per week is now a terabyte a day. One reason for the log information explosion is that as companies become comfortable with the technology, they expand the log monitoring coverage scope. Another significant driving factor: virtualization and the advent of the virtualized data center.
Virtualization brings new challenges to network monitoring and log management. Virtualization enables administrators and users to install multiple unique server instances on a single hardware component. The result is a marked increase in server and application installs and a concurrent increase in server and application log data. In addition to more log information, virtualization presents a few additional challenges as well.
Inter-VM traffic refers to data moving between virtual machines running on the same physical machine under a single hypervisor. Because the traffic isn’t moving off the physical device, it will not be seen by monitoring solutions that use physical network based monitoring points like span or mirror ports. Monitoring solutions that are installed directly on hosts will log the devices information, but if there is just one agent on the host and it is not integrated with the hypervisor itself inter-VM data transfer could still be missed. An alternative is to install agents on each virtual machine. Keep in mind, however, that this could impact corporate use licenses by increasing the total number of agent installs. And for companies that want an entirely agent-less solution, this alternative won’t work. Some additional alternatives for inter-VM traffic monitoring are presented below.
What else changes in the virtualized data environment? Well, zone based policy enforcement might. Consider databases. These are often repositories of sensitive information and only approved for install in protected network zones. Virtualization allows organizations to move servers and application quickly between locations and zones using V-motion functionality. The problem comes in when V-motion is used to move a service or server into a zone or location that has an incompatible protection policy. Think of a database of healthcare information that if V-motioned from a high sensitivity zone into a DMZ. Log management can help here by alerting administrators when a system or service is being moved to a zone with a different policy control level. In order to do this, the log management solution must have access to V-motion activity information. VMWare provides migration audit trail information which can be fed into an organizations log management console.
So how do we perform comprehensive log management in virtualized environments? First, it’s critical that the inter-VM “blind-spot” is removed. One option has already been discussed – installing host-based log management agents on every virtual machine instance. If that’s not a good fit for your company consider purchasing a log management or security information and event management solution that has hypervisor-aware agents that can monitor inter-VM traffic. VMWare has a partner program, VMSafe™, which provides application programming interfaces (APIs) so vendor partner solutions can monitor virtual machine memory pages, network traffic passing through the hypervisor, and activity on the virtual machines.
To keep a handle on mushrooming installs, track and report all new server, service and application instances to a central operations or log management console. In cases where unapproved services are being brought-online this can be particularly helpful. For example, if a mail server install is detected this could indicate the installation of a server that hasn’t had core services turned off – or worse – it could be an indication of an e-mail scam or bot-net.
If your log management provider isn’t VM-aware, check to see if any of your firewall or IPS vendors are. If so, the virtual-aware monitoring information from the firewall or IPS sensor on the hypervisor can be passed through to your log management solution in the same way that physical span port information is aggregated. Regardless of how the inter-VM traffic is (on host agent, inter-VM log management, inter-VM firewall/IPS or other sensor) collected, it’s imperative that the information is brought into the existing log management solution; otherwise, you’ll have a significant blind-spot in your log management solution.
Finally, don’t forget to review existing rules and update or amend them as needed for the virtual environment. For example, have rules that manage virtual machine migration audit trails been added? Are new rules required inter-VM traffic monitory for policy or compliance mandates?
Virtualization has introduced great flexibility into networks and data centers. But with this flexibility comes additional log data new monitoring challenges. To make sure you aren’t missing out on any critical information, implement VM-aware monitoring solutions that work with your existing log management installation and update rules and policies.
Related content: Managing the virtualized enterprise: New technologies, new challenges
Because of its many benefits, employing virtual technology is an apparent “no brainer” which explains why so many organizations are jumping on the bandwagon. This whitepaper examines the technology and management challenges that result from virtualization, and how EventTracker addresses them.
How CISO’s can leverage the internal audit process
Say the word auditor at any gathering of information security folks, and you can almost feel the hackles rise. Chief information security officers (CISOs) and internal auditors, by definition of their roles, are typically not the best of friends…Yet, the CISO’s traditional adversary can be an effective deputy.
Did you know? EventTracker provides a number of audit-friendly capabilities that can enhance your collaboration efforts such as over 2000 audit ready reports, automated-audit trail creation and more.
Lawsuit: Heartland knew data security standard was insufficient
Months before announcing the Heartland Payment Systems (HPY) data breach, company CEO Robert Carr told industry analysts that the Payment Card Industry Data Security Standard (PCI DSS) was an insufficient protective measure. This is the contention of a new master complaint filed in the class action suit against Heartland
Note: We have a different take – Read Steve Lafferty’s (Prism’s VP of Marketing) commentary titled, Lessons from the Heartland – What is the industry standard for security? Leave a comment, and tell us your thoughts.
Prism Microsystems named finalist in Government Security News annual homeland security awards
EventTracker recognized as a leader in the security incident and event management category
EventTracker officially in evaluation for Common Criteria EAL 2+
Internationally endorsed framework assures government agencies of EventTracker’s security functionality