Five reasons for log apathy — and the antidote

Five Reasons for Log Apathy – and the Antidote

How many times have you heard people just don’t care about logs? That IT guys are selfish, stupid or lazy? That they would rather play with new toys than do serious work?
I argue that IT guys are amazing, smart and do care about the systems they curate, but native tools are such that log management is often like running into a brick wall — they encourage disengagement.

Here are five reasons for this perception and what can be done about them.

#1 Obscure descriptions: Ever see a raw log? A Cisco intrusion or a Windows failed object access attempt or a Solaris BSM record to mount a volume? Blech… it’s a description even the author would find hard to love. Not written to be easy to understand, rather its purpose is either debugging by the developer or meant to satisfy certification requirements. This is not apathy, it’s intentional exclusion.

To make this relevant, you need a relevant description which highlights the elements of value, enrichs the information (e.g., lookup an IP address or event id) and not just spew them in time sequence but present information in priority order of risk.

#2 Lack of access: What easier way to spur disengagement than by hiding the logs away in an obscure part of the file system, out of sight to any but the most determined; if they cannot see it, they won’t care about it.

The antidote is to centralize logging and throw up an easy to under display which presents relevant information – preferably risk ordered

#3 Unsexiness:  All the security stories are about wikileaks and credit card theft. Log review is considered dull/boring, it’s a rare occurrence to make it to the plot line of Hawaii Five-O .

Compare it to working out at the gym, it can be boring and there are 10 reasons why other things are more “fun” but it’s good for you and pays handsomely in the long run.

#4 Unsung Heroes: Who is the Big Man on your Campus? Odds are, it’s the guys who make money for the enterprise (think sales guys or CEOs).

Rarely is it the folks who keep the railroad running or god forbid, reduce cost or prevent incidents.

However, they are the wind beneath the wings of the enterprise. The organization that recognizes and values the guys who show up for work everyday and do their job without fuss/drama is much more likely to succeed. Heroes are the ones who make a voluntary effort over a long period of time to accomplish serious goals, not chosen ones with marks on their forehead, destined from birth to save the day.

#5 Forced Compliance: As long as management looks at regulatory compliance as unwarranted interference, it will be resented and IT is forced into checkbox mentality that benefits nobody.

It’s the old question “What comes first? Compliance (chicken) or security (egg)?” We see compliance as a result of secure practices. By making it easy to crunch the data and present meaningful scores and alerts, there is less need to force this.

I’ll say it again, I know many IT guys and gals who are amazing, smart and care deeply about the systems they manage. To combat log apathy, make it easier to deal with them.

Tip of the hat to Dave Meslin whose recent talk at Tedx in Toronto spurred this blog entry

A.N. Ananth

Security Logging as a Detective and Deterrent Control Against Rogue Admins

Intrusion detection and compliance are the focus of log management, SIEM and security logging.  But security logs, when managed correctly are also the only control over rogue admins.  Once root or admin authority has been given to, or acquired by, a user, there is little they cannot do:  with admin authority, they can circumvent access or authorization controls by changing settings or using tools to leverage their root access to tamper with the internals of the operating system.

Audit logs, when properly managed, can serve as a control and deterrent against the privileged super user’s authority. Simply enabling auditing and deploying a log management solution may not suffice; to really be a deterrent, the audit log must be protected from deletion or tampering by rogue admins.

First and foremost, log data must be moved as frequently as possible from the system where it is generated to separate secure log repository.  Today’s enterprise log management solutions do a great job of frequent log collection and long term archiving.  However, who has privileged access to the log management solution and the systems on which it runs?

A log management process is not an effective control if administrators have privileged access to the log management components.  Though administrators should not be denied access to run reports, configure alerts or research logs, privileged access to the log management solution that allows someone to disable, erase or otherwise compromise the integrity of the log collection and archival process should be carefully managed.

A log management solution cannot serve as a deterrent over administrators who have privileged access at the application level or any of the infrastructure components on which it runs.  This includes:

  • the operating system of the log management solution
  • any database servers it uses
  • the Active Directory forest in which the log manage server resides if a Windows server
  • the NAS or SAN where it stores log data

And if the log management application or any of the above components run inside a virtual machine this also includes:

  • the virtualization host, such as VMWare ESX(i), if it runs inside a virtual machine
  • the virtualization manager, such as VMWare vCenter
  • any of the components listed earlier which are used by or host the virtualization manager

Physical access to any of these components could potentially allow administrators to compromise the integrity of the audit trail.  To the extent possible, the log management solution should run on a completely separate infrastructure.

Remember such separation is a protection against not just internal rogue admins but outsiders who succeed in obtaining privileged access.  Typically the larger the organization, the more important and practical it is to achieve maximum separation between the log management solution and the environment it monitors.

Beyond hardware and software separation, the log management application, database servers, storage, OS and other components also need careful management.  Larger organizations generally have dedicated information security teams, and usually within that group is someone responsible for the audit log management process.  For full accountability and separation of duty, that team should have no privileged access to production business systems monitored by the log management process.  Ideally that group would provide the oversight necessary for all components in the log management solution and supervise any action that touches the audit log to insure its integrity and prevent the introduction of backdoors into the system.

There are a host of reasons why even “supervised access” can be compromised:  staff in smaller IT shops aren’t always able to specialize so the possibility for separation of skills and duties may not exist. When an in-house log management system can’t be physically and logically separated, log management as a service may be an alternative to consider.  With cloud-based log management, the entire system is controlled by a professional service team at a separate site.  Serices can be set up with role-based access control so the ability to erase audit logs is controlled.  If organizations can overcome the frequent pushback to sending audit logs to the cloud, full isolation and integrity of log data can be achieved without building a separate log management system, and without the requirement of expertise for audit log management.

Whether an organization goes with an in-house audit log management or turns to the cloud-based service, it should carefully assess its choices in architecture and administrative responsibility. When the worst happens, audit logs may be the only deterrent and detective control over rogue admins.  Are they secure?