Log Management and Incident Response
I’m going to let you in on a little secret. It’s a tough message to get, but part of being Pragmatic is not deluding yourself about what you can and can’t do. The cold harsh reality of today’s information security environment is that you will be compromised. I don’t know whether it will be tomorrow, next Tuesday, or some other time in the future -but it will happen. There are just too many legitimate attack vectors, too many restrictions on what we can and can’t do, and too many limitations on budget and resources to ever be “truly secure.”
The good news is that all is not lost. What we do as information security professionals still matters, even though at some point you’ll have an incident. Here’s another secret for you: The way you deal with the inevitable incident will make the difference between being a hero and a goat. Heroes figure out how to contain the damage and make the incident into a learning experience for the organization.
Goats dust off their resume and hope they can get another gig.
One of the first things I counsel my user clients on is to develop a structured, documented and heavily practiced incident response plan. Most security professionals feel they are too busy to actually write stuff down, but there are a lot of reasons formality is a must when dealing with incident response. For a very detailed description of the Pragmatic incident response process, check out my book – The Pragmatic CSO at www.pragmaticcso.com.
In a nutshell, I advise you to build an “Incident Playbook” that details exactly what is going to happen if/when you are compromised. Here is a very high level structure to the plan.
- Section 1: Containment – This section specifies what happens in the event of an issue. Do you disconnect the specific device or the entire network? Do you patch all devices immediately, add a new IPS rule, turn off exposed applications or take some other remediation actions? The point is you have a problem, and you don’t want your team (or yourself) to freeze when you need to be taking action. So script out the first few “plays” in your playbook and make sure you can execute on these during trying times.
- Section 2: Notification – Depending on the nature of the incident, you may need both an internal and external component to the notification plan. First, nail down the internal stuff. When do you go to the CIO? When do you consult the general counsel? What about the CEO and entire senior team? Then deal with the external notification process. How and when do you tell customers about a potential privacy breach? These questions must all be answered BEFORE you have an issue.
- Section 3: Law Enforcement – You also want to have defined (and agreed upon) times when law enforcement will be brought into the situation. If you detect a criminal activity, obviously you want the cops (or special agents) involved sooner rather than later. But you want to make sure you aren’t just subjectively deciding when to notify the authorities.
I’ll also remind you that practice makes perfect. Seriously, we’re not just talking about Little League here. The Incident Playbook won’t be worth the paper it’s written on if you or your team bungles the response when you are playing for keeps. So practice, practice and then practice some more.
Since this column is about log management and incident response, let me talk a bit about the documentation required to handle incidents effectively. Basically there are two aspects where log management will prove an invaluable tool in your IR arsenal. First is operationally. Once you identify that you have a problem, you can use your log information to quickly pinpoint the problem, contain the damage, and ultimately remediate the issue. I discussed the Pragmatic operational security techniques that leverage log management last month.
Secondly, log information will prove invaluable as you (or an forensic specialist and/or law enforcement) investigate the compromise and try to figure out what happened and build a case against the perpetrators. Actually in some cases, your internal HR and legal team may suggest you just monitor the situation in order to gather even more evidence that could be used. In either case, the log files indicating what the perpetrators did and when will be critical to building up the evidence you need.
So what? You are already gathering log files, so what’s the big deal? Why do you need a purpose-built log management platform? Basically, your evidence needs to stand up in court. So you can’t risk having any of your logs roll, then you’d lose the data. Or even worse, have the logs tampered with to hide any tracks the attackers may leave, so even though you have the data – it doesn’t stand up in court.
This last point is one of the most important drivers for a separate log management infrastructure. A commonly used technique by the hackers is to cover their tracks by going into the logs of a compromised machine or application and altering the log files to remove any evidence they were there. If you are sending the log records in real time to a separate environment, there is no chance for tampering and inconsistencies will be identified very quickly.
Moreover, the log management infrastructure should be sequencing and hashing the log records to cryptographically prove that the logs have not been tampered with. This again is critical should your case ever be told in front of a jury. Secure logs provide much of the evidence needed to bring perpetrators to justice and make sure the charges stick.
The sad truth is that by the time most organizations realize they have an issue and they needed to be gathering log data in a secure fashion, it’s too late. Once the log files roll, the data is gone. Once the attackers have altered the logs, you can’t figure out what happened. It’s never too early to start gathering data that will be useful when you number comes up and you get compromised.
It will happen. At some point your luck will run out and wouldn’t you rather be the hero, who contains the damage and provides the data to facilitate a comprehensive investigation? I know I would.
EventTracker 6.0 launches; offers unprecedented scalability and flexibility to geographically-dispersed enterprises
The new release delivers major architectural and performance enhancements, more powerful analysis features, and a redesigned reports console that provides audit-friendly workflows for a pain-free compliance audit.
The right log management solution, used in conjunction with internal procedures and policies, provides Covered Entities with the capability to have a strong, yet cost effective compliance strategy in place, and to easily demonstrate adherence to external auditors.
Building and maintaining a compliance workflow process sounds daunting, but it’s not all that different from other enterprise business processes.
Arch Insurance demonstrates compliance with Sarbanes-Oxley compliance to external auditors and improves security posture with real-time monitoring of critical systems and user-activity tracking.