Archive

EventTracker Recommendation Engine


Online shopping continues to bring more and more business to “e-tailers.”  Comscore says there was a  16% increase in holiday shopping this past season over the previous season. Some of this is attributed to “recommendations” that are helpfully shown by the giants of the game such as Amazon.

Here is how Amazon describes its recommendation algorithm. “We determine your interests by examining the items you’ve purchased, items you’ve told us you own items you’ve rated, and items you’ve told us you like. We then compare your activity on our site with that of other customers, and using this comparison, are able to recommend other items that may interest you.

Did you know that EventTracker has its own recommendation engine? It’s called Behavior Correlation and is part of the EventTracker Enterprise. Just as Amazon, learns about your browsing and buying habits and uses it to “suggest” other items, so also, EventTracker auto-learns what is “normal”  in your enterprise during an adaptive learning period. This can be as short as 3 days or as long as 15 days depending on the nature of your network. In this period, various items such as IP addresses, users, administrators, process names machines, USB serial numbers etc. are learned. Once learning is complete, data from the most recent period is compared to the learned behavior to pinpoint both unusual activities as well as those never-before-seen. EventTracker then “recommends” that you review these to determine if they point to trouble.

Learning never ends, so the baseline is adaptive, refreshing itself continuously. User defined rules can also be implemented wherein the comparison periods are not learned but specified, and comparisons performed not  once a day but as frequently as once a minute.

If you shop online and feel drawn to a “recommendation”, pause to reflect how this concept can also improve your IT security by looking at logs.

Cyber Security Executive Order


Based on early media reports, the Cyber Security executive order would seem to portend voluntary compliance on the part of U.S. based companies to implement security standards developed in concert with the federal government.  Setting aside the irony of an executive order to voluntarily comply with standards that are yet to be developed, how should private and public sector organizations approach cyber security given today’s exploding threatscape and limited information technology budgets?  How best to prepare for more bad guys, more threats, more imposed standards with less people, time and money?

Back to basics.  First let’s identify the broader challenges: of course you’re watching the perimeter with every flavor of firewall technology and multiple layers of IDS, IPS, AV and other security tools.  But don’t get too comfortable: every organization that has suffered a damaging breach had all those things too.  Since every IT asset is a potential target, every IT asset must be monitored.  Easy to describe, hard to implement. Why?

Challenge number one: massive volumes of log data.  Every organization running a network with more than 100 nodes is already generating millions of audit and event logs.  Those logs are generated by users, administrators, security systems, servers, network devices and other paraphernalia.  They generate the raw data that tracks everything going on from innocent to evil, without prejudice.

Challenge number two: unstructured data. Despite talk and movement toward audit log standards, log data remains widely variable with no common format across platforms, systems and applications, and no universal glossary to define tokens and values.  Even if every major IT player from Microsoft to Oracle (and HP and Cisco), along with several thousand other IT organizations were to adopt uniform, universal log standards today, we would still have another decade or two of the dreaded “legacy data” with which to contend.

Challenge number three: cryptic or non-human readable logs. Unstructured data is difficult enough, but further adding to the complexity is that most of the log data content and structure are defined by developers for developers or administrators.  Don’t assume that security officers and analysts, senior management, help desk personnel or even tenured system administrators can quickly and accurately glance at a log and immediately understanding its relevance or more importantly what to do about it.

Solution?  Use what you already have more wisely.  Implement a log monitoring solution that will ingest all of the data you already generate (and largely ignore until after you discover there’s a real problem), process it in real-time using built-in intelligence, and present the analysis immediately in the form of alerts, dashboards, reports and search capabilities.  Take a poorly designed and voluminous asset (audit logs) and turn it into actionable intelligence.  It isn’t as difficult as it sounds, though it require rigorous discipline and a serious time commitment.

Cyber criminals employ the digital equivalent of what our military refers to as an “asymmetrical tactic.” Consider a hostile emerging super power in Asia that directly or indirectly funds a million cyber warriors at the U.S. equivalent of $10 a day; cheap labor in a global economy.  No organization, not even the federal government, the world’s largest bank or a 10 location retailer, has unlimited people, time and money to defend against millions of bad guys attacking on a much lower (asymmetrical) operational budget.

IT Operations Problem-Solvers Infrastructure Maintenance Solution Providers


On a recent flight returning from an engagement with a client, my seating companion and I exchanged a few words as we settled into the flight before donning and turning to the iPod music and games used to distract ourselves from the hassles of travel. He was a cardiologist, and introduced himself as such, before quickly describing his job as basically ‘a glorified plumber’. We both chuckled knowing that while sharing fundamentals in basic concepts, there was much more to cardiology than managing and controlling flow. BTW, my own practical plumbing experiences convinced me of the value of a good plumber.

SIEM in the Social Era


The value proposition of our SIEM Simplified offering is that you can leave the heavy lifting to us. What is undeniable is that getting value from SIEM solutions requires patient sifting through millions of logs, dozens of reports and alerts to find nuggets of value. It’s quite similar to detective work.

But does that not mean you are somehow giving up power? Letting someone else get a claw hold in your domain?

Valid question, but consider this from Nilofer Merchant who says “In the Social Era, value will be (maybe even already is) no longer created primarily by people who work for you or your organization“.

Isn’t power about being the boss?
The Social Era has disrupted the traditional view of power which has always been your title, span of control and budget. Look at Wikipedia or Kickstarter where being powerful is about championing an idea. With SIEM Simplified, you remain in control, notified as necessary, in charge of any remediation.

Aren’t I paid to know the answer?
Not really. Being the keeper of all the answers has become less important with the rise of fantastic search tools and the ease of sharing, as compared to say even 10 years ago. Merchant says “When an organization crowns a few people as chiefs of answers, it forces ideas to move slowly up and down the hierarchy, which makes the organization resistant to change and less competitive. The Social Era raises the pressure on leaders to move from knowing everything to knowing what needs to be addressed and then engaging many people in solving that, together.” Our staff does this every day, for many different environments. This allows us to see the commonalities and bring issues to the fore.

Does it mean blame if there is failure and no praise if it works?
In a crowd sourcing environment, there are many more hands in every pie. In practice, this leads to more ownership from more people than the other way around. Consider Wikipedia as an example of this. It does require different skills, collaborating instead of commanding, sharing power rather than hoarding it. After all, we are only successful, if you are. Indeed, as a provider of the service, we are always mindful that this applies to us more than it does you.

As a provider of services, we see clearly that the most effective engagements are the ones where we can avoid the classic us/them paradigm and instead act as a badgeless team. The Hubble Space Telescope is an excellent example of this type of effort.

It’s a Brave New World, and it’s coming at you, ready or not.

Big Data and Information Inequality


Mike Wu writing in Tech Crunch observed that in all realistic data sets (especially big data), the amount of information one can extract from the data is always much less than the data volume (see figure below): information data.

Big Data

In his view, given the above, the value of big data is hugely exaggerated. He then goes on to infer that this is actually a strong argument for why we need even bigger data. Because the amount of valuable insights we can derive from big data is so very tiny, we need to collect even more data and use more powerful analytics to increase our chance of finding them.

Now machine data (aka log data) is certainly big data, and it is certainly true that obtaining insights from such dataset’s is a painstaking (and often thankless) job, but I wonder if this means we need even more data. Methinks we need to be able to better interpret the big data set and its relevance to “events”.

Over the past two years, we have been deeply involved in “eating our own dog food” as it were. At multiple EventTracker installations that are nationwide in scope, and span thousands of log sources, we have been working to extract insights for presentation to the network owners. In some cases, this is done with a lot of cooperation from the network owner and we have a good understanding of IT assets and the actors who use/abuse them. We find that with such involvement we are better able to risk prioritize what we observe in the data set and map to business concerns. In other cases where there is less interaction with the network owner and we know less about the actors or the relative criticality of assets, then we fall back on past experience and/or vendor-provided info as to what is an incident.  It is the same dataset in both cases but there is more value in one case than the other.

To say it another way, to get more information from the same data we need other types of context to extract signal from noise. Enabling logging at a more granular level from the same devices thereby generating an ever bigger dataset won’t increase the signal level. EventTracker can merge change audit data netflow information as well as vulnerability scan data to enable a greater signal-to-noise ratio. That is a big deal.