The sailor in The Rime of the Ancient Mariner relates his experiences after long sea voyage when his ship is blown off course:
“Water, water, every where,
And all the boards did shrink;
Water, water, every where,
Nor any drop to drink.”
An albatross appears and leads them out, but is shot by the Mariner and the ship winds up in unknown waters. His shipmates blame the Mariner and force him to wear the dead albatross around his neck.
Replace water with data, boards with disk space, and drink with value and the lament would apply to the modern IT infrastructure. We are all drowning in data, but not so much in value. “Big data” are datasets that grow so large that managing them with on-hand tools is awkward. They are seen as the next frontier in innovation, competition, and productivity.
Log management is not immune to this trend. As the basic log collection problem (different sources, different protocols and different formats) has been resolved, we’re now collecting even larger datasets of logs. Many years ago we refuted the argument that log data belonged in a RDBMS, precisely because we saw the side problem of efficient data archival begin to overwhelm the true problem of extracting value from the data. As log data volumes continue to explode, that decision continues to be validated.
However, while storing raw logs in a database was not sensible, their power in extracting patterns and value from data is well established. Recognizing this, EventVault Explorer was released in 2011. Users can extract selected datasets to their choice of external RDBMS (a datamart) for fuzzy searching, pivot tables etc. As was noted here , the key to managing big data is to personalize the results for maximum impact.
As you look under the covers of SIEM technology, pay attention to that albatross called log archives. It can lead you out of trouble, but you don’t want it around your neck.