Flexible Enterprise Log- and Event-Data Management

- Bruno Kurtic, Vice President of Product Management at SenSage (www.sensage.com), says:


Why is getting a handle on log data and actually turning it into useful information a challenge?

Sophisticated, long term analysis of log data is the key to addressing emerging security threats, compliance mandates and a host of risk management initiatives. Why? Compliance mandates require firms to retain and analyze event data for up to seven years. As a result, nearly every organization is required to create secure, centralized log and event data repositories.

The first place most organizations turn to is legacy data management suppliers. However, traditional data management systems were built for transactional data - not event data. The requirements to manage event data are different:

• Data - Log and event data can never be updated or changed
• Collection - Difficult due to hundreds of data formats and dispersed endpoints
• Analysis - Data must be analyzed in real-time and over extremely long time frames
• Users - Typically few users but they need access to years of data
• Queries - Often ad hoc, time-sensitive and dispersed across huge data sets
• Volume - Enormous volumes of data creation and collection

What are the biggest issues with log management tools?

Security vendors espouse the benefits of legacy log management and SIEM tools to manage event data. Unfortunately, these point solutions don't scale, are difficult to customize and often can't address many of the emerging use cases of event data management.

On the other hand, traditional data management systems and data warehouses are designed for transactional data, not event data, which leads to dramatically higher costs and complexity. Some vendors still try to convince customers that the “one database, one data warehouse” approach is the correct one, forcing them to over spend and endure extremely long implementation cycles.

The bottom line – IT and data center managers need a flexible enterprise log and event data management approach that provides true business intelligence, offers rapid delivery on queries and can scale on-demand.

How can data center/IT managers overcome these issues?

All of these issues point to the need for a data warehouse that is specifically designed for the unique requirements presented by event data, such as log files. SenSage log management software provides automated collection, storage, correlation and reporting to allow organizations to effectively monitor, report and investigate activity and events from thousands of different log sources throughout the enterprise.

SenSage log management functionality enables organizations to monitor end-users as well as administrators for the purpose of detecting suspicious behavior and intrusion attempts, establish audit trails for change control, enforce accountability over administrators and conduct better investigations and forensic analysis.

SenSage log management technology is based on a patented columnar database architecture approach for event data. Unlike traditional relational database management systems that use a row format, data is organized by column in a single, centralized data repository specifically designed for event data. While the difference may sound trivial, the performance gains are dramatic. Indexes are unnecessary as each column is actually an index, reducing storage and maintenance requirements.

Data is compressed at a 40:1 advantage vs. relational databases and stored in a hierarchical series of folders and flat files on each node's local disk. The SenSage Event Data Warehouse easily scales by adding new nodes and takes advantage of new hardware features, such as multi-core processors and faster local drives. To maintain constant availability, backup copies of each node's data are stored on another node for data redundancy and automatic failover. With SenSage, organizations can easily query years of data from multiple sources at any detail level to support their business requirements.

SenSage provides an intuitive graphical user interface that makes it easy for business analysts and executive users to create new business-specific rules, generate reports and analyze event data efficiently. In addition, SenSage provides powerful real-time engine as well, making the process of gleaning actionable intelligence from the reams of event data even faster and more cost-effective.

More Than "Just Logging"

 - Pam Casale, Chief Marketing Officer at Intellitactics (www.intellitactics.com), says:

Why are enterprises looking for more than “just logging"?

For many reasons, logs are not useful in real time and are not helpful if you are trying to be proactive. There just aren’t enough human eye balls to look at them no matter how sophisticated your search engine or indexes are. And, of course you don’t want to wait for something bad to “show up” you want to be in a position to be proactive. Once you transform billions of logs into a fewer number of more meaningful security events you can do more: run reports that isolate bad actors or suspicious activity; run reports that show control violations that put you at risk; create notifications for other IT groups or stakeholders; or, even take small actions like close ports, reroute network traffic, stop transactions, make passwords improperly used inactive.

How can data center/IT managers overcome issues with log management tools?

• First, don’t settle for just logs. No matter how fancy the search capabilities of a logging tool might appear to be, logs alone limit your ability to understand and investigate what’s going on. The technology is available and affordable that will transform logs into more meaningful, actionable events – choose one of these. Today they almost cost the same as the logging only tools.

• Second, choose a product that is fully capable. By this we mean find a vendor with ONE appliance that does all the work: collects and stores all the logs, provides a user interface that enable proactive management of security events and generates all the reports you need for an audit and most important find a product that generates reports you can give to administrators in exchange for the logs the security people need. By this we mean that firewall or anti-virus reports can be provided back to the domain owner that are valuable for improving the effectiveness of operations. Security gets the logs they need to defend the enterprise and the domain owners get reports that help them sustain the availability and performance of service.

Collecting logs and using them to manage security and prove compliance is essential. Log management that strains resources, costs a lot directly or indirectly is not essential. Logs are interesting to look at but when every minute counts you want events, automated alerting and notification. Intellitactics learned about log and event management from some of the world’s most capable enterprise organizations. We took that knowledge and expertise and applied it to the appliances called SAFE. It’s the only security management appliance with all the capability on one affordable, right-sized box. Less rack space, no DBA required, self managing and monitors its own health. The SAFE dashboard can be configured for every domain owner and the reports are audit worthy and can also be used by operations for sustaining availability of key business services. If your needs change, or your sphere of influence grows you can layer capability without losing your investment.

Poring Over Logs: A Thing Of The Past

- Leigh Purdie, director at InterSect Alliance (www.intersectalliance.com), says:

Over the last few years, the SANS log management survey has highlighted that the process of actually collecting useful log data has become much easier for organizations. We'd like to think that, as the defacto standard for centralized log collection across a wide variety of platforms, the free/open source Snare series of agents have had a fair bit to do with that.

However, now that collecting log data has become a relatively painless task, IT security teams are flooded with logs. Managing this sort of volume of data over the network is generally not too much of a challenge for those that run a data centre, or network; but the constant stream of information all heading back to a single server, at all times of the day with no let-up, can be draining - particularly if the agents that collect the log data, are not spectacularly careful with their resource utilization. Snare was designed from the start, to be very careful with resources - we try to make sure that we have a very low memory footprint, we push data off the client as quickly as possible so that disk utilization is low, and we implement active filtering to limit log flow to only those events that are likely to be of interest from a security perspective. In addition, an agent-based solution like Snare, in contrast to older agent less 'octopus-style' log grabbers, decentralizes processing and filtering, doesn't require domain-administrator-level privileges, and allows practically real-time dissemination of log data back to a central location in a resilient and responsive fashion - all of which make the life of a system administrator easier, which means that the IT Security team will have more of a chance of getting effective auditing implemented. However, at the end of the day - there is a lot of raw data flowing across the network, which needs to be intelligently managed and analyzed.

Raw data is great for forensics, but raw data doesn't allow your average system administrator to detect at a glance whether their network is infested with malware, or to produce compliance driven reports. Raw data doesn't help the people that are responsible for key information resources within an organization, to work out whether their spreadsheets, databases, web content and so on, are getting to the right people - and staying out of the hands of people who shouldn't have access.

A few years ago, the corporate hierarchy saw IT security as a 'black box'. They employed people to manage IT related risks on their behalf. As businesses have increasingly become information-focused, and information-dependent, it became more and more obvious that delegating the responsibility for such a key component of the organization was unsustainable. As such, decision makers needed to be broadly aware of the current threats to their information, and be convinced that the countermeasures employed were up to the task of providing reasonable levels of protection. Security logs are a key resource in evaluating both the current threat profile, and also determining the success of the deployed countermeasures.

Turning a couple of gigabytes of raw data into a one-page summary that is tailored to a particular 'data owner' or CIO? That generally requires two things - raw computing power is one - and we all have access to that.. but the second is a tool with enough flexibility to map the raw data, into organizational security goals. The trick is, of course - that every organization has different security requirements. One company may focus on the corporate gateway to the Internet, another may have their internal network air-gapped, and will focus on attacks against the public-facing web server. Another may not be as concerned with the gateway environment, but will be critically interested in changes made to a particular internal database by users outside a 'authorized' list of staff.

Get A Grip On Your Logs

- Dominique Levin, EVP of Marketing and Strategy for LogLogic (www.loglogic.com), says:

Why is getting a handle on log data and actually turning it into useful information a challenge?
Over one terabyte of log data is produced each day by the average enterprise. Multiply that figure by seven years (the length of time that the logs must be archived by law), and those logs, printed out, would stretch to the moon and back more than 6000 times. To make matters more complicated, logs are generated in different formats, and the strings of letters and numbers must be deciphered before someone can understand and act upon the log data.

LogLogic offers five product lines that can be seamlessly integrated to provide the most comprehensive log-powered security solution available. LogLogic’s open log management platform provides the architectural base where logs are collected, normalized, indexed, and stored. Built on top of this platform are four log-powered applications, which enable users to leverage the logs for a variety of security and business purposes. LogLogic Security Event Manager analyzes thousands of complex events in real time to reveal the most critical security incidents and provide deep insights into the security posture of the IT infrastructure. LogLogic Change Manager streamlines the end-to-end design and generation of network security rules. LogLogic Compliance Manager provides a workflow-based management system, role-based dashboards, and automatic regulatory mapping to reduce compliance time and increase efficiency. LogLogic Database Security Manager goes beyond native database audit functionality, providing a real-time detection and prevention system that monitors and analyzes all database activities—without impacting performance.

In this age of increasing data breaches, many organizations have implemented Database Security Monitoring. The problem is, most of these products employ passive monitoring: instead of being able to respond in real-time, the IT staff only discover a data breach later on when they review their logs. By that point, much damage has already been done. The better alternative is to combine active monitoring with real-time response. LogLogic’s soon-to-be-released Database Security Manager goes beyond passive monitoring by empowering users with real-time, proactive controls.