Network history: Why it's important and who's responsible for it

This vendor-written tech primer has been edited by Network World to eliminate product promotion, but readers should note it will likely favor the submitter's approach.

The network monitoring industry has been around for a long time, but it's still an immature science relative to other information technologies.

Having participated in the creation of network monitoring solutions for banks, Fortune 500 enterprises, telcos and government agencies, I know a few things about what today's network monitoring technology can do easily and what's more difficult, what works and what doesn't. To get started, there are two fundamental questions that must be addressed: Why is network history critical? And who should own network history?

CLEAR CHOICE TEST: HP, IBM, CA deliver highly scalable network management suites

Why is network history critical?

Most large organizations have deployed some kind of SIM/SIEM (security information and event management) solution to help manage their security posture, as well as some kind of network management system to help manage the network. These are typically major investments that take years to deploy, but they don't provide the actual packet data and network history associated with security events. The most sophisticated deployments pull in hundreds of different data feeds in an attempt to create a single real-time view of what's actually going on across the entire network.

Under the hood, these are large, highly sophisticated correlation engines sifting through vast amounts of data in order to generate intelligence (alarms) that someone somewhere inside the NOC or SOC has to deal with.

The problem with any system based on statistical correlation is that it is non-deterministic. As with any system that's not fact-based, there's a risk that it can generate false positives. While many of the alarms are black-and-white and the remediation process is very straightforward, there's also a lot of gray involved in the process and where there's gray there's risk.

Engineers can't just ignore or dismiss alarms, they are required to act -- either to acknowledge and dismiss or engage and react. In many instances, the act of remediation is very straightforward and has no impact on anyone (a simple firewall rule change perhaps), but in certain instances the act can be significant and impact a lot of people.

If you're about to take the CEO's videoconference down because of a suspected security breach, then you'll want to be certain you have your facts straight before you hit the button. But how do you get that assurance? Well, there's really only one option: Analyze the actual packets associated with the event before you act.

A comprehensive network recording strategy is the practical answer here, and large organizations are starting to deploy network recording infrastructure to complement their net management and SIEM investments.

In this context, the benefits of network recording are twofold: Not only does it reduce the time it takes for an engineer to investigate and respond to an alarm (ask any Tier 2 NetOps or SecOps engineer what would make his life better and he'll say "a perfectly accurate packet trace for every ticket in my queue"), but it also enables engineers to be more definitive in their actions.

Integration between your SIM/net management system and a pervasive network visibility/recording fabric is an effective way to improve the efficiency of engineers and enhance the value of your SIM/management investments. The added benefits (in the case of SIM) is the dramatic effect it can have your containment strategy, helping you to not only diagnose and find root cause, and the absolute clarity regarding exactly what was taken.

Who should own network history?

Both network operations and security groups need to use network history data. These groups generally have the right skills to operate network recording equipment and there have been successful deployments by both groups. But should network recording and flow collector tools be operated by the security team or by the network operations team? The cop-out answer is "it depends on the organization."

There is a clear trend here: Network history is becoming a core network service, and the best practice in most organizations is for it to be owned by the network operations group. Forward-looking network operations teams are keeping network history for their own purposes -- to respond to difficult issues and understand network traffic patterns -- and they are providing appropriate access to security teams and cooperating with them to deal with security incidents.

More and more security teams are expecting and demanding that network history be provided by the network itself, and are deploying their own network history equipment only when the network operations team absolutely can't be convinced.

Here are a few of the reasons why this is so:

1. Network monitoring points are most commonly placed deep inside a corporate network -- in data centers and at aggregation trunks. While security is relevant everywhere, most security groups still spend a lot of their energy on the network perimeter.

2. In best-practice organizations, there is a standard design for monitoring and recording at each layer of network infrastructure. Every network infrastructure project, whether greenfield or upgrade, incorporates that company's standard monitoring and recording design for that part of the network. Involving the security group in every such project is possible, but in many situations it only adds overhead that's difficult to justify.

3. Security teams are generally designed to import raw data but not to export it -- and rightly so. Network operations teams generally have better processes for bidirectional sharing. Network history data are enormously valuable, and no one wants to incur the expense of collecting it twice, so sharing it across groups is nearly always the right answer.

There are two common counter-arguments:

1. Network history data -- especially full packet captures -- are so sensitive that only the security team can access them. However, network operations staff has access to an enormous amount of sensitive information already, so the addition of properly managed network history generally doesn't change your risk profile materially.

2. The network operations team doesn't have budget (in either money or time) to provide network history, so sometimes the only recourse for a security group is to take on the task themselves. This means the security group monitors the points in the network of greatest interest to them either for the long term or until network operations sees the light.

Ultimately, best practices surrounding the why and who of network monitoring will depend on how valuable your network is to your business and what your business can afford to know, what it can afford not to know and who knows it.

Read more about infrastructure management in Network World's Infrastructure Management section.