How to root out rootkits

Malicious traffic can also piggyback on accepted outbound traffic - for example attaching to outbound DNS packets. So Alme also recommends monitoring these types of outbound channels for bursts of traffic, large files and other anomalies that might indicate remote control commands are being sent and received.

Traditionally, detecting a rootkit on a system can be even more difficult than detecting rootkit-hidden traffic on the network, because the rootkit always had as high or higher privilege than antivirus software, Dalton says.

However, VMware's recent addition of antivirus support with their new VMSafe extensions allows antivirus products to run with VMM (virtual machine monitor, aka hypervisor) protection, at higher privilege and visibility into the kernel.

"It's always been a game of cat and mouse with antivirus looking for rootkits and rootkits looking for antivirus, so the rootkit can take control of the security software and continue controlling the infected computer," Dalton says. "Now, by putting security in the Virtual Machine Manager, a kernel rootkit can't even find the security to disable it."

Rootkit toolkit

Rootkit-specific tools such as F-Secure's BlackLight and RootkitRevealer look for discrepancies between the kernel system calls and direct inspection of the disk to detect hidden files, registry keys and other properties, Dai Zovi says. For example, on a Windows machine, they work by looking for discrepancies between Windows Task Manager process list and the internal system task list.

Note, however, that these tools also operate at a lower level of privilege than the rootkit.

"Rootkit defenders running in user-land are trying to do dynamic analysis of the machine to see whether the machine itself is lying. Now does that sound smart?" asks Gary McGraw, CTO of Cigital, and editor of the definitive book, "Rootkits", by Greg Hoglund and James Butler.

Digging deeper

The newest kernel rootkits, containing all types of malicious packaging, can also jump to processors and reboot back into the kernel at bios - even after a computer's been cleaned and restored. Bios is the first place software starts to run, finds its startup routines such as Ethernet and flash/ROM bios extensions.

Dai Zovi says this type is called a "persistent" rootkit. Researcher John Heasman debuted such a rootkit at BlackHat 06 that hides in the Advanced Configuration and Power Interface. Heasman has also discussed similar techniques against the System Management Memory, which two researchers from Clear Hat Consulting were slated to demonstrate at last week's BlackHat.

"If you can control the processing on a computer, how do you monetize that? You sell bots for spam, identity theft and [distributed denial of service]," McGraw says. "But the most efficient way to exploit processors for money is in online games. This is where the cutting edge of bot technology is being carried out."