How to root out rootkits

If you want to know about the latest malicious rootkit, ask security researcher Dino Dai Zovi. He'll tell you all about his proof of concept rootkit called Vitriol that uses virtual machine instructions in Intel processors to hide a rootkit at the virtualization layer.

He presented this information at BlackHat 2006, the same conference at which Joanna Rutkowski demonstrated her BluePill virtual rootkit that exploited AMD processors.

The good news is that neither rootkit has shown up in the wild. And Dai Zovi says such a hack is not imminent. The bad news: Dai Zovi says these hacks haven't been unleashed on unsuspecting enterprise networks because existing rootkits are working so well, there's no need for hackers to develop these more devious attacks.

"If I'm an attacker and my user and kernel rootkits work 80 per cent of the time, then why go create a virtual rootkit, which is infinitely harder to deploy?" asks Mike Dalton, CTO at Revelogic.

That's not to say hackers are resting on their laurels either. User and kernel-level rootkits continue to get more insidious, burrowing deeper into enterprise networks, hiding themselves in the processor, and exploiting multi-processor systems for gaming-based hacks.

And, although it's hard to say how prevalent rootkits are because they're so darn hard to find, one need only look at the rate of rootkits being used in families of profit-driven malware - most commonly to hide remote-controllers, keyloggers, spambots and gameware.

Rootkits of all evil

"The use of rootkit technologies is prevalent in the malware families our filters are picking up today," says Christoph Alme, Secure Computing's antimalware team lead. "Most commonly these tend to be spambots. Recent examples include Srizbi and Rustock."

Detected in the wild in 2007, Rustock.C spreads like a virus to infect kernel drivers, uses polymorphism (self-changing) to avoid signature detection, loads and hides beneath Microsoft's trusted system driver, and includes a back door Trojan to open and hide two-way communications channels over Port 80.

When analyzed at Rootkit.com this year, Rustock.C was called the "most powerful rootkit ever found under Windows" because of these and other advanced hiding features. The analysis went on to predict that Trojans (back doors) and rootkits will ultimately blend into one malware family.

By combining such hiding technologies, rootkits such as Rustock.C can easily cloak a bot's existence not only from the system, but from the network, where monitoring for suspicious machine behaviors is the last line of defense in detecting the possible presence of rootkit-infected systems.

"Companies need to keep Port 80 open so their employees can use the Internet. Some malware uses that channel to piggyback HTTP traffic," Alme says. "HTTP traffic mainly goes inbound [rather than outbound] over this port, so you need to train your filters to scan outbound HTTP traffic with your network gateway appliance."