A year after meltdown: No silver bullet for DoS

A year after distributed denial-of-service attacks blasted the likes of Yahoo!, eBay and E-Trade Group, no one has found an easy way to defend against a flood of unwanted IP packets.

In fact, everyone's still pretty much in the dark - literally, in one case - when it comes to finding a silver bullet.

A recent meeting of the DDoS Working Group, a forum organised last year to plot network defences, was conducted solely by the light of laptops after KPMG International's Silicon Valley office was visited by one of California's rolling blackouts. In the ghostly glow could be discerned John Zent, manager of risk management for Yahoo!, and Allen Yousefi, information security officer at eBay, along with representatives from security vendors eager to woo these top e-commerce firms.

The talk was no brighter than the lighting. According to several attendees, Yahoo! and eBay are more than just dismayed by the slow pace of finding technical defences to denial-of-service attacks and the even more nefarious distributed denial-of-service attacks, which let an individual launch IP attack streams from hundreds, or even thousands, of compromised computers.

Web site operators are frustrated by the apparent inability of ISPs and Web-hosting providers to quickly filter out denial-of-service attack traffic when it pours into their routers and servers. Whether a low-grade nuisance or the kind of multibarrelled assault that upended Microsoft's sites for three days recently, this "bad" traffic is eating up bandwidth and at times blocking legitimate traffic to the most prominent e-commerce sites.

"People are getting a little radical about it," said one attendee. For companies such as Yahoo! and eBay, "it's a service-level agreement [SLA] issue with the ISPs and collocation providers." He predicted this year would see lawyers battling over whether distributed denial-of-service traffic should have to be filtered out to satisfy SLAs.

Despite the gloom, there are many efforts under way to cope with all manner of denial-of service threats . . . and rays of hope are visible:q Service and software providers have united to share information and forge common defences.q Promising security startups focusing on the problem are attracting big-name backers.q Law enforcement groups, working with the network industry and its customers, are nailing the bad guys.

The DDoS Working Group is doing what it can to spur cooperation among ISPs. The group plans to publish recommendations for automated distributed denial-of-service defences by the end of March.

"There are political issues and technical issues," says Tom Clare, a product manager for Check Point Software Technologies and DDoS Working Group member.

The document is expected to define a common intrusion-detection method for collecting and measuring the percentage of bandwidth being consumed and a flow tag to identify traffic and other Layer 2 data collected from the packets. A firewall or other network device that implemented the DDoS Working Group specification would be able to report the start of an attack to the ISP, and other ISPs using compatible equipment would be able to share the information.

But it's uncertain whether ISPs can interact smoothly even if equipment makers support a common security specification, which may leave this as yet another security proposal that never got off the drawing board.

ISPs in the middle

This much is clear: ISPs play the critical role in the distributed denial-of-service endgame against attackers, who are heavily armed with denial-of-service "malware", software posted at hacker sites for free use. And most of the intrusion-detection analysis and filtering that ISPs do is manual and difficult.

"We can't be held responsible for attacks on our customers," says Amir Moujtahed, director of systems engineering and corporate security at Epoch Internet, a US-based ISP. "But if customers give us the IP addresses [of the source], we will block them." Epoch has intrusion-detection equipment from NFR Security on its external and internal networks, and Epoch engineers watch the logs closely for evidence of attack signatures. But it's a labour-intensive process.

Moujtahed says ISPs are trying to do their part by installing anti-spoofing filters and cooperating with competitors through informal agreements hashed out in the ISP Service Consortium, which meets monthly.

"This is all part of the lesson learned after what happened last year," Moujtahed says. "ISPs like [Genuity], UUNET Technologies and AOL (America Online) compete, but we are working together on this."

It's small comfort to the high-tech industry that the 16-year-old perpetrator of last February's incidents, a Canadian hacker nicknamed Mafiaboy, last month pled guilty to single-handedly attacking, eBay, Yahoo!, Charles Schwab & Co, CNN and eTrade, among others.

Mafiaboy carried out his distributed denial-of-service spree using attack tools available on the Internet. The tools let him launch a remotely coordinated blitz of IP packets from servers compromised by agent attack "zombies". Mafiaboy awaits sentencing, but it's expected he won't get much more than two years in a juvenile detention centre.

Those attacks forced most of the victimised e-commerce sites offline for about three hours. In the heat of battle to block the blitz of IP packets, ISPs did what they could through filtering bad traffic and claimed victory when it ended. But security experts familiar with what occurred agree that this filtering accomplished little and that relief came because Mafiaboy simply stopped his attacks after three-hour intervals.

"The attacks happened Monday through Wednesday, and those guys were still working Friday and Saturday to figure out what happened," says Frank Huerta, CEO of Recourse Technologies, which makes security gear to detect and trace denial-of- service attacks.

Like many experts, Huerta says the work ISPs did manually filtering bad traffic didn't stop Mafiaboy's attacks. And though law enforcement officials did extensive work bringing him to justice, one reason they succeeded was that he bragged about his exploits in an Internet chat room.

There are stopgap measures that Web sites can take to shore up defences, such as using as many load-balancing and high-speed pipes as they can, as well as intrusion-detection systems that can indicate suspicious activity is suddenly on the radar screen.

And that is better than nothing. US-based Fidelity Investments and Bear Stearns reportedly deployed Top Layer Networks's AppSwitch with its intrusion-detection features after last February's attacks on e-commerce sites.

Finding a cure

Overall, there's a more sober-minded assessment of the problem among vendors than a year ago.

Cisco Systems last February claimed that making use of ingress filtering in routers, a technology described in IETF draft RFC 2267plus, would stop denial-of-service attacks. But the router manufacturer has abandoned that stance.

"There is no silver bullet for a [denial-of-service] attack," says Lance Hayden, a manager with Cisco's consulting services team. But Cisco and a number of venture capital firms are investing in startups that are promising to develop comprehensive defence systems for distributed denial-of-service attacks. Another startup, Arbor Networks, is also striving to find a cure.

So too are established security vendors including Internet Security Systems (ISS). Allen Wilson, director of emerging technologies at ISS and a DDoS Working Group member, says tracing this type of attack remains "very manually intensive and time-consuming. For ISPs, it's one hop at a time, and you need to get hold of people and let them know that your network is attacking theirs."

ISS claims to be developing technologies that depend on what it calls "the moving target defence". The idea is that if an attack is launched at a Web site, the victim and ISP work together to identify the source and then create a "black hole for the IP address", Wilson says. "You drop the packets but don't kill the connection, which helps trace back the attackers."

At the same time, you create a temporary IP address for your site that gets broadcast out to enable legitimate traffic to still find you.

Quantifying the denial-of-service problem is not easy. Whenever a Web outage occurs, security experts always suspect denial of service, even if the business blames internal screw-ups. Online auction vendor eBay has suffered several Web outages in recent months that many security experts suspect were denial-of-service attacks, something eBay vehemently denies. However, it was clearly a denial-of-service attack that disabled much of the Undernet, part of the Internet Relay Chat network, in early January.

After last February's attacks, the Clinton administration asked the IT industry what it could do to help combat what everyone suddenly realised was a dangerous situation on the Net.

It took 11 months to come forward with a plan, but 19 high-tech corporations recently formed an organisation called IT Information Sharing and Analysis Center (IT-ISAC), which will run a so-called "virtual centre" to share information about denial-of-service attacks and software vulnerabilities in general. Founding members are paying almost $US1 million for the privilege, although general membership fees, which won't include access to all the information, are as low as $5000.

The organisation's database of shared information, which will be managed by ISS, is intended to help solve security problems, so vendors accessing this sensitive information have agreed not to use it as a marketing weapon.

Those who expected ISPs to roll out new technologies or services to help stop these attacks in the past 12 months have surely been disappointed. ISPs are essentially using the same spot-filtering and monitoring techniques today as a year ago. Nevertheless, ISPs claim heightened awareness and vigorous monitoring have helped reduce damage.

"We regularly see attacks, but nothing at the level of last year's on multiple, highly visible customers," says Kelly Cooper, security engineer at Genuity. "If we were to offer filtering and monitoring services to our customers for an extra charge, that would sort of be like blackmailing them."

Genuity expects new capabilities from router and switch vendors that will integrate IP address filtering directly into the operating system of the device. One of the most common reasons why ISPs are not setting up IP address filtering is because it can slow the network. However, if filtering is integrated into network devices, performance should not be hurt, Cooper says.

Vint Cerf, senior vice president of Internet architecture and technology at WorldCom, says that standard load-balancing and content-distribution techniques, which many Web-hosting service providers use, reduce the negative impact of these attacks.

"Load sharing across multiple servers helps reduce the impact of classic [DDos] attacks because there are multiple versions of a Web site operating across the Internet," Cerf says. In addition to distributing legitimate traffic, load balancing and caching distribute rogue distributed denial-of-service packets so one server is not crumbling under the weight of an attack.

ISPs also see hope in specifications being developed by the Internet Engineering Task Force. I-Trace is one preliminary technology that will allow ISPs to quickly find where a distributed denial-of-service attack originates. Once the ISP recognises the source of an attack it can immediately set up a filter.

But this technology is very much in the early stages of development. All in all, it certainly seems like the industry will experience at least another year of being in the dark on distributed denial of service.