Helen Yeatman takes a look at the Australian government’s proposed legislation banning spam, the impact it will have on the local antispam industry and the industry’s critical role in the fighting the junk mail epidemic.
Under the Australian government’s proposed legislation that will see electronic junk mail (spam) banned, ISPs will be forced to implement antispam technologies from an approved list and will be encouraged to educate customers about how they can control spam most effectively. While the legislation alone will do little to stop spam from being generated from its source, it is an essential first step in the long-term strategy of developing anti-spamming enforcement networks internationally, and for local antispam developers, resellers and service providers, it presents significant sales opportunities.
Spam has reached epidemic porportions. According to antispam company, Brightmail, 40 per cent of all email is now spam, and nearly 15 per cent of all spam is pornographic, up from 5 per cent last year.
Perhaps one of the most significant catalysts behind this explosive growth is the increasing uptake of broadband Internet connectivity, which allows spammers to send greater volumes of junk mail. Further adding to the problem, many broadband users are connected to the Internet for greater periods and don’t understand how to secure their machines. A lot of the spam email being generated today is being bounced through broadband customers, usually home users, allowing spammers to bypass restrictions set by ISPs on their mail servers.
In response to mounting pressure from both the private and public sector, the Australian government announced last month that it would be introducing legislation banning spam in 2004. Earlier this year, the government held several conferences and summits with industry and corporate bodies to discuss the myriad of problems associated with controlling the spam epidemic.
The National Office for the Information Economy released a report titled The spam problem and how it can be countered in which it issued recommendations for the proposed legislation that were largely accepted by the government.
The antispam measures that the Australian Government will introduce include national legislation, to be enforced by the ACA, banning the sending of commercial electronic messaging without the prior consent of end-users unless there is an existing customer-business relationship (an opt-in regime); civil sanctions for unlawful conduct including financial penalties, an infringement notice scheme and the ability to seek enforceable undertakings and injunctions; the requirement for all commercial electronic messaging to contain accurate details of the sender’s name and physical addresses and a functional ‘unsubscribe’ facility to enable people to opt-out; banning the distribution and use of email ‘harvesting’ or list-generating software, and working with international organisations to develop global guidelines and mechanisms to combat the global spam problem.
The Government has proposed that the Internet Industry Association (IIA), the Australian Information Industries Association (AIIA) and their members should develop better practice guidelines which provide a resource for both members and end users to combat spam and require ISPs to make available to clients filtering options from an approved schedule of spam filtering tools at reasonable cost. The proposal also suggests that the two associations evaluate and publicise spam filtering options and products.
IIA’s chief executive, Peter Coroneos, said it was way too early to talk about what kind of antispam technologies the IIA was going to recommend to ISPs in its codes of practice.
“We’ve got industry codes for ISPs for Internet content and ISPs have to provide filters. So what we have is a schedule at the back of the code that sets out a whole lot of criteria that content filters have to meet,” Coroneos said. “A filter provider comes to us with their product, they then have to get the product independently tested by the CSIRO and if they come back to us and say that it’s an effective tool according to our criteria, then we will add it to the schedule. It then becomes a commercial decision for ISPs to determine which of the 20 available filters that are now on our code schedule they will deploy.
“Some ISPs are client side so rather than deploy them themselves, they simply make them available to their customers. Others are server based where the ISP will implement the solution in-house”.
It’s not certain that the exactly the same systems and processes are going to be used in determining the schedule of recommended spam filter products. While the government and the IIA are satisfied with using the same process to divine the schedule of listed antispam products, Coroneos suggested the industry may not agree.
“A lot of Australian ISPs are moving to provide spam filtering solutions to their customers anyway as a value-add,” he said. “Some industry players are saying we are going to do this anyway and we don’t need the government of the industry bodies telling us what we have to supply. All of these issues still have to be negotiated.”
The best opportunities emerging for ISPs lie in bundling antispam solutions with other email or Web-based services, CEO of Australian messaging and antispam service provider Messagecare. Andrew Kent, said.
He suggests ISPs bundle their email services with antispam solutions will make customers more sticky.
“If customers don’t get a lot of spam then they’re going to stay with that ISP for longer,” Kent said. “Until recently, the best solution to getting rid of spam for end users was cancelling your account and going to another ISP. In the long term this is disastrous to the ISP.”
The Government’s new initiatives will create heightened awareness about the nature of spam, the necessity of antispam filtering technologies and a surge in demand for antispam technology. Subsequently, several vendors have moved out of their traditional playing fields in order to secure a stake in the growing market.
In April, Trend Micro announced its foray into the antispam market with the launch of the McAfee SpamKiller for Microsoft Exchange Small which uses a combination of whitelists, blacklists, content filtering and heuristics to block spam.
Vendors like Microsoft are also integrating antispam technology into their products. The vendor recently included antispam technologies in the releases of Exchange and Outlook in Office 2003.
Emerging antispam technologies
Even hardware vendors are joining the fray. HP recently announced plans to bundle antispam software with its consumer desktop computers, though the program will only block pornographic and other offensive unsolicited email.
There is a plethora of spam filtering options on the market and no one solution is infallible or suitable for all types of customers. They all have their respective merits and shortcomings. To achieve optimum results from these widely varying technologies, it is essential that ISPs and resellers understand how the technologies work, what customer types they are best suited to and, of course, the needs of your customers.
A number of antispam vendors and service providers integrate a number of different techniques in their products to block spam, including filters, blacklists, whitelists, challenge/response systems, traffic and IP address/source information monitoring.
SpamAssassin, an effective Perl-based spam detector that runs on Unix and Windows, is one such solution that applies a range of techniques to identify and block spam, such as rule-based heuristic filtering, blacklists, the collaborative spam-tracking database, Vipul’s Razor, which works by taking a signature of spam messages, and Bayesian statistics to classify mail.
Bayesian filtering has become a popular method and is used in a wide variety of antispam software and service offerings on the market. According to industry pundits, the new generation of statistical spam filters offer improved performance on the first generation of spam filters which used rules to recognise specific spam features. Statistical spam filters, or Bayesian filters after Thomas Bayes, mathematician that established the branch of logic applied to decision making using statistics garnered from past events to predict future events. Statistical filters look at the entire contents of each incoming email and decide whether it is spam based on its overall similarity to previous spams. This new kind of filter generally catches more spam than other filters, however it tends to generate a greater number of false positives.
Australian messaging and antispam service provider Messagecare, takes a rule-based filtering approach with its Spam Trap service offering.
“We have a network of spam traps around the world where we collect spam — we establish what appear to be normal email addresses and try to get them added to spammers’ lists and signed up to as much spam as possible,” Kent said. “We scan the email and its content and make a fingerprint and attribute each spam email with a unique identifier number. This enables us to automatically recognise and block the message each time a customer is sent it.”
Unlike the Bayesian filtering method which uses algorithms that detect the frequency with which certain words appear in an email to identify spam messages, Messagecare’s Spam Trap service uses the MD5 Checksum method to identify spam, where each transmitted message is given a numerical value (identifier number) based on the number of set bits in the message. In other words, the Checksum method looks at the whole of the email to form an identifying fingerprint while Bayesian method looks for specific words within the content of the email.
“We’re able to tailor the fingerprints to make sure that when spammers start adapting the messages. That’s what we spend a lot of our time doing — writing, maintaining and managing proprietary approaches on top of the fingerprinting.”
While Kent conceded that Bayesian filters generally do block a greater percentage of spam than other filtering techniques using Checksum methodology, he said Bayesian filters produced a much higher rate of false positives and could be bypassed as spammers learnt to understand the algorithms.
“Because Bayesian filters are statistically based you will catch false positives,” he said. “We shied away from that.”
For the general user that doesn’t mind doing some work, Kent said the Bayesian filter approach was good but it took a lot of training and there were risks associated with it.
“It doesn’t save bandwidth at all — if it’s over a dial-up connection, users end up downloading the spam and having to pay for the bandwidth even though the spam is going to be deleted,” he said.
This bandwidth consumption is one of the major disadvantages of software solutions deployed at the end-user level and lends a strong argument in favour of managed antispam services where the spam is deleted by the service provider before it reaches the ISP, corporate or end-user customer.
“For most organisations, the only safe way of managing their email is by using the outsourced services model”, Kent said. “If you have a small company that with an IT group suddenly having to spend 24 hours a day trying to adapt to filtering everyone’s email they’re just not going to be as good at it as an antispam services company that is constantly monitoring the global spam picture.”
The challenge-response system is another hot new technique being embraced by the industry. The concept is simple — incoming email is intercepted before users see it, and a “challenge” email is returned to the sender. If the sender replies to the challenge message, then the original message is forwarded on to the user, otherwise it is discarded.
According to Kent, the challenge-response system is often not suitable for enterprise customers.
“Users may end up having to ask a CEO of a large organisation to go through the pain of going to a Web site and entering a code before they can talk to the user and I don’t think that is very effective for that sort of client,” he said. “This kind of solution is really better suited to consumers.”
One of the most obvious problems with challenge-response systems is that passwords can easily be stolen. Yet this technology faces a larger problem.
What happens if the sender does not receive the challenge message from the intended recipient because the sender’s antispam software blocked it? The solution to this problem would be to create a loophole so that challenge messages are immune to spam controls. This loophole also creates a window of opportunity for spammers who would quickly exploit it by writing spam messages that resembled challenge messages.
The global solution
The proposed legislation and antispam initiatives are an important first step towards disempowering spammers and abolishing the spam industry, government and industry figures are aware of the virtual impotence of such measures in controlling spam globally.
The development of government-to-government or international legislation that outlaws spam is unlikely, largely due to core differences in national constitutional legislations. For example, the first amendment of the US Bill of Rights (or constitution), freedom of speech, differs from Australian laws relating to freedom of speech.
However, government-to-government co-operation is occurring through various governments’ independent statutory bodies.
“At the moment it’s not happening at a government-to-government basis, it’s happening at an agency-to-agency basis,” Coroneos said. “There are about 33 agencies around the world that are collaborating with each other and sharing information in an effort to eradicate spam. For example the ACCC works closely with the US Federal Trade Commission. OECD is also doing work in this area, particularly focusing on spam fraud.”