Telstra's contingency planning has come under fire from the service provider channel after an intercontinental cable was damaged early last week.
The SEA-ME-WE cable that links Australia to Asia, the Middle East and Western Europe, is one of the busiest Internet data and telephony cables in the world, servicing around 50 to 60 per cent of Telstra's traffic.
Internet access ground to a halt last Monday after the cable broke at 3:20pm AEDT. Chaos ensued for around five hours as Telstra's technicians frantically tried to reroute traffic through the eastern line and satellite hookups.
Over 400 ISP wholesalers and hosting companies also sprang into action as they tried to find alternative ways to service their customers. The situation has not endeared Australia's incumbent carrier to many in the channel left questioning Telstra's redundancy and disaster recovery planning.
"It is quite clear Telstra had no redundancy plan in place for international connectivity, or very little. When they started to try and do something about it, they had data routing in from all over the place," said Lloyd Ernst, CEO of hosting company Web Central. "[It was] Not very professionally handled."
Ernst's feelings were mirrored by numerous ISPs who took issue with how badly the situation impacted access to not only the regions connected by the cable but to sites in the US and Australia. Telstra began updating its router tables from around 7:00pm when by that stage over 650,000 users had been badly affected. Furthermore, unconfirmed reports suggest that during this rerouting, one of Telstra's core switches melted under the load, adding to the problem.
"The question left to ask is why was the domestic market so badly affected when it was an international link?" proposed Ernst. "It was really quite a nightmare there for a while."
Lyndon Vincent, director, services division, of commercial ISP Australia Wide Computer Resources (AWCR), claims his company was one of the first ISPs to notice there was a problem, reporting it to its upstream carrier AAPT and then Telstra.
"From 7:00pm to 2:30am, a small team at AWCR reconfigured routers and changed Proxy destinations to ensure that when our customers arrived at work on Tuesday morning, both their e-mail and Web browsing worked first go," Vincent said.
Fortunately for AWCR, it has pairing arrangements with alternative carriers, but Vincent expressed concern for ISPs solely at the mercy of Telstra's network. "For some of the smaller ISPs, it could have caused some major headaches," he said.
SingTel is coordinating repair operations on behalf of the consortium of 90 companies from around the world who own the cable, including Telstra, France Telecom and Hong Kong Telecom, confirmed Stuart Gray, public affairs manager for online services at Telstra.
While the cause was still unknown by the end of last week, Gray claims the belief is the cable suffered a "tension break", which could have been the result of a ship's anchor or a shifting of tectonic plates. Although the cable can withstand "tens of thousands of tonnes" of pressure, it is lying only 25 metres underwater -- 63km south of Singapore.
A fire in the Sydney suburb of Paddington close to where Telstra houses a number of its core routers was initially thought to contribute to the traffic bottleneck, but Gray denies there was any connection.
Telstra expected the cable to be fixed by Tuesday this week.
* Diameter: 46mm
* Length: 39,000km
* Throughput: 20Gbps
* Cost: $1.7 billion
* Over 650,000 Big Pond subscribers and over 400 ISPs were affected* The cable has 40 landing points in 34 countries* Owned by a consortium of 90 companies.ends