A storm has caused an outage at Internode’s Adelaide datacentre, which affected services in South Australia.
According to notes provided on the ISP’s support website, the ADL6 datacentre facility encountered a mains failure at 2.37pm Australian Central Standard Time (ACST) on Monday September 21. UPS power was used after issues with emergency power generators, which kicked in automatically.
“Despite being regularly tested, one of the backup generators did not sychronise and was not able to contribute to power demand during the mains failure,” Internode managing director, Simon Hackett, said.
Once UPS battery power was exhausted, some customer colocation server racks were powered down to lower the strain on generator capacity. Power was unavailable to the subset of customers servers affected for about 30 to 60 minutes.
At 3.35pm ACST, Internode began repairing the power systems to colocation customer equipment with mains power restored by 3.59pm ACST. All datacentre services were restored 30 minutes later.
Internode engineers remain onsite to monitor and investigate the situation. The ISP will conduct tests this week to ensure backup power capacity is fully operational.
“Internode sincerely apologises for the incident and will take all appropriate steps to prevent any recurrence,” Hackett said.
More information was provided by Internode on September 22:
- The generators have an output circuit current limiting system to protect them, which had been installed in the datacentre with an incorrect power cutoff configuration. This fault has been present, and undetected, since the data centre generators were first commissioned.
- The fault would have been detected and fixed at the next regular scheduled full load generator test in November but it problem only became apparent after the storm hit the facility.
- Configurations were corrected today and the datacentre is running at full load on generator power.
- Internode does not expect a repeat of the generator fault.