Problems flagged months before HPE hardware failure hit ATO systems

Problems flagged months before HPE hardware failure hit ATO systems

At least 77 events related to components that failed in the December 2016 outage were logged months earlier

No ATO access under HPE deal

Meanwhile, the report revealed that ATO IT staff had no direct access to the new infrastructure being operated by HPE under the agreement the ATO struck with its technology partner for its replacement of the pre-existing EMC Corporation SAN with its own 3PAR SAN2 hardware in 2015.

The storage solution provided by HPE to the ATO comprised a primary 3PAR SAN in Sydney with a backup 3PAR SAN in Western Sydney.

Under the deal, the ATO engaged HPE to provide turn-key IT solutions, whereby HPE designed, owned and operated the computing infrastructure and provided services to the “required ATO standard”.

Under the turn-key deal, instead of having direct access to the SAN hardware, the ATO relied upon HPE to provide a full service. The agency also contracted with Leidos as service integrator.

Leidos operates a virtual dashboard over myriad ATO IT systems, and provides a problem management process should issues arise with parts of our IT infrastructure.

While procedures were in place to provide manual fail over for selected applications in the event of a failure, full automated fail over for the entire suite of applications and services in the event of a complete SAN failure in Sydney was not part of the storage solution for the SAN, the report stated.

What’s next?

According to the report, the ATO has developed a new storage strategy to enhance its IT stability and resilience. This involves rebuilding its primary and back up storage systems with the newest technology from the HPE product portfolio working in conjunction with its 3PAR SAN technology.

“All production system workloads are now utilising the enhanced storage system,” the ATO said. “Once data transfer activities are completed, the existing 3PAR SAN will be replaced by a new 3PAR and the current 3PAR SAN decommissioned by late July 2017 for forensic analysis.”

While the ATO is still working with HPE in relation to the storage infrastructure, insights from its experiences relating to the outage will also inform its future IT acquisitions, the agency said.

“As contracts come up for renewal, we need to balance service, stability, resilience and cost,” the ATO said. “Our IT program continues to prioritise government policy reforms and ATO corporate priorities, with a primary focus on another successful Tax Time for 2017.

“Future sourcing of IT is also influenced by whole of government initiatives, including closer collaboration with the Digital Transformation Agency,” it said.

The release of the report comes just days after Australian Commissioner of Taxation, Chris Jordan, told Senate Estimates committee members that the ATO had reached a commercial settlement with HPE, the detailed terms of which are subject to contractual confidentiality.

The move to strike a settlement agreement following the hardware failure and subsequent systems outage echoes the settlement IBM reached with Australian Government over the troubled eCensus project with the Australian Bureau of Statistics (ABS) last year.

Jordan also took the opportunity to reveal the preliminary findings of various investigations into the storage hardware failure last year, citing a combination of factors, suggesting that the design of the SAN infrastructure played a part in its failure.

“The SAN design and configuration meant we had an over emphasis on performance features rather than stability or resilience - a relatively small disk drive failure had a large impact - only 12 of some 800 disk drives failed, but they impacted most ATO systems,” Jordan said.

“The recovery was slower because some of the recovery tools required were themselves stored on the same SAN that failed,” he said.

In early February, Jordan took aim at HPE, suggesting that the agency’s technology partner had “failed” to reliably provide it with its contracted services.

“Initial indications are there has been a failure by Hewlett Packard Enterprise (HPE) to provide contracted services in a reliable way and ensure stability of our systems,” Jordan said in a statement published on 8 February.

Follow Us

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

Tags ATOsan3PARLeidosHPEDXC Technology

Brand Post

Show Comments