Flashback to the 1980s, when this big bank's main data-processing site is underground and takes up most of a city block, according to a pilot fish on the scene.
"It had armed guards, with power supplied from two separate electrical lines coming from different generating plants, plus emergency generators on the roof," says fish. "All underground floors were waterproofed, with pumps rated to handle flooding.
"The entire computer room was covered with a full halon fire-suppression system, and the fire and security alarms were direct-wired to the local police and fire emergency center, with a link to the local FBI office. It was really quite a state-of-the-art site, and all tested on a regular schedule."
And on the wall, right next to the exit at about eye-level, there's a big, bright, two-inch red push switch, labeled in large letters "Emergency Shutdown." It's designed to be easy to hit with a fist as the computer operations staff evacuates the room before the halon system displaces all the oxygen in the room.
But one day, there's an outside audit of the bank's procedures. The auditor is admitted to the data center. He's shown around, checks the systems and verifies the logs of tests.
After checking all that out, he's escorted to the exit. As he's about to walk out, he notices the Emergency Shutdown switch. He hasn't tested that.
So he reaches out and hits it.
"Power was cut off to all the computer hardware immediately," fish reports. "All the lights went off and the emergency exit lighting came on. Alarms started ringing throughout the building. Hundreds of programmers on upper floors lost whatever they were working on, and they started to evacuate the building."
As they're leaving, fire trucks and police cars are arriving with sirens blaring -- and the Federal Reserve Bank a block down the street goes into precautionary lockdown.
Meanwhile a few thousand ATMs lose access to account records, and suddenly have to limit cash withdrawals to $50. Bank tellers in three states lose access to account information, and have to switch to fallback procedures until they can be reconnected to one of the live alternate sites.
It eventually takes more than 30 hours to repair machine damage caused by the abrupt power loss -- and IBM has to pull in field engineers from all over the state and fly in parts from three states away.
"On the other hand, the Data Processing emergency procedures passed the audit easily," says fish.
"The next day, the emergency shutdown switch was behind a glass panel, with a hammer chained to the wall for breaking the glass.
"And the next year, the company had a different auditing firm."
In case of emergency, tell Sharky.Actually, it doesn't take an emergency to send me your true tale of IT life at email@example.com. You'll snag a snazzy Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.
Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.