A software bug in a program running in an air traffic control computer grounded all flights across Japan last Saturday morning.
The computer, which handles the distribution of flight information to airports, failed at 7am resulting in the immediate halting of all departures from Japanese airports, said an official of the Ministry of Land, Infrastructure and Transport's civil aviation bureau, which is responsible for the system.
The fault is thought to have been caused by a bug in new software that was loaded onto the machine at 1am the same morning, the official said.
The software handles exchange of data between the control center's Flight Data Processing (FDP) computer and the similar Flight Service and Air Movement Identification System Data Processing (FADP) computer at the Defence Ministry. It was required because the FADP system was upgraded.
The new software worked for the first six hours. However, it failed at exactly 7am, coinciding with the running of a program that collects data on the previous day's air traffic. The ministry is looking into the possibility that an incompatibility between the two programs caused the problem.
One half of the machine, that has total redundancy to guard against hardware failure, began operating at 7.40am. The second half of the machine came back online at 10.58 a.m. However, delays continued all day because of a back-log in the system.
Flights resumed at around 7.30am, although at a much reduced capacity as controllers were forced to use manual methods and communicate by telephone.
The failure caused the cancellation of 192 flights and delayed 1342 flights for 30 minutes or more, according to local media reports. The longest delay was more than 6 hours and the problems allegedly inconvenienced an estimated 270,000 people including 39,000 travellers who were forced to give up their travel plans.
The ministry official said the new software was tested on a back-up system for two weeks from February 1-16 and no problems were observed. However, the official could not confirm whether data exchange with the new Defense Ministry system or the daily data gathering software was part of the test.
The Flight Data Processing computer that failed was supplied to the center by NEC.
The Tokyo-based computer maker declined to comment on the incident.
The incident is the second major computer failure to affect a large number of people in Japan in the last year.
The failure of ATMs, delayed deposits and double withdrawals were some of the problems that grew out of computer glitches following the merger of Dai-ichi Kangyo Bank, Fuji Bank and Industrial Bank of Japan. The banks, which merged to form Mizuho Bank and Mizuho Corporate Bank , ran computer systems from Fujitsu, IBM and Hitachi and the problems were blamed on poor integration of the systems.
As a result of that incident, which lasted for about a month, the head of the bank was called in front of Japan's parliament to explain himself.