Telstra blames missing software update and undocumented design change for massive outage

Telstra CEO Vicki Brady is due to appear before a Senate inquiry to be questioned about a massive outage last week. Photograph: Dean Sewell/DEAN SEWELL View image in fullscreen Telstra CEO Vicki Brady is due to appear before a Senate inquiry to be questioned about a massive outage last week. Photograph: Dean Sewell/DEAN SEWELL Telstra staff unaware of mass outage risk as critical software failure ‘rippled slowly across the network’ Telco outlines reasons for network failure that hit mobiles, trains and retailers ahead of appearance at Senate inquiry Follow our Australia news live blog for latest updates Get our breaking news email , free app or daily news podcast Telstra has blamed the lack of a software update for a key time-keeping system for outage that caused nationwide chaos last week, with its maintenance teams also unaware of a design change that affected how it would reset. Ahead of an appearance by its chief executive, Vicki Brady, at a Senate inquiry into the mobile outage on Friday, Telstra provided a written submission that revealed the cause. In it, the company stated it did not lack redundancy in its network – but that such redundancy did not prevent the outage. The submission confirmed reporting that one of Telstra’s network time protocol (NTP) servers, designed to ensure the systems had the correct time, had reset to be “2006 ”. Telstra said it has three NTP servers, in Melbourne, Sydney and Perth. During maintenance, the company shut down and restarted the server in Melbourne. Due to an “underlying software configuration” within the server, it restarted with the wrong 2006 date. “Over the next few hours, the incorrect date rippled slowly across the network, causing authentication certificates in other servers to become invalid,” Telstra said. “Customers were intermittently unable to authenticate onto the network (‘no service’), which affected their ability to place voice calls and use data across Telstra’s mobile network.” Sign up for the Breaking News Australia email Telstra said it had made an intentional design change to the equipment to fix an earlier fault – but that this had not been properly documented. This meant maintenance workers arriving at the Melbourne site early last Wednesday were not aware of how the device would be reset. A software update had also not been applied to the device, and if that had been done, the outage may not have occurred, Telstra said. Why pay a premium fee for a service that isn’t? How the nationwide outage could hurt Telstra Read more The submission also said when the Melbourne NTP server disconnected for maintenance, the other two worked as redundancy and backup as expected. “The failure mode here was not inherently related to hardware, levels of redundancy, or the architecture of our network,” Telstra said. But when the Melbourne server supplied an incorrect date once switched back on, “downstream systems used that date in security, authentication, session and policy-control processes.” “Th