Toyota Australia rebuilt IT from incomplete info after cyber attack

Toyota Motor Company Australia rebuilt its IT environment after a 2019 cyber attack without the aid of a central list of all its IT assets and how they were interconnected, because the system used to hold that data was “vanilla” and incomplete.

IT infrastructure manager Michael Mirabito told ServiceNow’s Knowledge 2021 conference that the carmaker was in the process of rebuilding its IT helpdesk systems and configuration management database (CMDB) when the attackers struck.

The impetus for rebuilding IT service management (ITSM) systems came years earlier.

When Toyota closed its Australian manufacturing operations in late 2017, it also moved its IT  support “from more of an insourced type model to an outsourced model,” Mirabito said.

A managed services provider was appointed that used its own proprietary – but basic – ticketing system.

Toyota decided not to renew the contract, and appointed another provider in their place.

“The old vendor wasn’t happy about not renewing the contract and it was a very quick exodus,” Mirabito said.

“They refused to stay longer than two to three months, and it was pretty much, ‘That’s it. We’re gone at this point, whether you like it or not’.”

At that point, Toyota decided to stand up its own ITSM platform in ServiceNow, but with only three months, which fell over year-end holidays, the company had to “make some pretty difficult decisions … on what was critical and essential” functionality, and what had to be skipped.

“The best way for me to describe [the result was] a very ‘vanilla’ build – very plain, very, very basic,” Mirabito said.

The CMDB – which acts as a central list of IT assets and how they are interconnected – was a casualty of the rush, and was still under repair when attackers struck.

“We had a pivotal moment a few years ago, where we had a cyber incident, and we had to proactively pull down our network essentially and rebuild,” Mirabito said.

“I won’t talk about those times, because I remember the long days and lack of sleep. We did a really good job from an IT perspective to get it up and running really quickly but it was painful.

“And I can tell you now, it made us realise how important the CMDB is. We wished that we had a better CMDB at that point because it would have made that rebuilding process better.

“Unfortunately, because we didn’t, we had unknown infrastructure out there, we had apps and services that we didn’t know how they connected together, and knowledge within the business had been lost over time.

“We had to just scramble at that point and work as well as we could together to rebuild and get the information that we needed.”

The recovery led IT to servers it didn’t know existed, and to repair systems that had been long-forgotten by the people that originally set them up.

“If you can imagine after an incident, what really becomes evident is this has been down now for say, a month, we’ve rebuilt [it] but we didn’t know it was there. Is that actually needed?” Mirabito said.

He said the company had since benefited substantially from service discovery and mapping: firstly, finding IT assets it wasn’t previously aware of, and then mapping how they connected into other systems and processes.

The company also turned on software asset management (SAM) to keep track of paid licences and to challenge users whose licences sat unused for an extended time.

“We could immediately see who was using licences and who wasn’t, and more importantly we could see that people hadn’t used licences in two years, yet we were still paying for it,” Mirabito said.

“We saw multiple versions of software that were out there, so we were able to target upgrades and patches to ensure that everyone was on the same version of the different pieces of software.

“We also identified software that people shouldn’t have even had on their machines, and we were able to immediately save money because we removed licenses that weren’t being used.”

Mirabito said that workflows were created to automatically challenge users if a licence sat unused for six months; if that continued for an additional six months, the licence is automatically repatriated.