Ransomware Damage Report

Good evening, this is Cardy.
I had planned to spend the year-end and New Year holidays doing nothing but chairing, taking advantage of having free time, but due to strong winds and various other excuses, I ended up just enjoying coffee in my garden.
So, what was I doing instead? Well... I was in front of my PC the whole time, writing for the Gijutsu-sho-ten (Technical Book Festival) and writing articles for note. It's the kind of thing I could do even if it weren't a holiday, but I'm enjoying using the ample time I have precisely because it is a holiday lol

Now, last year, ransomware damage at large companies like Asahi and Askul was quite noticeable.
I, too, experienced ransomware damage in 2019. I'll write a short report while recalling those days.

It's a bit long, but I really want you to at least read the final "Lessons Learned" section.

1. Background

At the time, I was working in the IT department of a small manufacturing company. Even though it was small, we had factories overseas, and the situation was that we were connected via a network with Hong Kong as a hub.
Cloud migration was lagging, the "do it on-prem" mentality typical of traditional Japanese companies was rampant, and we were even using Windows servers for our file servers.
For backups, we used an LTO deck added to the file server, with a full backup on weekends and daily incremental backups.

2. It came without a sound

One day when I arrived at the office, a colleague who had arrived earlier complained that "the files won't open." Thinking "Yeah, right," I opened the file server console, and the files that should have been there were gone.
No, the files were there, but they had all been replaced with different file extensions.

"We've been hit!!!"

I thought, and the best move would have been to immediately disconnect all servers and PCs from the network... but my mind was already in a panic.
Disconnecting them would stop factory production. Just looking at the situation in front of me, I was too scared to make that decision on my own...
That day happened to be a management meeting, and the executives, including my boss, were in transit. I forgot to call for instructions and just ran around putting out fires.

3. Expanding screams

I don't remember what I was doing, facing the console, trying this and that. Was I looking for the virus? How did they get in? Was I checking the logs...
Then the internal phone rang. Even though I explained, "It's like this and that...", the phone kept ringing incessantly, so I explained the situation over the PA system. I asked them to disconnect the network, but I wonder how many people actually understood that...

4. Seeking instructions

I called my boss and explained the situation. The management meeting was canceled, and they decided to return.
I couldn't do anything when I was alone and panicked, but perhaps because of the peace of mind that my boss was there, my memory starts to become clear from around this point.
※ At the time, I respected my boss immensely. I still respect them today!

We disconnected the dedicated line to Hong Kong and disabled the core system.
I found out later that it seems they entered through an RDP port that had been opened on the Hong Kong firewall.

Then, we checked the extent of the damage. The file server was almost dead. Other things like over a dozen PCs and data within sub-systems were also hit quite extensively.

At the same time, I worked to pinpoint the virus somewhere inside the file server. I suspected unencrypted executable files outside the system folder, and when I searched for them, I arrived at one file.

"This is it..."

I thought, and when I saved it to a USB memory stick, it would be saved for a moment, but then it would immediately disappear.

"This thing... does it have a self-deletion function?!"

No, there's no way something would have such a function (I don't think so!), but since it actually disappeared from the USB memory, you start to think that such things might exist. In reality,
VirusBuster was detecting and removing it in the background. It seems the pattern file had been updated before I disconnected from the network, and when I realized that it could be removed with the latest pattern file, it was a joy to see a glimmer of hope.

5. Recovery

I understood that updating Virus Buster to the latest version would allow for detection and removal, so I proceeded with that task. However, since the devices were disconnected from the network, I had to use tethering on each terminal to connect to the internet.
However, the computers were desktop PCs. Since they didn't have Wi-Fi, I had to start by getting them to recognize a USB dongle.

Furthermore, due to asset management tools, standard terminals wereset to a state where USB memory and media drives could not be used.
Therefore, I first had to run a command with administrator privileges to enable USB and media usage, then restart. From there, I had to get the media recognized and install the Wi-Fi driver.
Setting up the Wi-Fi and updating the Virus Buster pattern files via tethering wasa very time-consuming taskeven for just one machine.
I had to do this for over 100 PCs, isolate the PCs where viruses were found, and perform a clean installation of the OS.

In this way, I built a clean network connecting only clean terminals. However, even at this point, the route to Hong Kong remained closed, and the core system could not be used (because the server was in Hong Kong).
It seemed that some terminals in Hong Kong had also been infected, so we could not connect until that situation was also clean.

In parallel, I worked on restoring the file server. The file server used LTO6 tapes. However, thanks to Virus Buster providing the pattern files, I held onto a faint hope that maybe the encryption could be decrypted?!
If so, if I specified the current file server as the restoration destination from the backup, I would have to delete the encrypted files to free up space.
Therefore, I pulled out the old file server and attempted to restore to it. However, the LTO deck attached to the old file server was LTO5. It did not support LTO6 tapes.
So, I had to restore files from another server equipped with an LTO6-compatible deck to the old file servervia the network(the old file server had been shut down, so it was uninfected).
Here, the slowness of the network took its toll.The LAN was built on the premise of 100BASEand all cables and switches only supported 100BASE.
Moreover, due to the lack of capacity on the old file server, I could not restore the entire file server at once.

I restored by department (folder), shared them on the old file server for the time being, and waited for good news from Trend Micro.
But no matter how long I waited, no good news came. Eventually, the old file server also became full, and I could no longer restore files.

At this point, I finally gave up on restoring the encrypted files and shifted to restoring from backups.
I rebuilt the file server from the OS installation, returned data from the old file server over the network again, and restored the remaining folders from the backup.

It took3 daysuntil the file server could be connected to the clean network.

Next was the restoration of the subsystems. I requested the development vendor to carry out this work. With the help of support members for the PC restoration, I remember it took about two weeks to restore all the machines.

In parallel, I also proceeded with the restoration of another subsystem that I had built in-house to supplement the core system.
The core system was not infected, but this subsystem was infected, soit was impossible to output forms.
Not being able to output forms meant that shipping instructions to the warehouse and processing instructions to the production floor could not be issued. In other words, there was a risk that production would stop.
Since there were some printed in advance, shipping and production were fine for about two days, but conversely, if I couldn't output them by the third day, the factory line would stop. That was a very bad situation for the factory.

In any case, by performing the restoration work in stages, the line did not stop, but personally, this was the restoration work that made me sweat the most.
After all, since it was developed in-house,there was no one else to rely on...

6. Grasping and Reporting the Damage Situation

Once the restoration was complete, it was time to summarize and report the damage.
Infected servers: several
Infected PCs: over a dozen
Data loss:
- File server: Returned to the state one day before the onset via backup
- Subsystem: Returned to the state one week ago via backup, etc.
Infection route: Thought to be from the Hong Kong FWthought to be
Data leakage:unknown

Data was restored because there were backups. At the time, I didn't know about the 3-2-1 rule, but I was only conscious of "backups being offline" and "storing them at a separate site", and it was a blessing in disguise that the backups were not destroyed by the ransomware.

However, I could not find out anything more about the intrusion route. We had introduced conventional EPP, but EDR that records detailed behavior had not been introduced, and there was no mechanism to centrally manage and collect logs from FWs and routers, so tracking fragmented logs with human eyes was not realistic.
As a result, I was able to confirm the "visible damage" of file server and PC data being encrypted, but I did not know if information had been taken out behind the scenes. Although the data was restored from backups, I ended up finishing the restorationwithout being able to explain 'what happened'.

7. Lessons Learned

Backuprestoration traininghad not been conducted. I had performed restoration operations at the file level, but I had not anticipated a situation like this when trying to restore the whole thing, and it ended up being an ad-hoc response.

I was not conscious of RPO/RTO, and it was nota realistic backup operation.

Thinking "it's fine if it can be restored," only backup support was considered, and there was no mechanism todetect or trackmovements after an intrusion.

Training based on the premise of being intruded upontraininghad not been conducted, and even if I understood the response in normal times, when I actually stood on the front lines, Ipanicked.

8. Conclusion

There is so much to reflect on. However, at the time, it must have been a
painful memory for me. I had actually sealed away this experience to a large extent (and haven't really talked about it in front of others).

Recently, when the topic of security comes up with sales staff or while I'm out, I share this story as a real-life experience. Since it is a vivid account, people have listened quite seriously. I hope that by making my experience public, it will help reduce ransomware damage, even if just a little.

It is a bit further down the road, but I am currently writing a technical doujinshi that focuses on backups. I plan to make it a book that explains the latest backup situation in detail, while also including the vivid experience I wrote about this time as a column, so please look forward to it. *The title is tentative.

For security (SASE), please also refer to this book (^^)/