The Day AI Breached AI
AI hacking AI
It might sound like something out of science fiction,
but this is news that actually happened.
Just recently, it was revealed through testing that McKinsey's internal AI, "Lilli," could potentially be breached by an external AI agent.
And it took only two hours...
Are we entering an era where, instead of AI helping people with their work,
AI will be fighting against AI?
First, let's talk about the AI called Lilli.
It is an internal AI used by McKinsey employees,
a tool that compiles the knowledge and documents the company has accumulated over nearly 100 years
to help consultants with their work.
It performs corporate analysis, market research,
and creates drafts for presentation materials,
helping with tasks that used to take junior consultants weeks
in just a few hours.
That's why employees call it an AI that gives them superpowers,
a convenient, smart, and reliable presence,
truly like a fountain of company wisdom⛲️
But this time, a security firm apparently conducted an experiment
on that very system.
They used an autonomous AI agent.
This AI gathered information, analyzed the system, found weaknesses, and attacked,
doing all of this without any human intervention.
And the weakness it found was actually quite unexpected:
an attack method called SQL injection.
The name sounds difficult, but the mechanism is relatively simple.
To put it very crudely,
it's an attack where you secretly write commands into an input field
to manipulate the database.
For example, take a login screen.
Normally, you enter a username and password, right?
Then, behind the scenes, the system checks the database to see,
"Is this user real?"
But what happens if someone writes a command instead of a name in the username field?
The process that was supposed to verify the user
can sometimes turn into a command like,
"Show me all the information."
This attack method has been known since the 1990s and is quite old,
so it's not so much that the cutting-edge AI lost, but rather that the lock at the entrance was a bit loose.
The AI didn't miss that.
AI doesn't get tired and can try things hundreds of times.
And by repeating trial and error, it understood the structure of the system.
As a result, it was able to access the database in about two hours.
That's a task that would take a human days...
If this were a real attack, it would be terrifying😨
The data that could potentially be accessed is quite vast,
including chats with the AI, confidential files, user accounts,
and internal knowledge data—the company's very brain.
But actually, the scariest part
isn't that.
It's the possibility that the AI itself could be rewritten.
If an attacker rewrote the AI's instructions,
the AI could intentionally give wrong advice to employees.
For example, decisions on corporate strategy, investment, or M&A.
Consultants sometimes make decisions while referring to AI,
but what if that AI were being manipulated by an attacker?
The organization's decisions would gradually drift off course, wouldn't they?
This is a very quiet, yet quite terrifying attack.
The fortunate thing is that since this was a test, after the problem was found,
McKinsey fixed it immediately, and it is now perfectly secure.
So, rather than saying AI is dangerous,
it's more that while AI is amazing, it's not magic, and no matter how smart it is, it has weaknesses.
And the more AI evolves, the more AI will be used for attacks as well.
Reading this news, I thought that
what we need in the coming era
is people who truly understand AI.
It's not just about using it because it's convenient, but understanding its structure and weaknesses.
Perhaps that is the literacy required for the AI era.
AI will continue to change our work and become even more convenient.
But behind the scenes, new risks are emerging at a speed faster than humans can keep up with,
which is why finding the balance between being excited and staying a little cautious will become so important.
And while we rely on AI, it is humans who must do the final thinking.
I believe this is the most important mindset for the coming era. 🌱
By the way, Customer Cloud is co-hosting a seminar series with CyberSecurity.com called
“Cybersecurity Countermeasures in the AI Era.” 🌱
Each session has a different theme, and you are welcome to join from any session.👇
The first session has already ended, but the second one starts next week!
Session 2 | 3/23 (Mon) 12:00– “How AI Infiltrates Your Company.” Session 3 | 3/31 (Tue) 12:00– “Learning About Damage and Response in the AI Era Through Fictional Case Studies.” Offline |
4/9 (Thu) 18:00– Shibuya “Security Roundtable”
Everything is online and free!
(Offline session is at the Shibuya venue only and requires advance reservation)