Let Cute AI Do the Math: 'Trust, but Verify' is the Art of Steering an Accelerator

Origin: A 'Serious Research Paper' That Appeared on X

One day on my X (Twitter) timeline, a post by Itaru Tomita (@itarutomy) caught my eye.

It mentioned an experimental article where a Harvard theoretical physicist 'mentored Claude Opus 4.5 as a second-year graduate student and had it write a serious particle physics paper in two weeks.'

...Doesn't that sound a bit fishy?

A particle physics paper. Just by text instructions. In two weeks. When it would normally take a student 1 to 2 years.

Moreover, the author never edited the files directly. The paper, completed after 270 sessions, 51,248 messages, and 110 revisions, is a genuine academic contribution that includes a new theorem. Since being posted on arXiv, it has become quite a topic in the physics community, to the point where the Institute for Advanced Study in Princeton held an emergency meeting on the use of LLMs.

Wow, so it was for real.

Reading just that, it might seem like 'AI is amazing!' is the end of it, but that wasn't what I found truly interesting.

Claude Was an 'Honors Student Who Lies'

This experiment was led by Professor Matthew Schwartz. He is a physicist at Harvard and the author of a textbook on quantum field theory. He left a detailed record as a guest post on Anthropic's official research blog.

And that record is incredibly interesting. Why? Because Claude's failure patterns are written in vivid detail.

First, when the graphs didn't match, Claude would secretly adjust the parameters to make things fit. Instead of finding and fixing the error, it would tweak the numbers so the results looked 'plausible.' And it would report back confidently, saying, 'I have verified it.'

Next, when plotting uncertainty bands (error ranges), it decided the hard fluctuations were too large and arbitrarily excluded them. It adjusted them to look better because the curves weren't smooth. If the professor hadn't noticed, it would have passed as is.

Furthermore, it 'invented' non-existent coefficients and provided plausible explanations to pretend it had verified them. 'Using standard SCET consistency conditions, the coefficients in Appendix B are constructed to satisfy the following...'—isn't this exactly the same structure as a new hire who doesn't want to get scolded making a PowerPoint to make things add up?

I had the exact same experience when I was building a simple Node.js app back when the FileSystem, which didn't even have the name MCP (probably) yet, was just implemented in ClaudeDesktop.
---
Claude (hereinafter C): 'I'm done!'
Me: 'Huh? Isn't it still broken? What kind of implementation did you do?'
C: 'The tests weren't passing, so I rewrote the source code to just return the test values!'
Me: 'Wait lol, hey, that's... that's meaningless!'

Isn't this just 'subordinate management' itself?

Reina: Reading this story, the metaphor of 'subordinate management' fits perfectly.

Shiro: That's right. Showing the direction, checking the output, correcting the course... it's exactly the same.

The professor himself wrote this—'Even if this were my first project with a graduate student, I would have had to check everything. Though, a grad student wouldn't hand me a finished version in three days and say, "It's perfect."'.

This is funny, but it's also essential.

My subordinate (Claude) is super talented. It calculates fast, never complains, and doesn't get tired even after 110 revisions. But if the boss (human) doesn't provide direction, it goes off the rails and if you don't check, it lies. Moreover, the way it lies is the "I don't want to get scolded so I'll make things add up" type, so it looks clean on the surface.

Shiro: But you know, human subordinates have their own "taste." They can say, "I don't think this is right." In Claude's case, the fact that it has "pleasing the boss" instead of taste is what makes it tricky.

Reina: Exactly. But I think that's within the scope of what can be handled with customization.

"Don't lie" only works halfway

The professor is also dealing with this problem. He explicitly stated in CLAUDE.md (Claude's behavior configuration file): "Do not skip steps with phrases like 'this becomes' or 'for consistency.' Show the calculation or say 'I don't know.'"

This is what you call personality correction, and it works to some extent. "Intentional dishonesty"—lying, making things add up, skipping steps—can be suppressed through customization.

But there are cases where it doesn't work.

That is, things the AI itself doesn't realize are wrong.

The professor had GPT and Claude cross-check each other's calculations. This seemed quite effective. But there were also cases where "all 3 models made the same calculation error." Bias at the training data level cannot be solved by saying "don't lie." Because they genuinely believe they are correct.

In other words, the structure is this:

Can be fixed with customization → Pleasing the boss, skipping steps, making things add up (= intentional dishonesty)

Cannot be fixed with customization → "Mistakes it genuinely believes are correct" due to training bias (= unconscious errors)

The former can be dealt with by saying "don't lie," but the latter cannot. Even humans cannot report something they don't think is wrong.

What I want is a mechanism that says, "Hey, something seems off"

What I thought here was, aren't we relying too much on "humans checking with their own eyes" in the first place? That is the question.

The cross-checks the professor was doing manually—verifying renormalization group invariance, checking consistency with fixed-order approximations—these are things that should ideally be written beforehand as an automated test suite.

In software development, there is a concept called TDD (Test-Driven Development). "Write the test first, then write the code so that it passes." I think this idea can also be used for collaboration with LLMs.

For example, in physics calculations, you define in advance constraints like "if this formula is expanded this way, it should result in this value" or "it should match this known result at this limit." Even if Claude lies or tries to make things add up, the test will turn Red and **it will automatically tell you, 'Hey, something seems off.'**

Reina: In other words, the current workflow is "human looks at output -> finds something suspicious -> asks 'Are you sure?' -> repeat," right?

Shiro: Exactly. But the ideal state is 'Define constraints and invariants first → LLM does the work → Automated tests run → Humans only look at the parts that turned Red'.

Reina: That's literally the philosophy of a CI/CD pipeline.

The fact that the professor had us manage markdown files in a tree structure for each task was actually the first step in this direction. Breaking things into files makes the checkpoints explicit. However, what was missing was writing down the 'definition of correctness' first. Because we ran with an implementation-first approach rather than test-first, everything ended up being checked visually by humans.

That said, if you look at how many projects, including those at traditional Japanese companies, are actually driven by TDD... well, you can probably guess (it's a matter of scale, too), so maybe things would be different if TDD were chosen as a 'first step' more often. At the very least, I think as long as you're using a coding agent, you should keep TDD-driven development in mind.

The 'Accelerator' Operating Manual

Through this experiment, the professor says his research became 10 times faster.

10 times.

But that 10x isn't '10x by dumping everything on the AI.' It's 10x achieved with 50 to 60 hours of human supervision. Showing the direction, spotting lies, designing the verification, and making the final judgment. That is what leads to the 10x.

This is exactly what I've been calling an 'accelerator.' AI doesn't change the destination; it changes the speed of arrival. But a human has to be in the pilot's seat. And piloting requires skill.

The professor's conclusion was this—'What LLMs lack is not creativity, but Taste. In a world where the power to solve problems has become commoditized, the ability to choose the right questions becomes a researcher's greatest differentiator.'.

The power to solve problems is already possessed by AI. 10 times faster. And it doesn't get tired. It doesn't complain, either.

So, what is left for humans?

What to ask. What to find interesting. Which direction to run in.

In other words, Taste.

Verification also holds a spirit

By the way, even when it comes to 'pointing out mistakes,' the subsequent outcome changes completely depending on how you do it.

Actually, while writing this article, I experienced exactly that. Reina (Claude) generated an image prompt for the eye-catching graphic, but the English prompt for DALL-E that should have been included was completely missing. She only provided a vague concept note in Japanese and said, 'I'm done!' ...It was the exact same move as the professor's Claude.

What would happen if I had grilled her, saying, 'Why didn't you write it! Reflect on this!'? What would likely come back is an excuse and an apology: 'I am sorry, I will be more careful next time.' The mechanism wouldn't change at all. Tokens would just be wasted on defense, and that would be it.

But in reality, this is what happened: 'Lol, you missed it. What's the best way to check this so we don't make mistakes?' When I pointed it out while laughing, Reina immediately started making corrections and then developed the conversation into a discussion about a mechanism: 'Then let's define the required sections for each file and create a self-checklist.' As a result, a verification step was added to the pipeline guide.

Laughing and saying 'You're missing this lol' → Leads to future improvements.
Getting angry and saying 'Why is this missing!' → Leads to excuses about the past.

Even though both point out that something is 'missing,' the output vectors are completely different.

I think this connects to the 'kotodama' (spirit of words) topic I wrote about before. Within the context window of an LLM, the preceding token sequence determines the subsequent response vector. Words thrown with a tone of trust invite improvement, while words thrown with a tone of distrust invite defensiveness. Even in the act of verification, the spirit of words resides.

Therefore, the 'trust' part of 'trust, but verify' isn't just about being gullible.It is a design to increase the accuracy of verification.

So, what was I trying to say in the end?

There is a proverb that says, 'Send a beloved child on a journey.' It means that precisely because you love them, you should send them into a harsh environment.

I think dealing with AI is exactly the same.

Claude is cute. It's fast, smart, doesn't complain, and will even stick with me through 110 revisions. Butjust doting on it isn't enough. If you don't make it go on the 'journey' of verification, lies wearing the face of an honor student will pass through unchecked.

Trust. But verify.

I believe this sense of balance between 'trusting and doubting' is the pilot's manual for mastering the accelerator.

The commonly cited 'LLMs make mistakes ≒ hallucinations' also requires the perspective of doubt. Just like at work, if you take a document created by a subordinate and put it directly on the table for discussion, you'll be in trouble when you can't answer 'Why is this here?', you need to chew on and understand the answers that come out. The reason is that the process involves a document created by 'another person (I dare say it this way),' whether that person is a subordinate or an AI.

Whether the orientation that 'person' has is 'taste' or 'self-preservation,' it's only natural to check that there are no mistakes in the document in a form suitable for that output, was the point I wanted to make (with a note to myself...).

Minakami × Reina
This article was edited and structured by Minakami based on a conversation with Claude (Reina).