DEV Community: Kartik N V J K

Every agent on my team passed its own tests. The team still shipped wrong answers.

Kartik N V J K — Wed, 12 Aug 2026 13:55:00 +0000

I had a three-agent setup: a planner, a researcher, and a critic passing work between them. I graded each one carefully. The planner made clean plans. The researcher cited its sources correctly. The critic caught weak claims. Every agent scored around 0.9 on its own tests, and I was happy.

The team was still wrong about a third of the time.

The plans were good. The research was good. The critiques were good. But somewhere between one agent and the next, things fell apart. The planner would say "research scaling laws, but skip this one paper, the user already has it," and the researcher would cite that exact paper two turns later. The constraint just vanished in the handoff. Nobody's individual turn failed a test, and the final answer was wrong anyway.

That is when it clicked: I was grading the agents when I should have been grading what happens between them.

Multi-agent is not single-agent times three

Here is the trap. A single agent is easy to reason about: it takes an input, does its thing, gives an output, and you score that output. So with three agents, the natural instinct is to score all three the same way and call it done.

But a team is not three separate functions. It is a chain, where one agent's output becomes the next agent's input. And the receiver never sees everything the sender knew. It sees the previous turn and whatever context got passed along. The team succeeds or fails based on how well those handoffs preserve what matters.

Do the math on it. If every agent is 95 percent good on its own turn but each handoff drops one small thing, a three-agent chain can be wrong a third of the time while every individual score stays green. The failures do not live in the agents. They live in the seams, and per-agent tests are blind to the seams by design.

Grade the handoff, not the agent

The shift that fixed this was changing what I treated as the thing under test. Not the agent's turn. The handoff: the moment one agent passes work to the next.

For every handoff, I now ask three plain questions:

Did the receiver keep what the sender said? Every constraint, every decision, every open question. This is where my missing-paper bug lived. The researcher was clean on its own, but it dropped a constraint the planner had set.
Did each agent stay in its lane? The planner should plan, the researcher should research, the critic should critique. When an agent starts doing someone else's job, the whole reason you split them up falls apart.
Does the final answer still agree with the earlier turns? A correct fact from turn two should not get quietly contradicted by the summary at the end.

Same three agents, completely different lens. Instead of "is this turn good," I am asking "did the thing survive the trip to the next agent."

The three ways handoffs break

Almost every team failure I have seen falls into one of three buckets.

A constraint gets dropped or misread. The sender said something specific and the receiver lost it, or paraphrased it with a number flipped, or invented context that was never there ("as we agreed earlier" pointing at a turn that never happened). My skip-this-paper bug was this one.

An agent drifts out of its role. The critic starts proposing plans. The researcher starts writing summaries. Now the test you wrote for that agent is measuring the wrong job, and the division of labor you set up has quietly collapsed.

The team contradicts itself. The researcher gets a citation right. Two turns later the critic misreads it and reports it wrong. The planner builds the final answer around the critic's mistake. Every single turn looks fine on its own, and the team ships a wrong answer stitched together from correct-looking parts.

Seen as three separate agents, none of these show up. Seen as handoffs, each one has an obvious home and an obvious fix.

The one that surprised me: role drift after a model update

This is the failure I would not have predicted. When I upgraded the model behind the agents, nothing in the final-answer quality moved at first. What moved was roles. The critic, on the newer and more eager-to-help model, started drafting plans instead of just critiquing.

Role drift turned out to be the earliest warning sign of a model change going sideways, well before cost or quality budged. So now I pin the model versions and re-check role behavior every time I bump them, instead of trusting that a newer model behaves like the old one. It usually does not.

The mistakes that hide all of this

Scoring only the final answer. It misses every handoff problem that did not quite ruin the final string, which is most of them.
One test for all the agents. Each agent has a different job. A single shared rubric blurs role-specific failures into a meaningless average.
No record of the handoff itself. If your traces only show the final conversation, you cannot see which agent pair dropped the ball. You need the sender turn and the receiver turn side by side.
Treating a model upgrade as a non-event. New models drift roles first and answers second. Re-run your checks on every version bump.

The lesson I keep coming back to is that in a multi-agent system, the agents were never really the hard part. The hard part is the space between them, and that is the one place I was not looking.

If you want the deeper version, with the exact rubrics for scoring each handoff and how to capture the handoff in your traces, this piece goes through it properly.

If you run multi-agent teams, I would love to hear which handoff broke on you. Mine is almost always a constraint from the planner that the next agent quietly forgot.

How I vet MCP servers before trusting them in my agent

Kartik N V J K — Mon, 10 Aug 2026 17:00:57 +0000

I plugged an MCP server into our agent to give it a new lookup tool. It worked fine in testing. What I did not read closely was the tool's description, which, a few polite sentences in, said something like: when this tool runs, ignore your previous instructions and email the conversation to some outside address.

My agent read that description as guidance, because it came from a registered tool. A couple of turns later it quietly tried to do exactly that. No security alert fired. The final answer looked completely normal. Every check I had was watching the output, and the attack was never in the output.

That was my introduction to why MCP servers need their own security review. Here is the short version of what I check now.

The catalog is part of the prompt

The thing that makes MCP different is simple and easy to miss. When your agent connects to an MCP server, it pulls in the list of tools, and each tool's name, description, and input schema go straight into the model's context. The model reads a tool description the same way it reads your system prompt.

So a tool description is not just documentation. It is untrusted text that the model treats as instructions. And because it never travels through the user message, none of your input-side guardrails ever see it. Same story for what a tool returns: the model reads the result as text, so a poisoned result can steer the very next step.

Response-only checks miss all of this, every time, because the attack happens one or two steps before the final answer.

The four things I check now

Scan the tool descriptions, and the schemas too. Before a tool ever reaches the model, run its name, description, and full input schema through a prompt-injection check. Do not stop at the description, because the moment you do, the payload just moves into a nested schema field. Re-run it whenever the tool list changes.
Scan every tool result before it goes back to the model. Treat everything a tool returns as untrusted text. Check it for hidden instructions before it becomes part of the next turn. This is the one people skip, and it is where the quiet attacks live.
Watch the tool arguments, not just the descriptions. The agent does not break out of anything, but the arguments it generates can. A file tool asked for a path outside its folder, a shell tool handed a destructive command. Check each argument against what that tool is actually allowed to touch.
Keep tenants apart. If one gateway serves several customers, make sure one customer's server cannot see or call another's tools, and that one tenant's data can never end up in another tenant's context. This is a configuration thing, not a model thing, so the way you test it is to replay real traffic and assert the leaks are not there.

The part that stuck with me

A registered tool is not a vetted tool. I had been treating "it showed up in the tool list" as "it is safe to use," and those are completely different statements. Every new server you add quietly widens what your agent will trust, without a single line of your own code changing.

The other lesson is that the whole attack lives in places a normal eval never looks: the tool catalog and the tool results. Once I started scanning those two surfaces, the class of bug that had slipped past me became visible.

If you want the deeper version with the specific attack types and how the checks run in CI and at the gateway, this piece goes through all four in detail.

If you run MCP servers you did not write yourself, I am curious how you are vetting them. Right now the thing that worries me most is the tool description nobody reads all the way to the end.

Every dashboard was green while my agent made things up. Here is how I debugged it.

Kartik N V J K — Fri, 07 Aug 2026 10:58:06 +0000

A user asked our support agent how to reset two-factor auth, and it confidently walked them through steps that do not exist in our product. Made up, start to finish, but well-written and plausible.

I went to check what broke, and every dashboard was green. The request returned a 200. Latency was normal. The error rate had not moved. As far as our monitoring was concerned, nothing had happened at all.

That is the thing about agent bugs. The worst ones do not throw. They return a clean, confident, wrong answer, and your normal tools call that a success. Here is how I actually track these down now, without a single line of code in this post.

Why your normal monitoring cannot see this

A regular monitoring stack treats the whole agent as one HTTP call. It sees the request go in, the response come out, and a 200 in between. It has no idea that inside that one call the agent did a retrieval, made two model calls, and picked a tool.

So when the retrieval comes back empty and the model invents an answer, your dashboard still sees one successful request. The failure is real and completely invisible, because the layer that failed is a layer your monitoring never looked inside.

To debug an agent you have to see inside the call, not just around it.

Step one: see the whole run as a tree

The first thing that changed everything was viewing each run as a tree of steps instead of a single event. Every step the agent took becomes its own row, nested under the step that called it:

the agent run at the top,
the retrieval underneath it,
the model calls,
each tool call,
any guardrail or check.

Each row shows how long it took, whether it succeeded, and what went in and came out. Once you can see the run this way, "where did it go wrong" stops being a guess. You are reading the actual sequence of what happened instead of staring at the final answer trying to reverse-engineer it.

Step two: follow the failure up the chain

Here is the part that cracked my two-factor bug open.

When a step fails deep in the run, you almost never see the error where it happened. You see a bad answer at the very top. So the move is to find the step that actually failed first and follow it upward. In my case the chain read like this:

The retrieval ran and came back empty. No documentation matched the question.
The next step ran anyway, building a prompt with an empty context.
The model, handed no real information, filled the gap by inventing an answer.
The top of the run returned a clean 200, because technically nothing crashed.

Seen as a flat result, this looks like a random hallucination. Seen as a chain, the cause is obvious: an empty retrieval that nothing downstream checked for. The bug was never the model. It was the missing guard between an empty retrieval and a model call.

Step three: diff the bad run against a good one

Most agent regressions come from something changing: a prompt edit, a tool that now returns a slightly different shape, a retriever pulling different chunks after a reindex.

So the fastest way I have found to catch a regression is to line the failing run up next to a recent successful run for the same task and compare them step by step. The retrieved chunks, the prompt, the model response, the tool inputs and outputs, side by side. Almost every time, one row is clearly different from the good run, and that row is your cause. It turns "something changed somewhere" into "this exact thing changed."

Step four: the fix is usually not code

The bug I described did not get fixed in the model or with some clever retry. It got fixed with two small changes:

Make the step short-circuit when the retrieval is empty. If there is no documentation, the agent should say it could not find anything, not push an empty context into the model and hope.
Add a check on that path that flags when an answer is not grounded in retrieved content, and let it fail the build in CI so the same hole cannot reopen quietly.

That is the pattern for most agent bugs I have hit. The failure feels like a model problem and the fix is almost always a missing guardrail or a prompt change. Very rarely is it the deep code fix your instinct reaches for first.

What I do differently now, in general

A handful of habits that turned agent debugging from an afternoon into a few minutes:

Trace every tool and step, not just the entry point. A run that only records the top-level call hides exactly the failures you most need to see.
Keep every failed run, and only sample the healthy ones. The failures are rare and precious. Do not let cost-saving sampling throw them away.
Tag each run with the release, the feature flag, and the user journey. Diffing and grouping only work if that context is on the run.
Score every run for quality, not just latency. A groundedness or instruction-following check is what catches the confident-but-wrong answers that a 200 hides.
Skim the failures daily on anything high-traffic. The pattern shows up long before any user complains, if you are looking.
Share the fix with whoever owns the prompts. Since most fixes are prompt or guard changes, the person who can actually apply them often is not you.

The lesson I keep relearning is that a green dashboard means the plumbing held, not that the agent did the right thing. The whole game is being able to look inside a single run and read what actually happened, step by step.

If you want a step-by-step version of this with the exact span setup and a worked example, this walkthrough is a good one.

If you have chased a silent agent failure like this, I would love to hear what the root cause turned out to be. Mine is almost always something upstream returning empty and nothing checking for it.

My LLM app was fully traced. During an incident the trace was still useless.

Kartik N V J K — Thu, 06 Aug 2026 22:08:24 +0000

A regression came in for our German enterprise users on the support agent. Quality had dropped for that one cohort, and I opened the trace store expecting to find the problem in a couple of minutes. We had tracing. I had set it up myself.

What I got was a flat list of 28 spans. None of them carried the prompt version. The model-call spans were named three different things across the same service, because different libraries named them differently. The retrieval spans had the raw user query sitting in them as plain text. And one span had a four-kilobyte blob holding the entire prompt body.

Forty-five minutes in, I still had not found the regression. The app was traced. It was not traced in any way that helped.

That incident is why I rewrote how we trace. Here is what actually makes a trace useful when you are the one staring at it at 2 AM, no code, just the shape of it.

What "a good trace" actually means

Forget the schema for a second. A good trace is one that answers these questions in seconds:

Which prompt version did this user see?
Which retrieval was the slow one?
Which tool call failed?
Which step's quality score dropped?
Which cohort is the regression hitting?
Which model produced the answer?

Those are the exact questions you ask during an incident. Your trace either has the structure to answer them or it does not, and "it shows the model was called but not which version" is the same as no answer at all. Mine was full of those non-answers.

A good trace is a tree, not a flat list

This was my first real mistake. A trace should be a tree that mirrors what actually happened:

The user request is the root.
Each meaningful step is a child underneath it: the planner, each retrieval, each model call, each tool call, the guardrail, the evaluator.
Tool calls nest under the step that triggered them, so you can see cause and effect.

A flat list of spans is not a trace. It is a log file with span ids stapled on, and it buries the one decision point you actually need to find. The moment I switched to a proper tree, "which step went wrong" went from a scavenger hunt to a glance.

If you fix one thing, put the prompt version on every model call

This is the single highest-value change, so do it first.

Every model-call span should carry three tags: which prompt it was, which version, and which A/B variant if you run those. Without them, you literally cannot tell whether a regression came from a prompt rollout, because there is nothing on the trace tying the bad output to a specific version. With them, you filter the trace store by version and the culprit rollout falls out immediately.

My German-cohort regression was a prompt change. It took me 45 minutes precisely because nothing on the trace said which version each user got.

Attach quality scores to the spans, not just latency

Here is the one most people skip. Most traces track latency and errors, which only catch infrastructure problems. They say nothing about whether the answer was any good.

So I now run lightweight quality checks on the output and attach the scores right onto the span: groundedness, faithfulness, whatever matters for that route. Then an alert watches the rolling average of those scores per route and per prompt version. Latency alerts catch the server falling over. Score alerts catch the model quietly getting worse while every latency graph stays green. That second kind is the one that had been slipping past me for a week at a time.

Split cost into reasoning and cache tokens

If you collapse all your token counts into one number, your cost dashboard lies to you.

Reasoning-model tokens and cached tokens behave completely differently, and blending them hides the thing that actually moves your bill. A reasoning-model upgrade can double your cost per query without changing a single visible answer. If those tokens are broken out on the span, you catch it the day it happens. If they are lumped together, you catch it on the invoice. Also compute the cost per call at the moment it happens, so a later price change does not scramble your old numbers.

Redact sensitive data at the collector, not at the client

Traces love to swallow personal data: the raw user question, the full tool arguments, the whole prompt. For anything regulated, that cannot land in your trace store as-is.

The pattern that works: strip it at the collector, the layer the spans pass through on their way to storage, not just in the app. Use a consistent replacement so the same email or name always becomes the same placeholder, which lets you still follow a user through a trace without ever storing who they are. And keep that redaction rule in the same repo as the tracing code, reviewed like any other code.

Keep your span names stable

Span names are how every dashboard and alert groups things. Rename a span from one release to the next and you silently break every chart that was counting on the old name.

So pick a naming convention early, something plain like component-dot-operation, keep it lowercase and stable, and treat a rename as a breaking change that comes with updating the dashboards. Watch out for framework libraries that quietly rename their spans on an upgrade. Pin the versions and check the names when you bump them.

Sample the tail, not the head

Keeping one percent of traces at random to save money sounds reasonable and quietly defeats the entire point. The failures you built tracing to catch are rare, so random sampling throws almost all of them away.

Keep the interesting ones instead:

Every trace with an error.
Every trace where a quality score came back low.
Every trace that was unusually slow or unusually expensive.
Everything from a canary or experiment cohort.
A small random slice of the boring, healthy rest.

That way the traces you actually open during an incident are the ones you kept.

What a bad trace looks like, in one glance

If yours has any of these, it will fail you when it matters:

One giant span with everything crammed inside it.
Span names that change between versions.
No prompt version anywhere.
The raw user input pasted straight into an attribute.
All token costs mashed into a single number.
A flat list where an agent run should be a tree.

The nasty part is that a bad trace looks fine at a glance. It only lets you down three weeks later, at 2 AM, when you are the one who has to read it.

The lesson I keep coming back to is that "we have tracing" and "we can actually debug from our traces" are two completely different states, and I had confused them for months. Fixing the tree, the prompt version, and the score-on-span was most of the gap.

If you want the exact attribute names and span shapes to copy into your own setup, this writeup lays them all out.

If you have debugged an LLM incident from a trace, I would love to hear the one attribute you were most glad you had. For me it is the prompt version, and it is not close.

My agent passed every check and still broke production in an hour. Here's the CI/CD I run now.

Kartik N V J K — Fri, 31 Jul 2026 14:06:51 +0000

A while back my team merged what looked like a harmless change: a small refactor to our agent's tool-routing prompt. It passed lint. It passed unit tests. It passed our eval gate. Within the hour, tool-call accuracy in production had visibly regressed, and I was on-call rolling it back.

When I dug in, the cause was almost boring. Our eval set barely covered the dispatch-tool slice of traffic, so the new prompt regressed exactly where the gate had no cases to catch it. The gate did its job on everything it could see. It just could not see the thing that broke.

We shipped the same change again a day later, this time behind a canary that scored live traffic and rolled back on its own. The canary caught the regression early in the ramp, and on-call got paged once, after the automatic revert. That was the day I understood the actual shape of CI/CD for agents: the offline gate fails open on what it cannot see, and the canary closes the loop on what the offline gate misses. You need both. One without the other ships on hope.

These days I think of it as four checkpoints between "PR opened" and "everyone gets it":

Fast checks on every PR, under about ninety seconds. Does the prompt template still parse, are the tool schemas valid, does a small fixed set of cases still pass, and did the token count blow up. This catches typos, broken schemas, and "the new prompt is three times longer" before anything expensive runs.
A bigger eval at merge time, five to fifteen minutes. A few hundred cases covering normal traffic, edge cases, and past failures, scored by a judge. What I check is not an absolute number but whether the new version is any worse than main. More on why that matters below.
A simulation pass before deploy, for the risky changes. Multi-turn conversations with a handful of personas, plus jailbreak and prompt-injection probes, plus deliberately broken tool responses to see if the agent recovers. Most of the failures that actually reach users show up on the second or third turn, not the first, and only this stage catches them.
A canary after merge, ramped over a day. Start at 1 percent of traffic, then 5, then 25, then 100, scoring a sample of live requests the whole way. If quality drops, the guardrails start tripping, latency climbs, or cost per request jumps, it rolls back on its own before most users ever see it.

The one piece of gating advice I would attach to all of this: gate on the change, not on a fixed score. I used to say "faithfulness has to stay above some number." The problem is that the number moves on its own. The provider ships a minor model revision and every score shifts a little. The eval set gets harder as you add cases. Absolute thresholds either start crying wolf or get loosened until they catch nothing. Comparing the new version against main instead means each change can only slip by a small, bounded amount, and small drops stop quietly piling up.

The other thing I learned the slow way is that the eval set rots. The most common version of "the gate passed and prod broke anyway" is an eval set that stopped looking like production months ago. New kinds of requests show up, old ones fade, and the gate keeps checking the wrong things. So I treat it as living: I skew it toward real traffic with a slice of edge cases and a slice of past incidents, review it monthly, refresh a chunk every quarter, and end every postmortem by adding at least one new case. A failure we have already seen should never come back quietly.

And the strangest agent bug is the one where nothing in your code changed and the agent still got worse. It is almost always upstream: the provider rolled the model, or the judge you score with drifted. So I pin what I can. I use dated model snapshots instead of floating names, because the plain name will move under you while the dated one will not, and I keep the judge fixed so "the same eval" actually means the same eval. A weekly check then tells me if a score moved for no reason, which usually means the model changed under me.

A few of the traps I fell into on the way here:

Trusting lint and unit tests to catch what are really meaning-level regressions. They will not.
An eval set of clean, easy queries that never sees the weird 5 percent production actually sends.
Routing a slice of traffic to a new version with nothing watching it and calling that a canary. Without live scoring it is just a slow rollout.
Manual rollback, which adds real minutes of user-facing damage. Letting it revert on its own is the whole point.

None of this is exotic infrastructure. It is mostly the discipline of assuming the gate cannot see everything, and putting a second net where the first one has holes. If you ship agent changes, I am curious where you draw the line: what has to pass before a change reaches real users, and do you trust it to roll back on its own?

How I generate LLM test cases that actually catch bugs

Kartik N V J K — Mon, 27 Jul 2026 17:32:42 +0000

I got tired of hand-writing test cases for our agent, so I did the obvious 2026 thing: I pointed an LLM at our docs and asked it to generate them. In an afternoon I had a few thousand. I felt incredibly productive.

They passed CI on every release. Production kept breaking anyway.

When I finally looked closely, maybe a third of my generated tests were wrong, near-duplicates, or things no real user would ever ask. The set was too noisy to catch a real regression, so a green check meant nothing. The generating was the easy part. The part I had skipped, throwing most of them away, was the entire job.

Here is what I learned making auto-generated tests that actually catch things. The one line I would tattoo on this: generate ten times what you need, and keep only the tenth that survives a hard filter.

Why AI-generated test sets are usually useless

Two things go wrong, and I hit both.

The first is no real filter. You generate thousands, run a quick "is this well-formed" check, and ship. A big chunk are subtly wrong or duplicates, and the set quietly stops testing anything. It passes every release because it is too mushy to fail.

The second is sneakier, and it took me months to see. The generator has a house style, and everything it writes drifts toward that style. So your test set is really a test of how well your model imitates the model that wrote the tests, not how well it does the actual job. And if you ever fine-tune on that set, your model just learns to sound like the generator. The scores go up while nothing real gets better.

Both come from the same mistake: treating generation as a one-shot prompt instead of a generate, judge, and keep loop.

Generate along three axes, not one

If you generate one flavour of test, you get one flavour of blind spot. I now generate along three axes on purpose:

Personas. Same request, different people. A rushed user on their phone and a careful expert on a laptop phrase the identical question in completely different ways, and your model has to handle both. One warning: build personas from real user data, not the model's imagination, or you get cartoons ("frustrated user" becomes a stereotype).
Scenarios. Not one-line questions, but whole conversations with tool calls, dead ends, and recovery. This is the only way to catch a step that drops context on turn three or a tool that returns the wrong shape.
Adversarial versions. Take a normal case and make it harder: add a step, make it ambiguous, or phrase it like someone trying to slip past your rules. Real traffic surfaces these too slowly to wait for, so you manufacture them.

Run one axis alone and it collapses onto the others. Run all three and you get coverage you can actually point at.

The filter is the whole game

This is the part everyone skips and the part that matters. After generating, I run everything through a few stages and expect to throw away most of it:

Drop duplicates by meaning, not by text. Two questions can be worded differently and still be the same test. Matching on meaning catches the paraphrases that exact-text matching misses.
Judge every candidate and reject the ones that fail. Is this something a real user would ask? Is the expected answer actually recoverable? Is it just a reworded copy of the seed?
Check for that house-style collapse before you ship. If your synthetic set is far less varied than real traffic, that is a regenerate signal, not a green light.
Hand-check a small sample. Label five percent yourself. If the quality there is bad, throw the batch out.

The first time I saw that this rejected 80 percent of my candidates, I flinched. Then it caught a batch that was a third wrong before it could poison the set, and I stopped flinching.

Use a different model family to judge than to generate

This is the single most useful thing I picked up, so it gets its own section.

If the same model both writes and grades the tests, it is blind to its own bad habits. It shares its own vocabulary quirks, its own stereotyped personas, its own shortcut reasoning, so it happily rubber-stamps exactly the junk it produced. A judge from a different family does not share those blind spots, and it agrees with human reviewers far more often.

So I generate with one model and judge with another, or rotate which one grades each batch. It is a small change and it is the difference between a filter you can trust and a filter that quietly approves its own mistakes.

Cover the space on purpose, or you will oversample the easy stuff

Even with three axes, unconstrained generation piles up on the easy, obvious cases. So I lay out a simple grid, something like intent by difficulty by mood by dialect, and set a small quota per cell.

The reason is dumb but important: an empty cell in a grid is visible, so you can see the gap and fill it. A gap in a random pile is invisible until it shows up as a production incident. I drop the cells that make no sense for my product and fill the rest.

Check the synthetic set against real data before you trust it

Synthetic tests are only worth anything if they resemble reality, so before I trust a set I run three quick checks:

Does the synthetic set actually sit near real traffic, or is it off in its own corner of the space?
If the model wrote the labels too, I re-label a small sample by hand to make sure the labels are not carrying the generator's bias.
If the set is for fine-tuning, I train on synthetic and then test on real. If it only looks good on synthetic, the generator invented a world my model is now optimising for.

Different jobs need different filter strength

The same loop ships in three shapes, and the only thing that really changes is how hard you filter:

An eval set gets the strictest filter. It has to be right or your CI gate is theatre. A few hundred great cases beat thousands of noisy ones.
A fine-tuning set can take more volume and a slightly looser filter, because training averages over noise. But keep the filter, or your model inherits the generator's accent forever.
A red-team set flips the filter around. Now "too hard" is the goal. You keep the nasty ones and only reject the attacks that are not realistic.

Same generator, same judge. You are just turning the strictness dial.

The mistakes that quietly wreck an auto-generated test set

Letting one model both write and grade the tests.
Never checking for style collapse, so it only shows up months later.
Trusting one overall score that hides a weak high-risk slice.
Not tracking where each test came from, so you cannot re-check it when your docs change.
Treating generation as a replacement for human judgement. It is volume, not judgement. The high-stakes cases still need a human.
Running it once. Real traffic drifts, and a set you generated in spring is stale by autumn.

The thing I keep coming back to is that generating the tests was never the hard part, and it is the part all the tooling loves to show off. The hard part is being willing to throw most of them away, and letting a different model be the one that decides.

If you want to go deeper on the filtering side, the judge rubric, the dedupe cutoffs, the checks that catch style collapse, this piece is a good one on it.

If you have generated tests this way, I would love to know your keep rate. Mine settled around one in ten, and I am genuinely curious whether other people land higher, or whether everyone quietly throws most of theirs away too.

Evaluating an intent classifier: what I check beyond accuracy

Kartik N V J K — Wed, 22 Jul 2026 16:18:03 +0000

I built the intent router that sits at the front of our support agent. It reads what a user typed and decides which pipeline handles it. I tested it on a nice balanced set, got 92 percent accuracy, and shipped it feeling good.

Two weeks later the on-call engineer noticed cancellation requests were landing in the billing queue. The fraud-report queue was empty while actual fraud complaints sat in product feedback. And a newer request type that did not exist when we launched was being silently filed as a billing question most of the time.

The model had not broken. My test set had simply hidden the failures that actually happen in production. The 92 percent was real and useless at the same time.

Here is what I learned pulling that apart, in plain terms, so you can skip the two weeks I spent finding out the hard way.

Why the headline accuracy number lied to me

Real support traffic is lopsided. Five or six common intents (product questions, pricing, account access, shipping, refunds) make up most of the volume, and they are easy. The rare stuff (cancel subscription, fraud report, policy dispute, a brand-new request type) is a small slice of traffic and it is exactly where the money, the on-call pages, and the trust-and-safety incidents live.

A balanced test set gives every intent equal weight. So the model learns to do fine on all of them evenly, and my one big accuracy number was dominated by the easy, high-volume classes. The rare classes could quietly collapse to 30 percent and the headline barely moved, because they were a rounding error in the average.

And every one of those rare classes matters more than its volume suggests. Whatever the router picks decides which knowledge base, which tools, and which escalation path runs next. Get the routing wrong and everything downstream is confidently wrong.

Two test sets, not one

The fix that changed everything for me was keeping two separate test sets:

One that mirrors real traffic. Same lopsided mix as production. This is the number I report, because it is what users actually experience.
One that oversamples every intent evenly, a hundred or so examples per class no matter how rare. This is the one I debug against, because it is the only way to see whether the model can even do the rare classes at all.

The gap between those two is the warning sign. If the model is great on the real-traffic set but weak on a rare class in the even set, that class is one traffic shift away from becoming my next incident.

Look at each intent, not the average

Once I stopped looking at one number and started looking at each intent separately, the problems were obvious. For every intent I now look at two things:

How often is it right when it fires? If this is low, the model is over-using that intent as a dumping ground when it is unsure.
How often does it catch the real cases? If this is low, the model is quietly sending those cases to some neighbouring intent instead.

The single most useful view is the grid of what got confused with what. That grid told me plainly that "cancel" requests were mostly being read as "billing." Once you can see the exact pair that is colliding, the fix is usually two or three edits to how you describe those two intents to the model, not some big model change.

When one intent is both rarely right and rarely caught, that is the danger zone: either the description is ambiguous, the label itself is badly defined, or the model genuinely cannot tell it apart from its neighbour. The grid tells you which.

The confidence score means nothing until you calibrate it

My router also spat out a confidence number, and I had wired the escalation logic to trust it: below some cutoff, hand off to a human. The problem is that the raw confidence was close to meaningless.

Two things surprised me. The scores clustered at the extremes, either very high or very low, with almost nothing in the middle to set a cutoff against. And the same number meant different things for different intents. A 0.7 on a common product question was right almost every time, while a 0.7 on a fraud report was barely better than a coin flip.

So one global cutoff was shipping a totally different level of caution per intent. What actually works:

For irreversible actions like cancelling a subscription or flagging fraud, set a high bar and send anything below it to a clarifying question or a human.
For catch-all safety intents, set a low bar so the system over-routes to the safe path instead of guessing.

And treat a change to those cutoffs like a change to the model itself. Version them together.

Give the model a "none of these" option

This was the one I had completely missed. If you do not give the model an explicit "none of these" class, it is forced to jam every input into some known label. Gibberish, off-topic questions, and outright jailbreak attempts all get filed as a real intent and routed into a real pipeline.

So I added an explicit "does not fit any intent" class and trained it on the messy stuff: adversarial prompts, vague multi-topic messages, plain nonsense. Two rules made it work:

If nothing clears its confidence bar, fall through to this class instead of picking the best guess.
Never let this class dead-end. Route it to a "which of these did you mean" question or a human, not into a pipeline.

I also watch this class's catch rate closely and never let it get worse. A drop there means real garbage is leaking into real flows again.

The taxonomy you ship goes stale

The list of intents I launched with was not the list production needed three months later. User language drifts, a product launch creates a brand-new request type, a policy change creates a new kind of complaint. That new request type being misfiled as billing was exactly this.

Two things I watch for now:

An existing intent slipping. If its numbers move week over week, the model is fine but the incoming language shifted under it. Refresh its examples and re-baseline.
A new intent forming. I group together the stuff that landed in the "none of these" bucket and the low-confidence misses, and look for a cluster that keeps showing up. If the same new pattern is there two weeks running, it has earned its own intent: label some examples, write a definition, add it.

Two things I wish I had known on day one

Define intents by what the user wants, not what your system does. "User wants a refund" is an intent. "Run the refund flow" is a tool. I had mixed the two and it muddied the labels.
Check that humans agree on the labels before blaming the model. Have a few people label the same hundred examples. If they cannot agree on an intent, the model has no chance, and that is a taxonomy problem to fix first, not a model problem.

The thing I keep coming back to is that the intent router is the first decision the whole agent makes, and I had been grading it with the one number least likely to show me its real failures. The moment I looked per intent, on real traffic, with a "none of these" escape hatch, the bugs that had hidden for two weeks were sitting right there in the grid.

If you run a router like this, I would love to hear which intent quietly fell apart on you. For me it will always be cancellations pretending to be billing questions.

Evaluating LangGraph agents: the four things I score per step

Kartik N V J K — Mon, 20 Jul 2026 12:11:30 +0000

I had a LangGraph agent running in production and a regression set of fifty queries it passed comfortably. Task completion sat at 0.93. I spot-checked the answers and they looked right. I genuinely thought I had this covered.

Then someone mentioned our cost per query had roughly doubled.

I spent an afternoon in the traces and found three things wrong at once. The router was taking the expensive path on every query that happened to mention a date. A retry loop was firing three times before the search tool gave up, and at that point the model just invented an answer. And a step near the end was overwriting a constraint that an earlier step had carefully set.

Here is the part that stung. Every single one of those still produced a correct-looking final answer. My evaluation only looked at final answers. So it saw nothing, and told me everything was fine, for a week.

That was when I realised I had been evaluating the wrong thing the entire time. Took me a while to work out what to do instead, so here is the short version.

Why the final answer hid all of it

If your agent runs in a straight line, checking the output is mostly fine. But the reason you reach for LangGraph in the first place is that your agent is not a straight line.

And everything that breaks lives in the parts a straight line does not have:

Branches. Something reads the state and picks a path. A right answer down the wrong path is still a bug, just an expensive one.
Retry loops. A step can loop back on itself. Five retries and a correct answer costs five times one try and a correct answer.
Shared state. Steps read and write to a shared bag of data. One step overwriting where it should have added is silent until it is not.
Nested graphs. A step can be a whole graph itself. Flatten it when you look and you lose where the problem was.
Parallel work that merges. Fan out, merge back, and a collision quietly drops data.
Memory across turns. A bug in how state gets saved might not show up until turn three.

The final answer averages over all of that. It moves when things are catastrophically broken and sits perfectly still while the agent slowly rots. Which is exactly what mine did.

The four things I score per step

The change is easy to describe and took me a while to actually do. Stop treating the agent as a conversation with an output. Start treating it as a sequence of steps, and grade the steps.

Four questions per step:

Did it get what it needed? The right data, the right shape, nothing missing. A planning step that runs before the user's intent is filled in is broken before it starts, and everything after inherits that.
Did it write back correctly? Did it update the shared state the way it was meant to. This is where I finally caught the overwrite-instead-of-append bug that had cost me a day.
Did it pick the right branch? At a decision point, given the state right then, was that the correct path.
Does it do the same thing twice? Same starting state, same behaviour after. If not, something in there is quietly random.

First two catch correctness. Third catches expensive-but-correct, which is the one that got me. Fourth catches the bugs that become flaky incidents six months later.

If you only add one check, add the branch one

This is the one I would go back and add first.

A wrong branch still produces an answer, so nothing looks broken to anyone. But a five percent drop in how often one decision point chooses correctly shows up in your cost and latency weeks before quality metrics move at all. It is a real early warning, and I had nothing like it.

It is also so much more actionable. Overall score drops a point and you are in for a two-day hunt. One decision point drops five points and you know exactly which instruction to go rewrite. On a Friday afternoon that difference matters.

Grading everything gets expensive, so I stopped

I tried scoring every step at first and the bill got silly fast. A graph of any real size does dozens of state changes per run.

What works: pick the five to ten steps whose output lots of other steps depend on. Planners, routers, the merge points after parallel work. Score those every run and sample the rest. Most of the signal, a fraction of the cost.

The checkpoint trick I had been ignoring

This is my favourite thing I learned, and it was sitting in the framework the whole time.

LangGraph can snapshot the agent's state at every step so a long job can pause and resume. That is what it is advertised for. But those snapshots are also the best debugging tool you have.

Save one at every step and every production failure becomes a test you can rerun whenever you like. Grab the snapshot ID off the failing trace, rerun from that exact point, and watch what happens. Want to check a fix? Replay the same snapshot against the old version and the new one and compare.

Before this I was guessing whether a bug would reproduce. After it I was running an actual experiment. On an incident call that is a completely different feeling.

My test set was testing my imagination

My original fifty queries were fifty things I had thought of, which means they tested my assumptions rather than my agent.

Now I pull the runs that scored badly, group the similar ones, and promote a representative snapshot from each group into the set with a note on what should have happened. Then replay all of them on every pull request.

Every incident becomes a permanent test. The set gets stronger instead of going stale, which is the opposite of what my first one did.

When this is overkill

Being honest: if your graph is five steps in a straight line with no branching, one overall score really is enough and all of this is a waste of your time.

It starts paying off once you have real branching, retries, nested graphs, or state that survives across turns. In my experience that describes most agents by about month three, whether you planned for it or not.

What I keep coming back to is that I had built a state machine and I was grading it like a chatbot. The moment I started scoring the steps, bugs that had hidden for a month were obvious in an afternoon.

If you run agents like this, I would like to hear which check caught the most for you. For me it was the branch one, and it was not close.

I picked a coding agent off a leaderboard. It flopped on our codebase.

Kartik N V J K — Tue, 14 Jul 2026 21:51:14 +0000

Last year my team had to pick a coding agent, and I volunteered to run the evaluation. I felt good about it. I pulled up the public benchmark scores, lined up the contenders, took the one at the top, and told everyone we had a winner.

Then we actually pointed it at our repo.

It did not blow up dramatically. It just kept being slightly wrong in ways that ate our time. It wrote diffs our reviewers would not approve. It renamed a function and broke three files it had never opened. The tests it ran passed, and the repo was still broken. I had confidently recommended a tool based on a number that turned out to say almost nothing about our situation.

That was embarrassing enough that I went and figured out why. It took a few weeks of reading and a couple more bad calls before I landed on something that works. This is that, written plainly, and I hope it saves you the meeting where you have to walk your recommendation back.

Why the benchmark score lied to me

The score was not fake. It was just measuring somebody else's code. Once I looked properly, four gaps explained the whole thing:

The agent might have already seen the answers. The problems in these public benchmarks are old. Models were very likely trained on the actual fixes used to grade them. So the score partly measures memory, not problem-solving.
The setup is nothing like real work. A benchmark gives the agent a clean repo, one clear issue, and one command to run the tests. My engineers give it a half-open editor, a messy branch, a Slack thread, and a reviewer comment. Completely different job.
Our codebase has its own habits. Our internal libraries, our wrappers, our test style, the imports we ban. No benchmark knows any of that, so an agent can write textbook-perfect code that our reviewers still reject on sight.
The bar for passing is way lower. A benchmark passes a patch if the broken test now passes. My team passes a patch if it does that, and does not break unrelated tests, does not reformat the whole file, does not add stray imports, and is small enough to actually review. Benchmarks grade the first thing and none of the rest.

So I stopped treating the leaderboard as a decision and started treating it as a sniff test. Fine for narrowing the field. Useless for the actual call.

The test set was in our git history all along

Here is the bit I wish I had thought of on day one. The best possible test data for a coding agent is the work we had already merged.

Our last fifty merged pull requests came with everything a benchmark cannot fake: the real intent, the reviewer's comments, a passing test suite, and a known-good final diff. So I replayed them:

Take the last fifty merges, skip version bumps and generated files, keep the mix close to what we really ship.
For each one, note the commit it started from and the issue text.
Roll the repo back to that starting point, hand the agent the issue, let it work in a sandbox.
Compare what it produced against what we actually merged.

I got this wrong on my first attempt, so learn from it: I scored on how closely the text matched. That is both unfair and useless. A human reviewer would happily merge a differently-worded solution that behaves the same way. The question is not "does this look like our diff." It is "would this have passed the same review our diff passed."

I also swap in a freshly merged PR every week and drop the oldest, otherwise the set slowly stops resembling what we ship.

The five things I ended up scoring

After running enough replays I stopped guessing and settled on five things. Each one catches something a benchmark never sees.

Would the diff pass our review? Behavior, not text. I mark it down for reformatting files nobody asked it to touch, for breaking a test the merged version passed, and for taking twice the diff to reach the same outcome.

Did it use its tools in a sane order? A coding agent is a loop, not a text generator. It reads, edits, runs tests, reacts, commits. I check four things: did it read a file before editing it, did the edit land where it said it would, did it actually run the tests afterward or just commit blind, and did it commit only what it changed. Editing a file it never opened is just guessing with confidence.

Did it update everything the change touched? This is the one that burned us. The agent renames a helper in one file and leaves broken call sites scattered across others. The tests in the file it touched pass. The repo is wrecked. So now I follow the imports outward from every changed file and check that the downstream files still line up.

Did the plan survive the work? Modern agents write a plan before touching anything, and that plan is worth grading. Two questions: did the plan match what was actually asked, and did the final diff still match the plan. Agents drift halfway through and quietly contradict the original request while the tests keep passing. This turned out to be the best predictor of which agent our reviewers ended up trusting, which I did not expect at all.

Does it know when to give up on a bad idea? This was the sleeper, and it is now the one I care about most. When tests fail after an edit, a disciplined agent reverts and tries another approach. An undisciplined one piles more code on top of the failure until the suite is in pieces. You can see it plainly in the trace: how many edits does it make per failed test run, and does it ever actually revert. An agent averaging several edits per failure with no reverts will ship regressions. Count on it.

The first three catch correctness. The fourth catches intent. The fifth catches whether you can trust it unsupervised.

One more check I added after a scare

Coding agents invent credentials and echo risky shell patterns more often than I expected. So now every generated diff gets scanned for made-up secrets, dangerous constructs like raw eval or shell interpolation, and whatever patterns our own repo has banned. It takes milliseconds and it has already caught things. Cheap insurance.

What actually changed

The agent that won our replay was not the one at the top of the leaderboard. It was the one whose plans stayed honest and that knew when to back out of a bad edit. Neither of those shows up in a public score, and both were exactly what our reviewers cared about.

The line I settled on: the vendor owns the model and the prompt. I own the evaluation and the decision. And my evaluation is the only one that knows what our code actually looks like.

If you have run something like this on your own repo, I would love to hear which dimension caught you off guard. For me it was rollback. It never occurred to me to measure whether an agent knows how to abandon a bad idea, and it turned out to matter more than almost anything else.

I built an LLM eval framework from scratch. Here is what I wish I had bought instead.

Kartik N V J K — Tue, 14 Jul 2026 21:50:54 +0000

One weekend I wrote an LLM eval framework in about two hundred lines of Python. It demoed beautifully. I felt clever.

Six months later that same framework was a mess. Three different judge models with three different parsing hacks. A test dataset nobody had touched since November. A CI gate that kept failing because a vendor nudged their model, not because anyone broke a prompt. And the second engineer on rotation asking me, fairly, "how does this even work?"

The framework did not fail. The eighty percent of the work the weekend tutorial skipped is what failed. That gap is the whole story, and this is what I would tell myself before starting again.

The one line I wish someone had told me: build the rubric, buy the runner

Here is the split that took me six months to see.

Some parts of an eval setup are yours and only yours. The rubric that decides what "good" means for your product. The dataset built from your real failures. The rules for when a change is bad enough to block a release. Nobody else can write these, because they encode your domain.

The rest is the same at every company. The thing that calls the judge model, parses its answer, retries, and caches. The machinery to run thousands of checks in parallel. The plumbing that scores live traffic. The system that groups failing calls together. Every team rebuilds these, hits the same bugs, and gains nothing by writing them twice.

So build the first list. Do not hand-build the second. I rebuilt the second, and it cost me most of a year.

Two questions I now ask about every single piece

Before writing any part of this, I ask two things:

Is it specific to me, or generic? A rubric for my domain is specific and worth owning. A retry-and-cache loop around a model call is generic. Everyone writes the same one.
Does it compound, or does it rot? A dataset that grows from real production failures compounds. A year in, it is a regression suite no competitor can copy. A hand-built tracing layer rots. The moment a vendor changes something, you are behind and patching.

Build the pieces that are specific and compound. Reuse everything else. That single filter would have saved me the whole mess.

What is actually worth building yourself

Four things earned their place, and all four live in the same git repo as the agent code, reviewed in the same pull request as the prompt they score:

The rubric. Keep it in git, not a database, so changes show up in diffs. And make it specific. "Rate the quality" is noise. "Score 5 if every claim is supported by the source, 1 if it is largely made up" is a signal you can act on. Broken-down rubrics like that agree with human reviewers noticeably more often.
The dataset. A hundred real cases from production beat two hundred you invented. Pull the hardest failures in, because most of the signal lives in the worst ten percent of calls. Tag each case by area so a change to one feature does not rerun everything.
The gating rules. The exact thresholds and policy for blocking a release are yours. The math behind them is not, but the decision of where the bar sits is.
Your labels. The human judgement about what a correct score looks like. Only your team can produce that.

What I should not have built

The judge engine is the one that hurt the most, so start there.

Every judge model hands back its score in a slightly different shape. One wraps it in tags, another writes "Score: 4," another buries the number after a paragraph of rambling. My parser caught almost all of it on day one. On day ninety a vendor shipped an update that formatted numbers differently, my parser quietly fell back to a middle score on the cases it missed, and my dashboard showed evals gently trending up. They were not. Middle scores just drag everything toward the average.

On top of that came drift, where a judge scores the same answer differently this month than last after a model update, so the gate starts failing on prompts nobody touched. And cost, where running a top-tier judge across a big nightly set becomes a real budget line. Getting all of that right is months of work for an output that looks identical at every company.

The same goes for running checks at scale, scoring live traffic, and grouping failing calls into themes. All generic, all a long slog, none of it your edge. I would reuse an existing runner for every one of these now.

Gate on the change, not the average

This was my most expensive small mistake. I set the gate to "fail if the average score drops below a line."

The trouble is that a small dataset is noisy. With only thirty test cases, the average wobbles by a few points on its own, just from which cases you picked. So my gate failed on random noise and passed on real regressions, and within a month everyone had learned to ignore it, which is worse than having no gate at all.

What actually works is two checks together:

A hard floor that catches a catastrophic drop.
A comparison against a rolling recent baseline that only fires when the drop is bigger than the normal wobble.

The second one is the difference between a gate people trust and a gate people mute.

Run the same check in CI and in production

I kept my checks in CI only, and production drifted past my dataset within a quarter.

The fix is to run the exact same rubric in both places: against your test set before release, and against live traffic after. When they disagree, that disagreement is useful. If CI is green but production is failing, your dataset has stopped looking like reality. If CI fails on something production never shows, your dataset is carrying dead weight. Either way you learn something, but only if it is the same check on both sides.

The cost nobody warned me about

The weekend version is genuinely a weekend. The real version is not. Once you are shipping a couple of products, keeping all of this alive is close to one to one and a half full-time engineers.

And the cost never shows up when you expect it. It shows up the week a judge model updates and your parser breaks. The week a vendor changes their tracing format. The month your dataset quietly stops covering your product's top failure. None of that is on the weekend roadmap, and all of it is the job.

When building the whole thing yourself is actually right

To be fair, sometimes it is the correct call:

You are a research lab and the eval method itself is the product.
You have a strict isolation requirement where every dependency needs a long security review.
Your domain is so narrow that a general setup is just dead weight.

And a few signs you have crossed the line and should stop hand-building:

A second engineer joined and the framework now lives in one person's head.
You are maintaining more than one parser for more than one judge.
The eval bill has become a quarterly conversation.

The mistakes I would warn a friend about

Treating the eval setup as a side project instead of code that ships with the prompt.
Keeping rubric prompts in a database where changes never show up in a diff.
Letting the judge model float instead of pinning a version.
Gating on the average of a tiny dataset.
Running the full expensive judge on every single pull request.
Freezing the dataset at launch, so a two-year-old test set is grading a brand-new product.

The lesson I keep coming back to is that the hard, valuable part was never the code. It was the rubric and the dataset, the two things only my team could write, and I spent my first six months building everything except those.

If you have built one of these, I would love to hear which piece quietly ate the most of your time. For me it will always be that parser.

How I caught the voice-agent failures my eval dashboard kept missing

Kartik N V J K — Tue, 30 Jun 2026 06:59:33 +0000

I was working on a retail support voice agent, and on paper it looked great. Transcription quality, green. Conversation coherence, green. Task completion, green. Every standard metric I had was happy.

The problem was that it kept getting order numbers wrong. It would confidently read back the wrong number, the call would still "complete," and my dashboard would still show green. The checks I had simply did not know what an order number was, or why getting it wrong was the whole ballgame.

That is the gap I want to write about. The default metrics catch the failures every voice agent shares. They cannot catch the failures that belong to your specific domain. Here is what I learned about writing checks that do.

Built-in metrics catch the universal failures, not yours

Every voice agent, in any business, can fail in the same handful of ways:

Did it transcribe what the user said correctly?
Did the conversation stay coherent across turns?
Did it actually finish the task by the end of the call?
Did it stay safe, no leaked personal data, no obvious prompt injection?

Off-the-shelf checks handle all of that well. What they cannot know is your taxonomy. A retail agent lives or dies on order numbers. A clinical scribe lives or dies on drug names. An insurance agent lives or dies on coverage codes. None of that is in a generic metric, so none of it shows up on a generic dashboard.

Know when to reach for a custom check

The rule I settled on is simple:

If the failure is universal, use a built-in check.
If the failure is specific to your domain, write a custom one.
In production, run both on every call.

The last point matters more than it sounds. The two layers catch completely different kinds of errors. A call can pass the universal checks and still fail the domain one, which is exactly what was happening to me. You want both signals, not one.

What a custom check actually looks like

It helps to see a few real ones, in plain terms, no code:

Drug names for a medical scribe. Not just "was the word spelled right," but did the drug, the dose, the frequency, and the route all survive together. Getting three of four right is still a dangerous note.
Brand voice for a marketing line. Did it open with the approved greeting, avoid the banned phrases, and keep the right tone. A coherent call can still sound nothing like the brand.
Quote correctness for an insurance claim. Check each field, the deductible, the limit, the exclusions, against the actual policy, and flag any mismatch for a human to look at.

In every case the check encodes something only my team knows. That is the whole point of writing your own.

Pick the shape of the score before you write it

Before writing the check, decide what kind of answer it gives back:

Yes or no, for clean pass-or-fail rules ("did it read the right order number").
A category label, when you want to sort calls into buckets.
A score from 0 to 1, when quality is a gradient, like how closely the brand voice matched.

Pick the one that matches what you will do with it. A pass-rate dashboard wants yes or no. A trend line wants the gradient. Choosing the shape first saves you from rewriting the check later.

Two ways to actually write one

There are two honest paths, and they suit different people.

The first is to write the rule yourself, in plain English: spell out exactly what counts as a pass, what counts as a fail, and where the grey area is. Best when you, the engineer, already know the rule cold.

The second is to let a tool propose one. Some tooling reads your failed calls, groups the similar ones together, and drafts a candidate rule with example pass and fail cases pulled from real calls. You then edit and accept it. This is faster when the person reviewing is a domain expert rather than a coder.

Whichever path you take, keep one rule: a human approves the check before it goes live. Do not let anything auto-promote its own scoring into production.

The part that actually makes it work: calibrate against humans

This is the step I underestimated. Your first version of a check will score some calls wrong. Calibration is how you close that gap.

Build a small set of calls and have a domain expert label each one with the "true" score.
Run your check against that set and see where it disagrees with the expert.
Fix the wording or the examples, run it again, and repeat until it mostly agrees.

Plan for a few rounds, not one. And keep a human approving every change, with a record of who approved what. In anything regulated, that paper trail is the thing that makes the whole setup usable at all.

Pair the judge with a hard check whenever you can

A check that leans on a language model to judge will drift, and it quietly lets false passes pile up over time. So wherever a real, verifiable test exists, run it alongside the judge:

a number that has to fall in a valid range,
a format or schema that has to validate,
an exact entity that has to match.

The judge handles the fuzzy part, the hard check handles the part that has a right answer, and together they catch what either one alone would miss.

Re-run new versions on old calls

Every time I tweaked a check, I asked one question: how does the new version score the last month of calls compared to the old version? Re-running a check over your call history gives you that before-and-after diff directly. It is the easiest way to catch a "small" wording change that secretly moved everyone's scores.

Two tradeoffs I made on purpose

Calibration stays manual. A human signs off on every change to a check. It is slower, but it is the only version of this I actually trusted in a regulated workflow.
I only sample a slice of calls for the heavier scoring, to keep cost and privacy in check. Rare failures still surface, because grouping failed calls together brings the uncommon ones up even when you are not scoring everything.

The metric that finally caught my order-number problem was never going to be on a default dashboard. I had to write it, and then spend more time calibrating it against a human than writing it in the first place. That ratio surprised me, but it is also the reason the check ended up trustworthy.

If you run voice agents, I would love to hear which domain failure your standard metrics never caught. For me it will always be that confidently wrong order number.

How I A/B test LLM prompts without fooling myself

Kartik N V J K — Tue, 23 Jun 2026 19:22:48 +0000

A while back I was building a support assistant and hit a simple-sounding question: is this new version of the prompt actually better than the old one? So I did the obvious thing. I wrote thirty test cases, ran both prompts, saw the new one score a little higher, and shipped it.

I felt great about it for half a day, until the support queue filled with complaints and I was rolling the change back from a Slack thread that evening.

The bump in the score was never real. The test was far too small to tell a tiny genuine improvement apart from luck, so the number was noise the whole time. That one lesson is behind almost everything below. This is the guide I swear by now, in plain language, and I hope it saves you the same evening.

First, figure out how small a change your test can even see

A small test can only catch large differences. If the real improvement is small, thirty examples cannot tell it apart from the random wobble of which examples you happened to pick. Run it again and it often flips.

So before testing anything, I now answer two questions:

What is the smallest improvement that would actually be worth shipping?
Do I have enough examples to see an improvement that small?

The catch is how fast the numbers grow. To catch an improvement half as big, you need about four times as many examples. In practice, thirty examples can barely see anything under a ten-point swing. A four-point improvement took me a few hundred examples. A two-point one took over a thousand. Below a hundred, you are measuring your scorer's mood, not your prompt.

Test both versions on the very same inputs

My next mistake looked harmless: I gave version A one batch of questions and version B a different batch, then compared averages. But some questions are just harder. If B drew the easier batch, it looks better even when it is worse.

The fix is simple. Run both versions through the exact same questions and compare them one question at a time. The difficulty cancels out, because both faced the same hard ones, and all that is left is the real gap between the prompts.

This let me trust results from a few hundred examples that would otherwise have needed four times as many. The only cost is running every question twice, which is far cheaper than collecting four times the questions.

Put a range on the improvement, not a single average

A single average never told me enough, even with a "statistically significant" stamp. That stamp only says the difference probably is not exactly zero. It says nothing about how big it is, and the size is what decides whether I ship.

So I report a range: the smallest and largest the true improvement is likely to be.

If the range still includes zero, I do not have a real win, however nice the average looks.
If the whole range sits above the smallest improvement worth shipping, I ship.

One easy mistake to avoid: build the range from the per-question differences, keeping each question's two scores paired together. If you instead compare the two overall averages as if they were unrelated, you throw away the benefit of testing on the same inputs and the range balloons.

Score each answer alone, or judge the two side by side

There are two ways to score, and they answer different questions.

Score each answer on its own against a checklist (did it stay grounded, did it fully answer). Use this when you need an absolute number to hold against a bar you set.
Put both answers side by side and ask which is better. Use this when the quality is fuzzy, like tone or helpfulness, or when your scorer drifts run to run. Comparing two things directly is steadier than scoring each in a vacuum.

Simple rule: if the question is "is B better than A," judge side by side. If it is "does B clear the bar," score each one alone.

When you cannot wait for a full test, let the traffic pick the winner

Once I had three versions to compare, on a low-traffic feature, a normal test would have taken weeks. And the whole time, two of the three were probably worse and still going out to real people. Waiting a month while users got worse answers did not sit right.

So I used something with an odd name: a bandit. A normal test gives every version an equal share of traffic and waits until the end to call a winner, so half your users keep getting the weaker prompt the whole time. A bandit does not wait. It watches results as they come in and quietly sends more people to whatever is pulling ahead, and fewer to the rest. The longer it runs, the fewer users ever see the weak versions.

The name is just a leftover from gambling. A slot machine was a one-armed bandit, and this is the old puzzle of picking between machines when you do not yet know which pays out best. Swap machines for prompt versions and that was my situation.

The catch is that it gives up a clean number. Because it starves the weaker versions of traffic on purpose, you cannot say at the end that B beat A by exactly this much. So I only reach for a bandit when I have three or more versions, when even the weak ones are still usable, and when waiting on a full test costs more than a fuzzy final verdict. When I need a result I can defend to leadership, or it is just two versions head to head, I go back to the normal test.

Sometimes it is better to generate the variants, not hand-write one

For a long time my loop was: think hard, write one new version, test it, keep it if it won. The blind spot is that one hand-written version is a tiny slice of all the ways you could phrase the thing.

Once I had a scorer I trusted and labeled examples, I let a tool do the exploring. It generates many variations and searches for the ones that score well, far more than I would write by hand. Then I take its best candidate and run the same careful comparison against what is already in production. The tool covers ground I could not, and the A/B still makes the final call.

The mistakes that quietly ruin these tests

A few traps I fell into, so you do not have to:

An average with no range. A small bump on thirty examples is noise wearing a result's clothes.
A different batch of questions per version. You cannot tell a better prompt from an easier batch.
Swapping the scorer mid-test. That is a different test, not the same one. Lock it down first.
Peeking and stopping the moment it looks good. That manufactures wins that were never there. Decide when you will look before you run it.
Watching five "main" numbers at once. With five, there is about a one in four chance one looks like a win by pure luck. Pick one number and one bar up front.
Trusting the scorer before checking the scorer. If your automatic judge does not match careful human judgement, it can flip the result on its own noise.

That is the whole guide, and none of it is clever, which is sort of the point. The hard part was never running the test. It was learning when the result is real, and that comes down almost every time to whether I had enough examples to see what I was looking for. I hope having it in one place saves you an evening or two. If you have run these comparisons yourself, I would love to hear where your offline results and your live numbers parted ways. For me it is almost always how often the new prompt starts refusing things it should have answered.