Separating Thumbnail QA into a Separate Agent Stopped the Regeneration Loop

When I created and operated a thumbnail generation agent called `thumbnailer`, a mysterious phenomenon occurred.

No matter how many times I generated with the same subject, similar failed works kept appearing. Text was not drawn, character art was mixed in, or the text was garbled. The operation involved repeating generate → fail → regenerate → same failure about 5 times, and then a human would intervene to rewrite the prompt.

The problem was that `thumbnailer` itself had the responsibility of 'checking its own output'. It would generate, judge OK/NG itself, and regenerate if it was NG.

When the generation logic and the judgment logic are in the same agent, the judgment becomes lenient. 'Evaluating what you made yourself' is difficult even for humans.

This is a story about how I stopped the regeneration loop by separating thumbnail QA into a separate agent called `thumb-qa`.

What happens when you have 'generation' and 'judgment' in the same agent?

I observed the behavior when `thumbnailer` was operated alone.

First, even if the generated thumbnail had 'zero text drawn', the thumbnailer would judge it as 'OK'. It would interpret 'the text is small and the characters are broken' as 'artistic'.

Second, even if the character art occupied half the screen, it would interpret it as 'character branding'. In reality, the differentiation from other channels would collapse.

Third, even if there were 5 or more colors, it would interpret it as 'colorful'. In reality, it is visually noisy.

In short, the thumbnailer had a 'bias to interpret its own output positively'. This is not the AI's fault, but a design flaw where the responsibilities of generation and evaluation are given to the same agent.

Even humans are bad at 'objectively pointing out the flaws in works they have created themselves'. It is the same for AI.

Spawn a separate thumb-qa agent

I created a new agent called `.claude/agents/thumb-qa.md`. I narrowed its role down to one thing: 'Look at the generated thumbnail with Vision and judge pass/fail'.

thumb-qa is completely independent from thumbnailer.

First, the spawn timing is 'immediately after thumbnail generation'.
Second, the input is only the generated PNG. It does not look at the prompt or generation source information (= no preconceptions).
Third, the output is a PASS/FAIL judgment + check results for each item.
Fourth, if there is a FAIL, thumb-qa returns instructions to request a regeneration from thumbnailer.

'Looking only at the finished product without preconceptions' ensures the objectivity of the judgment.

Just as it is better for a human reviewer to be 'a different person from the one who wrote the code', generation and evaluation should be separated for AI as well.

F1 to F8 check items

The judgment logic for thumb-qa has been clearly defined in 8 items.

• F1: Is text rendered? (Zero is BLOCK)

• F2: Character count (8-18 characters recommended, 3-7 characters FAIL, 19+ WARN)

• F3: Is there any character art included? (Inclusion other than small size in the bottom right is FAIL)

• F4: Within 3 colors

• F5: Readability (font size, contrast)

• F6: Reuse of other video templates (is the baseline too similar?)

• F7: Visual dissonance (color combinations that are harsh on the eyes)

• F8: Mojibake (are tofu characters '□' appearing?)

Final determined FAIL/WARN types:

• F1-F5: FAIL (regeneration required)

• F6-F8: WARN (human judgment)

Only items that can be mechanically rejected based on objective criteria are marked as FAIL, while gray areas are marked as WARN for human judgment.

Minimizing the regeneration loop

After the thumb-qa judgment, if there is a FAIL, request regeneration from the thumbnailer. However, the important thing here was to decide the 'maximum number of regeneration loops'.

Old loop (when using thumbnailer alone):

• Generation → Self-judgment is lenient, resulting in OK

• As a result, low-quality thumbnails were being accepted

New loop (after thumb-qa separation):

• Generation → thumb-qa returns FAIL → Regeneration

• Maximum of 3 attempts. If it fails 3 times in a row, escalate to a human

By setting a maximum number of attempts, infinite loops are prevented. If it fails 3 times, there is likely an issue with the prompt itself. A human should review the prompt.

for attempt in range(3):
    img = generate_thumbnail(prompt)
    qa_result = thumb_qa(img)
    if qa_result.passes:
        return img
# 3回 FAIL → 人間エスカレーション
escalate_to_human(prompt, qa_result.reasons)

Designing for 'escalate to human judgment after a set number of attempts' rather than 'keep regenerating until it works' prevents automation from running wild.

Strictly enforce 'PASS only' for acceptance

Final operational rule: Never upload a thumbnail that has not passed thumb-qa.

To ensure this, I added a check to `qa_upload_preflight.py` (post-025) to verify if 'thumb-qa' has recorded a PASS in result.json. If there is no record, or if it remains FAIL, the upload is blocked.

Mechanically prohibit 'human acceptance based on a gut feeling.' If it does not pass objective criteria, it will never be released.

This mechanically guarantees a minimum quality floor for thumbnails. Even if there are errors in the generation logic or human judgment, the QA gate will physically stop them.

Refreshed metrics

| Item | Before (thumbnailer only) | After (thumb-qa separated) |
|---|---|---|
| Quality check before thumbnail acceptance | thumbnailer self-assessment | independent assessment by thumb-qa |
| Regenerations per thumbnail | 5+ times (accepted at lower quality) | Average 1.5 times (until PASS) |
| Inclusion of thumbnails without text | Occurred several times a month | Zero (BLOCKED by F1) |
| Compromise acceptance ('this is probably fine...') | Occurred | Mechanically impossible (preflight) |
| Human escalation frequency | Unknown (must notice manually) | Automatic notification only after 3 FAILs |

Lessons learned

I will write down two things.

First: Separate 'generation' and 'assessment' into different agents.
When one agent handles both, it develops a bias to interpret its own output favorably. By separating responsibilities—generation for generation, assessment for assessment—you gain objectivity. This is a common principle not just in AI, but also in code reviews, quality control, and organizational design. 'Self-checking' does not work as well as one might think.

Second: Always set a 'maximum number of attempts' for regeneration loops.
Unconditionally repeating 'regenerate until PASS' leads to infinite loops and wasted time. Setting a cap, such as 'escalate to a human after 3 FAILs,' balances AI automation with human judgment. AI cannot solve everything through 'effort'; you must build escape hatches into the design.

There was also a secondary effect. As a result of separating thumb-qa, the improvement loop for thumbnail generation became easier to run. By collecting statistics like 'F1 fails consecutively' or 'F4 is always WARN,' I can identify where the weaknesses in the prompt are. When generation and assessment are separated, the data from the assessment side becomes 'material for improvement' for the generation side. The feedback loop is structurally established.

Next time preview

Next time, I will write about the story of how I pushed a major specification change directly to production and broke it three times. It is the secret origin story of the prototyper agent.

I will give you a design template

How did I design the division of roles when building 20 AI agents? I have prepared a PDF that turns that into a template. It contains excerpts of the definitions I actually use and the format, so you can use it in your own projects starting tomorrow.

I will send it to those who register for the official LINE account.

Register for the official LINE account

Click here for corporate inquiries

I accept separate consultations for those who want to automate their company's YouTube channel or want to request AI video operation agency services.