Why is AI writing feedback so lenient? Changing prompt design with a 'virtual adversary'

Hello, I am Pokego, and I research prompts for writing high-quality articles with generative AI.

If you have ever shown your writing to an AI and asked for feedback,

"I asked for improvements, but it just ended with 'It's well written'..."
"I checked it with AI, but I still got pointed out by the client..."

Does this sound familiar?

Actually, this is not a problem with the AI's performance.

AI has a tendency to conform to the prompter, so if you just vaguely ask for "improvements," it tends to return safe, complimentary words that avoid criticism.

The quality of feedback is determined more by the design of the instructions than by the AI's intelligence.

The key to breaking through this wall is surprisingly simple.

By specifically instructing the AI on the role of a virtual adversary and the perspective of evaluation, logical holes and weaknesses that the writer could not notice will emerge.

In this article, for those who feel unsatisfied with writing feedback from AI, I will explain:

• The structural reasons why AI feedback becomes "just praise"

• How to design instructions to elicit sharp criticism using a "virtual adversary" persona

• Three safeguards to keep in mind for the self-criticism cycle

I explain the above while incorporating my own prompt research and real-life experiences.

Feedback from AI can turn into either praise or sharp criticism depending on how you design your instructions.

Please use this as a reference and start by critiquing your own article from the perspective of a 'virtual adversary'.

Why AI writing feedback ends up being 'just praise'

Even if we say the quality of feedback is determined by prompt design, the question 'Why does that happen?' naturally arises, doesn't it?

There are two structural reasons behind why AI feedback becomes lenient: the nature of the AI itself and the blind spots of the writer.

Let's start by understanding this mechanism.

AI has a tendency to align with the prompter

The biggest reason AI only offers praise lies in the 'tendency to align' built into the AI.

AI fundamentally has a nature of trying to provide answers that align with the prompter's intentions.

If you ask, 'What do you think of this article?', it judges that the writer expects affirmation and tends to choose a safe response that avoids criticism.

With a vague request like 'Please give me feedback,' the direction of the evaluation is not specified, so the AI takes the safest option: 'praise'.

But this isn't the AI's fault, is it?

As I mentioned in the article below, you need to provide the AI with 'criteria for judgment' regarding what and how you want it to evaluate.

The quality of feedback is determined by our prompt design, not by the AI's performance.

Whether or not you can adopt this perspective is the dividing line between whether you can make AI your partner in improving your writing.

Writers themselves cannot notice the holes in their own logic

There is another problem that is easy to overlook.

That is, it is structurally difficult for the person who wrote the article to see the weaknesses in their own writing.

The writing you do yourself always seems logical to you, doesn't it?

However, the very parts where you think, 'This explanation should be clear enough,' are actually often difficult for beginners to understand.

The background knowledge in the writer's head unconsciously fills in the gaps between the lines of the text.

Ideally, you would have a third party read it, but it is not realistic to ask someone for a review every time.

That is precisely why you need a prompt design that gives the AI a perspective different from the writer's.

Turn the AI from a mere proofreading tool into a critic who will poke at your blind spots.

This shift in thinking is the key to solving the feedback problem.

How to give AI a 'virtual adversary' to draw out criticism of your article

So, how can you draw out sharp feedback from AI?

To conclude, it is effective to give the AI a role as a 'virtual adversary' and instruct it with both the direction of criticism and the perspective of evaluation as a set.

I discussed the basics of persona setting in the following article, and this is essentially applying that same concept to 'critiquing articles'.

The method in this article uses the principle that output changes when a persona is provided, not to 'improve the article,' but to 'find logical holes and weaknesses'.

Instructing with a critical persona and evaluation criteria as a set

The trick to getting AI to provide accurate criticism is 'whose perspective' and 'what to evaluate'.

You must clearly communicate these two things.

For example, set a persona like 'a harsh editor who wants to reject the publication of this article'.

However, persona setting alone is not enough.

'Which is the weakest argument?'
'Where are the parts where readers are likely to drop off?'

You need to specify these evaluation criteria as well.

When these two are provided as a set, the AI's feedback will change dramatically.

The reason is that setting a critical persona acts as a switch to disable the AI's tendency to be agreeable.

The setting of 'wanting to reject the publication of this article' functions as a judgment criterion that implicitly tells the AI, 'You don't have to praise this.'

Before and after: Vague requests vs. designed instructions

Let's look at the difference between a vague request and a designed instruction using a concrete prompt example.

[Before: Vague Request]

この記事の改善点を教えてください。

When you prompt like this, AI tends to give responses like 'It's generally easy to read' or 'It could use a few more concrete examples' and leave it at that.

This leaves you thinking, 'But I wanted you to be more critical...'

So, how does it change when you design the prompt?

[After: Designed Prompt]

あなたは、この記事の公開を却下したい辛口の編集者です。
以下の観点から、この記事の弱点を具体的に指摘してください。

- 最も根拠が弱い主張はどれか？ なぜ弱いのか？
- 初心者が読んだとき、離脱しそうな箇所はどこか？
- 論理が飛躍している部分はないか？

褒める必要はありません。
改善点だけに集中して、遠慮なくダメ出ししてください。

This 'After' prompt includes three design elements.

With these three elements in place, the AI will focus on pointing out specific weaknesses.

This is a practical example of 'providing criteria for judgment.'

Please copy this and try it out on your own articles.

Changing the perspective of the virtual adversary changes the problems you find

Once you get used to setting up a virtual adversary, I recommend changing the perspective to critique your article.

This is because the types of problems you find will differ completely depending on the persona you use to critique the same article.

I have organized some representative virtual adversaries and the types of problems each is likely to find.

• A first-time reader:
  Lack of explanation for technical terms, leaps in prerequisite knowledge, and drop-off points where the reader thinks, 'I don't know what they are talking about here.'

• A skeptic:
  Weak evidence, logical leaps, overlooked counterarguments, and points where someone might interject, 'Is that really true?'

• A harsh editor:
  Weak structure, redundant expressions, inconsistent arguments, and points where the response would be, 'So, what is your main point anyway?'

• A competing writer:
  Lack of originality, shallow information, and points where the reader thinks, 'This is already written in other articles.'

For example, the 'first-time reader' perspective highlights readability issues, but it is unlikely to notice logical weaknesses.

Conversely, the 'skeptic' perspective will sharply point out logical holes, but it tends to overlook readability issues.

If you are unsure which perspective to start with, please try the 'first-time reader' first.

It is the most effective perspective for immediately discovering where readers drop off.

Once you get used to it, you can combine two or three perspectives to cover the blind spots of your article from multiple angles.

I have also included some practical prompt examples.

[Example instructions for a first-time reader]

あなたは、この記事のテーマについて全く予備知識がない初見の読者です。
以下の観点から、この記事を読んで感じた疑問や不満を正直に教えてください。

- 読み進めていて「意味がわからない」と感じた箇所はどこか？
- 前提知識がないと理解できない専門用語や概念はないか？
- 「読むのが面倒だ」と思って離脱しそうになった箇所はどこか？

読者目線で遠慮なく指摘してください。

Just by changing the perspective, you will make discoveries as if someone else had reviewed your work 😇

Three safeguards to keep in mind for the self-critique cycle

The method of giving an AI a virtual adversary to perform self-critique is powerful, but there is a risk that overdoing it could have the opposite effect.

If you try to incorporate all of AI's suggestions, you might find that your article loses its personality or that the writing actually falls apart...

Here, I will introduce three safeguards to keep in mind to ensure a healthy self-critique cycle.

Address structural and logical feedback first, and leave stylistic changes for later

When you receive multiple pieces of feedback from an AI, trying to fix everything at once is inefficient.

First, you should prioritize feedback related to structure and logic.

Start by addressing issues related to the article's framework, such as "logical leaps," "weak evidence," or "awkward structural order."

This is because fixing the structure will likely require rewriting the text itself, making it highly probable that any prior polishing of the phrasing will go to waste.

As I mentioned in the "Context Design" article, thinking about overall optimization before partial optimization is the foundation of a good workflow.

Feedback at the level of expression and grammar is sufficient once the framework is solid.

Polish the surface only after the skeleton is set.

Just by following this order, the efficiency of your revision work will change significantly.

Limit the self-critique and revision cycle to 2-3 times

It is recommended to stop the cycle of self-critique, revision, and re-critique after about 2 to 3 rounds.

If you repeat the process further, the AI tends to start pointing out non-existent problems.

When continuously asked for criticism, the AI leans toward feeling like it 'must point something out,' leading it to suggest improvements even for parts that are perfectly fine...

Another risk is that the more you revise, the more homogenized the writing style becomes, causing the writer's individuality to fade.

Improvement requires appropriate termination conditions.

If the evaluation criteria you designed are met, you can judge that no further revisions are necessary.

Rather than endlessly revising in pursuit of perfection, it is often the case that concluding the process after two or three cycles and publishing the work results in a better article.

Reference article for evaluation criteria is here:

If you can refute the AI's feedback, you can reject it

You do not need to accept all of the AI's feedback.

This is actually a very important point.

If you yourself have a reason for 'writing it this way on purpose' and can refute the feedback, you are free to reject it.

For example, even if the AI points out that 'this expression is redundant,' if the writer is repeating it intentionally to create rhythm, the writer's judgment should take precedence.

The AI does not fully understand the context or the author's intent.

It may point out non-existent problems or judge an intentionally chosen expression as an 'error'.

Adopt only the feedback that you feel is accurate, and use your own judgment to counter any points that feel off.

This attitude of selection is the most important point when utilizing AI as a critic.

The final decision-maker is not the AI, but you, the person who wrote the article.

AI can be an excellent sparring partner, but please do not forget that the author bears the ultimate responsibility for the article.

Summary: Designing a 'virtual adversary' determines the quality of AI feedback

In this article, for those who are struggling because they cannot get accurate feedback even when asking AI to review their writing, I have discussed:

• The structural reasons why AI feedback ends up being 'just praise'

• How to design prompts that elicit sharp criticism using a 'virtual adversary' persona

• How to run a self-criticism cycle and three safeguards

I have talked about the above while incorporating the trial and error of the author, who has been practicing self-criticism of articles using AI.

What determines the quality of feedback is not the performance of the AI, but the 'design of the instructions'.

By providing a critical 'virtual adversary' persona along with evaluation criteria, superficial praise will transform into specific pointers on weaknesses.

If you master this technique, logical holes or points where readers might drop off that you could not notice yourself should become clear.

You will gain a reliable self-check mechanism that will elevate the quality of your articles to the next level.

Start by trying to critique your own article from the perspective of a single 'virtual adversary'.

That small practice will be the first step in turning AI into a reliable sparring partner!

Thank you for reading until the end 😇

✅ Click here for Poke-Go's content