From Nominal HITL to Institutional HITL: A Selective Reading of Masato Fukushima's 'The Ecology of Learning'

Even if we introduce AI for reporting, is it safe as long as a human supervises it at the end?

I have been troubled by this question.
It was sparked by a podcast and articles about the military use of AI and human-in-the-loop, or HITL.
They discussed how HITL easily becomes a convenient slogan—'it's fine because a human is involved'—while also highlighting issues such as performance drops when humans are included, or the fact that only difficult cases the AI cannot handle are passed to humans, which increasingly burdens the remaining human decision-makers.
HITL / HOTL / HOOTL, approval checks, tool guardrails, human reviews, and monitoring and intervention in the era of multi-AI agents.
Such categorizations are useful, but at the same time, the question remains: 'having a human in the loop' is not the same as 'having a human in effective control'.

Is it enough just to have an approval button left?
Can a human really see through a massive volume of cases? Do they have the time and authority to doubt, stop, and overturn AI output?
As system speeds increase and multi-agent systems advance, won't human intervention turn into mere formal approval or a receptacle for responsibility rather than substantive judgment?

I believe this issue is not just about AI ethics, but about institutional design.

Who is responsible for which decision?
How much is left to the discretion of the front line, and where does organizational responsibility begin?
Is there the authority to stop?
Is the burden of supervision being measured?
When failure occurs, is the responsibility being pushed onto a single individual at the end of the line?
Thinking about 'human in the loop' is, in the end, not about 'whether we are keeping humans,' but about thinking about 'at what speed and granularity, and under what authority and responsibility, are we involving humans'.

This perspective also connects to labor-saving through AI and robots.
If labor-saving is pushed forward, the front line becomes understaffed and prone to one-person operations.
As DX progresses, labor costs become visualized and are easily discussed as targets for reduction.
However, labor costs include not just the cost of the work right in front of us, but also the costs of handling exceptions, mutual supervision, handovers, training successors, and maintaining judgment.
If we treat all of that as 'things to be reduced,' even if short-term numbers improve, the organization's learning capacity and accountability will wither in the long term.

With that awareness of the problem, I selectively read Masato Fukushima's 'The Ecology of Learning: Risk, Experimentation, and High Reliability'.
This book does not directly deal with AI or HITL. Even in the description by Chikuma Shobo, it is introduced as a book that asks how learning occurs in organizations where major accidents can happen, such as medical sites and nuclear power plants.
It is particularly important that it treats the repetition of 'everyday experiments,' including failures, as learning resources on the front line, and calls the space that allows for trial and error involving risk an 'experimental domain of learning.'

What I read this time were the commentary for the paperback edition, Chapter 4 'The Experimental Domain of Learning: Trials, Costs, and Exemption,' Chapter 5 'Organizations, Risk, and Technology: On High-Reliability Organization Research,' Chapter 7 'Wild Risk Management: Observing the Dynamics of a Ward,' and Chapter 8 'Organization and Learning in an Emergency and Critical Care Center.'
Even in the table of contents of this book, these five chapters create a reading path that connects learning, risk, high reliability, psychiatric wards, and emergency and critical care centers.

The order of reading was as follows.

The commentary for the paperback edition to grasp the overall problem awareness of the book.
Chapter 4 to consider that learning requires an 'experimental domain' that allows for failure and trial.
Chapter 5 to see from the discussion of high-reliability organizations that safety is supported not by individual excellence but by organizational redundancy.
Chapter 7 to consider the handling of discomfort and on-site knowledge that cannot be fully quantified from risk management in a psychiatric ward.
Chapter 8 to see the ideal state of a system that simultaneously establishes safety and education from the organization and learning in an emergency and critical care center.

Reading in this order, 'The Ecology of Learning' draws HITL away from 'the story of putting a human at the end' and toward the story of 'how to maintain a system where humans can make judgments'.

Commentary for the paperback edition

Viewing HITL not as 'individual judgment' but as an 'ecology of learning'

Using the commentary for the paperback edition as an entrance, the overall interest of the book becomes visible.
What is dealt with here is a perspective that views learning not as a process where knowledge is accumulated inside an individual's head, but as something that emerges within the interactions of the front line, tools, systems, training, failures, and others.

This is quite important when thinking about HITL.

In nominal HITL, the human is placed outside the system or at the very end.
The AI processes it. It is passed to a human only when there seems to be a problem.
The human confirms it at the end. As a form, the human is in the loop.

However, from the perspective of 'The Ecology of Learning,' judgment does not exist in isolation like that.
For a human to be able to judge, an accumulation of experience is needed.
Opportunities to learn from failure are needed. A place to verify with people around is needed. A relationship with records, procedures, and tools is needed.
A circuit to rephrase the situation one is seeing from another person's perspective is needed.
In other words, judgment is born not so much from 'individual ability' as from an 'environment where one can judge.'

From this point of view, the question of HITL does not end with 'whether to keep humans'.

What is that human looking at?
What information can they access?
At what stage can they stop it?
When they stop it, does the organization support that judgment?
When they make a mistake, does it become a learning resource, or is it handled as individual responsibility?

Reading this book with this question, HITL changes from a term of technical design into a word that measures an organization's learning capacity.

Chapter 4: 'Experimental Domains of Learning'

How to learn in places where failure is not an option

At the heart of Chapter 4 is the tension that learning requires trial and error, yet organizations often restrict that trial and error.

Organizations seek to streamline procedures, reduce complexity, and increase predictability.
This is only natural.
Medical settings, nuclear power plants, schools, government agencies, and corporations—in any field, failure carries a cost.
Especially in areas like medicine, security, and infrastructure, failure leads directly to loss of life or major accidents. That is why organizations try to reduce failure.

However, learning is difficult to achieve without failure, deviation, or a sense of discomfort. Only after experiencing things that did not go well, things that differed from expectations, or things that produced strange results despite following procedures, can a site update its understanding.

Chapter 4 is important because it considers this space for trial and error not merely as 'experience,' but as an 'experimental domain of learning.'
Learning requires room for experimentation.
However, that experimentation is not laissez-faire. It is necessary to control the cost of failure and to organizationally adjust how much can be tested within legal, economic, and ethical constraints.

This leads directly to HITL.

When humans supervise AI output, do they have an experimental domain?
Is it okay to doubt the AI's judgment?
Is it okay to take time to verify it? If the AI turns out to be correct, will the human be evaluated as having 'done something unnecessary'? Conversely, if an accident occurs after simply approving the AI's output, will the responsibility be concentrated solely on that human?

In nominal HITL, the human is placed as an 'approver.'
However, in institutional HITL, the human must be an entity permitted to doubt.
Moreover, just being able to doubt is not enough. There must be a circuit for the organization to accept the results of that doubt, examine them, and reflect them in the next design.

Efficiency tends to cut into this.

Increase processing speed. Reduce personnel. Standardize judgment. Reduce exceptions.
In the short term, the work appears to be in order.
But in the process, when small failures, vague discomforts, the margin to try stopping something, and the time for seniors to explain reasons to juniors disappear, the organization stops learning.

In other words, what becomes clear from Chapter 4 is that what HITL needs is not just 'human placement,' but institutional margins where humans can test, doubt, and learn from failure.

Chapter 5: 'Organizations, Risk, and Technology'

High reliability is created not by individual heroism, but by redundant organizational structures

Chapter 5 deals with the study of high-reliability organizations.
A high-reliability organization is a concept for considering organizations that maintain extremely high safety despite the fact that failure could lead to major accidents.
Fields such as aircraft carriers, air traffic control, and nuclear power plants are assumed as typical examples.

Reading this chapter with the awareness of HITL issues reveals something very important.
Safety is not protected by the judgment of one excellent person.
Rather, safety is supported by multiple checks, redundant procedures, circuits for dissent, the authority to stop from the front line, records, training, division of roles, and delegation of authority according to the situation.

In high-reliability organizations, hierarchies do not become unnecessary.
Hierarchies and procedures are necessary in normal times.
However, in situations where danger is imminent, the judgment of the person who sees the situation best or has the relevant expertise carries more weight than that of a formal superior.
This is critically important when thinking about HITL as an institution.

When thinking of HITL in the form of 'a human looking at it at the end,' one inevitably focuses on a single supervisor, a single approval point, and a single responsible entity.
But from the perspective of high-reliability organization theory, that is actually dangerous.
Safety is not something protected at a single point.
It is protected by multiple people watching, multiple tools supporting, multiple paths for sharing anomalies, and the ability to stop if necessary.

Therefore, institutional HITL is better thought of not as 'human in the loop,' but as humans, artifacts, procedures, and conversations in the loop. Instead of putting one human in the loop, it is necessary to design it as a structure where humans, tools, procedures, records, conversations, meetings, handovers, and reviews overlap.

Here, there is also the paradox of automation.

The more AI and automation smooth out routine processing, the further humans drift from the experience of making everyday judgments.
While the system is running well, humans act as monitors.
However, when an anomaly occurs, humans are suddenly required to make high-level judgments.
They are not allowed to make judgments in normal times, yet are expected to make high-level judgments only in emergencies.
This is a highly unreasonable design.

Judgment is not something that can be summoned only in emergencies.
It is maintained through small daily anomalies, minor deviations, confirmations, questioning, and the verbalization of judgments. If you strip that away through automation and then expect humans to step in only at the very end, that HITL is less of an institution and more of a ritual.

The lesson to be drawn from Chapter 5 is this:
To make HITL an institution, one must not simply place a human as the 'final responsible party,' but rather embed redundant observation points and stop points within the organization.

Chapter 7: 'Wild Risk Management'

Risk judgment deals not only with quantified dangers but also with the sense of unease felt on the front lines.

Chapter 7 deals with risk management in a psychiatric ward.
The risks here are not just dangers that can be measured engineering-wise. They involve a complex interplay of self-harm, violence, medication, isolation, patient autonomy, staff safety, intervention as treatment, intervention as management, relationships with families, and the atmosphere of the ward.

The 'wild risk management' referred to in this chapter does not mean immature risk management.
Rather, it refers to risk management that includes context-dependent and ambiguous judgments made on the ground, which cannot be fully tamed by manuals or numerical values.

In a psychiatric ward, risks cannot be grasped by scores or checklists alone.
Of course, such tools are necessary. But the patient's facial expressions, way of speaking, gait, behavior within the ward, the sense of unease felt by nurses, unusual silences, and small changes shared among staff—
these things become important materials for judgment.

This provides significant implications for HITL in the AI era as well.

AI systems process quantified information and input data.
Therefore, institutional design tends to lean toward scores, thresholds, approval flows, alerts, and logs. These are necessary.
However, if you rely only on these, the on-site feeling that 'something is wrong' becomes difficult to treat as legitimate knowledge.

In nominal HITL, humans become clerks who react to alerts issued by the system.
In institutional HITL, the sense of unease that emerges from the human side must also be an input that drives the system and the organization.

'I don't know why, but I have a bad feeling about this.'
'There is no problem with the procedure, but as a gut feeling on the ground, I want to stop it.'
'The premises for this case are different, so I don't want to apply the standard judgment.'
'The numbers aren't bad, but it's dangerous to apply this to this patient, this customer, or this site right now.'

Are these voices to be excluded as mere personal intuition? Or are they to be treated as knowledge that the organization should examine? This is where the difference between HITL as a mere name and HITL as an institution lies.

Chapter 7 is even more important because it shows that risk management always involves ethical questions.
It is easy to say that one is intervening for the sake of safety.
But whose safety is it?
Are you protecting the patient themselves?
Are you protecting other patients?
Are you protecting the staff?
Are you protecting the organization?
Is the intervention treatment, management, or punishment?
While measures like protection and isolation reduce danger, they also affect the individual's freedom and dignity.

The same applies to HITL.
One cannot say it is ethical just because a human is intervening.
Who is that intervention protecting?
On whom is the burden being shifted? Who bears the accountability?
Is human intervention merely a decoration to justify the AI's output?

What emerges from Chapter 7 is that institutional HITL requires a space to handle not only quantified risks but also on-site unease, case-by-case context, and the legitimacy of intervention methods.

Chapter 8: 'Organization and Learning in the Emergency and Critical Care Center'

HITL is not a single doctor, but a multi-layered loop including paperwork, meetings, nurses, and checklists.

Chapter 8 was the chapter most directly connected to the issues I am concerned with.
Emergency and critical care centers have no time.
The risks are high. The judgments are heavy.
The impact of failure is significant.
Moreover, they must train newcomers in that environment.
In other words, it is a site where safety and learning collide most intensely.

What is important here is that safety is not maintained solely by the judgment of one skilled doctor.
Doctor's orders, confirmation by nurses, coordination by lead nurses, handovers, conferences, records, checklists, manuals, and information sharing for each patient.
Safety is created by the overlapping of these multiple mechanisms.

In an emergency and critical care center, the speed of judgment is crucial.
However, increasing speed alone becomes dangerous.
That is why papers to confirm instructions, procedures for verbal confirmation, relationships where nurses can ask questions, roles for leaders to oversee the whole, and spaces for post-hoc reflection are necessary.
Here, humans are not entering the loop alone.
Humans, papers, meetings, records, and the division of roles constitute a single loop.

I believe this is a fairly concrete image of HITL as an institution.

The same questions arise in the field of AI implementation. Who looks at the AI's output?
Can the person looking at it see the context before and after?
Can they verify the basis for the judgment?
Can they consult with someone else?
Are records kept?
Can they look back on it later?
Is the structure such that one person on the front line is left to shoulder cases where judgment is difficult?

From Chapter 8, another important point emerges. It is the concentration of judgment load on experts.

Figures like lead nurses are extremely important for organizational safety.
They read the situation on the ground, confirm doctors' orders, coordinate the movements of nurses, and find dangerous oversights.
This is substantive control.
However, at the same time, if that safety depends on the cognitive labor of specific experts, there is another risk there.

The same thing happens with HITL.
Only difficult cases that AI cannot fully process are passed on to human reviewers.
Moreover, that person must make subtle judgments in large quantities and in a short time.
Formally, it is 'being confirmed by a human'.
But in reality, the judgment load and responsibility are concentrated solely on that person.

This is fragile as an institution.

HITL as an institution does not end with placing an expert.
Mechanisms to support the expert's judgment, mechanisms to distribute judgment, mechanisms to train successors, and mechanisms to share failures and hesitations are necessary.
Otherwise, HITL becomes a structure of 'pushing onto humans at the end' rather than 'having humans at the end'.

Another core of Chapter 8 is the tension between safety and education.
Newcomers cannot grow without experience.
However, letting newcomers gain experience in an emergency and critical care setting carries risks in itself.
Therefore, education cannot be left to chance. On the other hand, if you turn everything into procedures, eliminate risks, and limit them to observation, practical judgment will not develop.

Here, too, is an important point of HITL.
For humans to supervise AI, those humans must be growing.
However, if AI takes over too much of the daily judgment, the space for humans to grow decreases.
Opportunities for newcomers to experience small judgments, opportunities for seniors to explain the reasons for judgments, and opportunities for teams to verify judgments disappear.
The more the field is neatly organized through efficiency, the more the organization's learning ability may actually wither.

Viewed from Chapter 8, HITL is not simply a design to 'leave humans in'.It must be a design where humans continue to grow as judges.

What is nominal HITL?

Reading this far, the outline of nominal HITL becomes clear.

Nominal HITL is a state where humans are formally in the decision-making process but do not exercise substantive control.

For example, a human presses an approval button for AI output.
However, that human cannot verify the basis.
There is no time.
The number of judgments is too high.
They have no authority to object.
Stopping it is treated as a work delay.
Only the accountability for overturning the AI's judgment is heavy.
When an accident occurs, it is said that 'a human was checking'.

In this case, the human is not the subject of control.
They are an absorber of responsibility.

Alternatively, AI automatically processes cases it is confident about and passes only difficult cases to humans. This also seems rational at first glance.
However, on the side of the remaining humans, only cases that are difficult, ambiguous, highly responsible, and must be processed in a short time pile up.
Humans lose 'simple experience' and end up taking on only 'difficult exceptions'. This makes it difficult to maintain judgment and train successors.

Furthermore, human intervention is sometimes used as a display of 'ethical consideration'.
It's okay because a human is watching.
It's okay because it's human-centered. But if that human cannot stop it, cannot doubt it, and cannot learn from it, then it is not human-centered.
It is a facade of human-centeredness.

The problem with nominal HITL is not that there are no humans. It is that even though humans are present, there are no conditions for them to make judgments.

What is HITL as an institution?

So, what is HITL as an institution?

It is not about placing humans in the decision-making process. It is an institution that enables humans to understand situations, doubt, stop, verify with others, learn from failures, and distribute that burden organizationally.

I believe that HITL as an institution requires at least six conditions.

First, humans must be able to 'see.'
They need to be able to see not only the AI's output but also the premises, grounds, input data, context, past similar cases, and the scope of the decision's impact.
Approving something after being shown only the output is closer to a ritual than a judgment.

Second, humans must be able to 'doubt.'
They need to be able to ask back regarding the AI's output: Why is it like this? Are the premises different? Is it inapplicable to this case?
If doubting is treated as obstruction of business or inefficiency, HITL will not function.

Third, humans must be able to 'stop.' A person who senses danger must have the authority to actually stop the process.
Moreover, the person who stops it must be protected organizationally.
It is important that even if nothing happens as a result of stopping it, that judgment is not punished later.

Fourth, humans must be able to 'verify.'
A space is needed where decisions are not confined to one person, but can be discussed, confirmed, and challenged with others.
High reliability is created not by a single hero, but by redundant verification structures.

Fifth, humans and organizations must be able to 'learn.' It is necessary to use not only accidents and failures, but also near-misses, feelings of discomfort, decisions to stop, cases where AI was overruled, and conversely, cases where following AI caused no problems, as resources for future learning.
Just leaving logs is not enough.
There is a need for spaces to read logs, discuss them, and feed them back into the institution.

Sixth, humans must be able to 'grow.' The humans who take on HITL do not exist from the start.
They grow through daily decision-making experience, explanations from seniors, gradual participation, and learning from failures.
If that experience is taken away by automation, the talent to handle future HITL will also be lost.

Without these six, HITL approaches being merely nominal.
Conversely, I think HITL as an institution is not about human intervention itself, but about protecting the conditions under which humans can intervene.

What to ask in practice

What we should really be asking at the site of AI implementation or automation is not 'Are humans supervising?'

What we should be asking are things like the following:

Can that human see not only the AI's output but also the premises of the judgment?
At what point can they stop it?
When they stop it, does the organization protect them?
Is it institutionally permitted to overturn an AI's judgment?
Are the number of judgments and the time spent on them at a level that a human can truly oversee?
Are only difficult cases being concentrated on humans?
Is the supervision load being measured?
Is there a circuit for multiple people to verify?
Is the discomfort felt on the front lines treated as formal information?
Are small failures and near-misses treated as learning resources?
Has labor-saving cut into mutual supervision, handovers, and the training of successors?
Is keeping humans in the loop just an excuse to push responsibility onto them?

If you say 'It's HITL' without asking these things, it is not an institution, but a label.

Connection to the original reading plan

In my original problem awareness, Jerry Z. Muller's 'The Tyranny of Metrics,' Nicholas G. Carr's 'The Glass Cage,' and Stuart Russell's 'Human Compatible' were listed as reading candidates.
These concern the problem of numbers and evaluation metrics consuming the purpose, the problem of automation turning humans into monitors or emergency scapegoats rather than making their lives easier, and the problem of how to redesign AI with different control principles.

Placing 'The Ecology of Learning' there changes what becomes visible a little bit.

If 'The Tyranny of Metrics' teaches us the danger of metrics becoming the goal, 'The Ecology of Learning' makes us think about how on-site knowledge and discomfort that are not turned into metrics become learning resources for an organization.

If 'The Glass Cage' teaches us the danger of human skills thinning out due to automation, 'The Ecology of Learning' shows us in what kind of experimental domains and participatory structures those skills are nurtured in the first place.

If 'AI Rebirth' is a book that considers the control principles of AI, then 'The Ecology of Learning' can be read as a book that questions how actual organizations can handle that control.

In other words, 'The Ecology of Learning' is a book for shifting HITL from AI functional design to organizational institutional design.

Conclusion

Keeping humans in the loop does not guarantee safety

When introducing AI, keeping humans in the loop is important. However, keeping humans in the loop does not guarantee safety.

If humans cannot observe, cannot question, cannot stop, cannot consult, cannot learn from failure, or cannot grow, then those humans are not the controlling agents. They are merely being used as a place to dump responsibility.

What becomes clear from 'The Ecology of Learning' is that human judgment does not exist in isolation within an individual's mind. Judgment is supported by the field, tools, records, conversations, training, failures, authority, immunity, redundancy, and the environment for learning.

Therefore, to move from nominal HITL to institutional HITL, simply 'putting humans in the loop' is not enough.

Create a loop where humans can make judgments.
Create a loop where humans can stop the process.
Create a loop where humans can verify with others.
Create a loop where humans can learn from failure.
Create a loop where human burden and responsibility can be distributed.
And create a loop where humans can grow as the next decision-makers.

HITL is not about placing humans at the end.
It is about maintaining an institution where humans can continue to be the agents of judgment.

What is being questioned in the AI era is not whether to remove or keep humans.
Are we just pretending to keep humans in the loop while forcing them to absorb all the responsibility?
In the name of efficiency, are we cutting away the margins necessary for an organization to learn?
We must look at that.

Keeping humans in the loop does not guarantee safety.
Can we maintain an institution where humans can make judgments?
That is what is being questioned.

Book URL

Reference URL

Hashtags

#EcologyOfLearning
#MasatoFukushima
#HITL
#HumanInTheLoop
#AIImplementation
#AIGovernance
#InstitutionalDesign
#OrganizationalLearning
#RiskManagement
#HighReliabilityOrganization
#HRO
#SafetyCulture
#FieldKnowledge
#TacitKnowledge
#SituatedLearning
#PracticalWisdom
#MedicalSafety
#EmergencyMedicine
#Psychiatry
#ReadingNotes