Too Heavy for a Software Company

—An AI company selling cars, newspapers, and power plants using Beta-version practices

Observed August 2026
Observer Zero

Are conversational AI companies no longer just software companies?

Let me clarify first. This article is not about AI giving incorrect answers individually, nor is it about AGI threatening humanity.What I want to address here is the gap between the magnitude of the social impact that conversational AI and agent AI companies have begun to exert, and the way they provide their services and take responsibility for them.

Current AI company products are doing at least the following things:

As agents,
・They drive real-world tasks such as emails, files, schedules, purchases, and business processing.
・They require users to perform specialized tasks such as permission management, output auditing, security judgments, troubleshooting malfunctions, and model selection.
・Through data centers, they exert physical influence on electricity, water, land, power grids, and local communities.
・They generate text in a unique way for each individual, reflecting safety standards and search policies designed by the company regarding what to answer, what to refuse, and what to treat as reliable information.

And yet, the basic way companies provide these services is still as follows:

・They deliver unfinished products as Beta, Preview, or Experimental.
・They frequently change products while they are being provided, while watching user feedback.
・They demand that users handle technical terminology, permission settings, log verification, and model selection.
・Regarding agent malfunctions, they shift the responsibility for final confirmation and supervision back to the user.
・They let local communities bear the burden of the data center's electricity, water, and land usage.

In other words, when selling influence, they act like social infrastructure, but when held accountable, they revert to being probabilistic software.

Is this discrepancy not one of the primary structures creating the problems surrounding current AI? This is the starting point of this article.

From Conversation to Execution

This discrepancy existed even in the era when conversation itself was the main focus. However, it was not yet so blatant.

Even if AI advice was incorrect, humans ultimately decided whether or not to execute it. There was a step of human execution between the AI's words and the actual action.

Agent AI is beginning to incorporate that final step into the machine side as well. The AI that was once a proposer has become an executor.

Sending emails. Changing schedules. Manipulating files. Purchasing products. Connecting to external services. Errors no longer stop within the screen.

Unpaid Middle Management

For AI with the power to act on reality, the core of the safety measures prepared by companies is, in many cases, the single sentence: 'Final confirmation is performed by a human.'

However, being placed in the role of final confirmation and actually being able to perform that confirmation are not the same thing.

In fields that have studied situations where automation, robots, and humans collaborate, there is a concept called the 'moral crumple zone.' It is a term that refers to a structure where a human, given only a limited range of information and limited control, is placed in a position where they are forced to absorb the responsibility for a system failure when it occurs.While a car's crumple zone protects the occupants by crushing the vehicle body,in this structure, there is a reversal where the human closest to the system becomes the one to absorb the impact in order to protect the system side.This is not to argue that the supervisory responsibility for AI agents is exactly the same structure.Even so, there is an overlap in the sense that they are burdened with the role of final confirmation without being given sufficient authority or explanation.

The very idea that putting humans last makes things safer is also beginning to be seen as having its limits.Research is emerging suggesting that as the number of times approval or confirmation is required increases, the process can become a mere formality, potentially leading to more oversights.Supervisors do not have infinite processing capacity, and increasing the amount of supervision does not always enhance safety.

While claiming that AI will perform tasks on our behalf, are we not turning everyday people into unpaid middle managers for AI?

The Socialization of Beta Culture

In the field of autonomous driving, experts have long criticized the practice of using public roads themselves as testing grounds.The approach of handing over products that are deeply tied to safety to users while they are still unfinished, and improving them through use, is a practice thatcritics argue may be a valid contract between developers, but is not something that should be brought directly to the general public.

The same structure can be seen in the provision of conversational AI and agent AI.

The term 'Beta version' was originally a contract that made sense between developers, meaning 'it is still unfinished,' 'bugs will occur,' and 'behavior will change.'The user side also had its own etiquette: checking logs, reverting settings, switching to older versions, and isolating bugs.

However, conversational AI has entered spaces where ordinary people without specialized knowledge consult it, write text, entrust it with work, and rely on it for daily decisions. Into these spaces, unfinished products are being delivered using the same language intended for developers.

How should users interpret products that charge fees and are recommended for daily use, while simultaneously remaining 'Preview' or 'Experimental'?

The Power Plant Beyond the Outlet

The weight of conversational AI is not found only within its words.

Data centers that power AI require electricity, water, land, and power grids. It has been reported that in some regions, discussions and opposition have arisen between residents and local governments over electricity rates, water resources, and the expansion of power transmission facilities.

What I want to clarify here is that not all data centers are built and operated directly by AI companies, and the extent to which AI demand is the primary cause varies from case to case. Therefore, it is not the purpose of this article to name and condemn specific companies or facilities.

Even so, one point remains clear: residents living near data centers may be affected by them through electricity, water, and land, even if they have never used an AI service themselves.

At the point where the impact reaches residents who have not agreed to the terms of service, the framework of a contract solely between the AI company and the user may no longer be sufficient to handle this issue.

The Talking Newspaper

Conversational AI is constantly choosing what to answer, what to refuse, and what to treat as reliable information.

This does not mean that AI possesses conscious thought.It means that the data used for training, the adjustment policies, safety standards, search design, and corporate judgments appear somewhere in the output as an editorial policy.

Which information sources to prioritize, which positions to consider safe, which questions to answer, and which to refuse. Systems to verify these judgments from outside the company are still almost non-existent.In a June 2026 contribution to The Guardian, security technologist Bruce Schneier and data scientist Nathan E. Sanders argued that as long as AI rewrites the words of others and exercises editorial discretion like a newspaper, it should carry responsibilities similar to those of a publisher. They reject the idea that companies can avoid responsibility by treating AI as an independent legal entity or a neutral tool, and argue that AI agents should be treated as representatives of the individuals or organizations that deploy them.

With a newspaper, all readers share the same page, and if there is an error, it can be corrected in the next day's correction column. Conversational AI generates a different page for each individual user on the spot. It recommends to one person and discourages another. It speaks in detail to one person and refuses to answer another.

While it is a talking newspaper that delivers a million different pages to a million people, the readers cannot see the editor-in-chief, and there is no correction column that everyone can share.

An industry with neither vehicle inspections nor editorial responsibility

Automobiles, electricity, publishing. Each of these industries has, over time, developed systems of responsibility commensurate with the scale of their impact on society. Safety standards, accident investigations, recalls, insurance, vehicle inspections. Environmental assessments, regional consensus, obligations for stable supply. Editorial responsibility, correction systems, the location of publishers.

None of these were perfectly formed from the start. They were created after the fact, through a history of accidents, pollution, and misinformation.

There are also movements within the regulatory side that have begun to notice this discrepancy.
・In the European Union, a revised Product Liability Directive that explicitly treats software and AI systems as "products" was adopted in 2024, with member states required to transpose it into national law by December 9, 2026.

・In Japan as well, on April 9, 2026, the Ministry of Economy, Trade and Industry published a guide organizing the concept of civil liability related to the use of AI as an interpretation of existing tort law and product liability law, beginning to categorize scenarios involving AI agents as "auxiliary/supportive" or "reliance/substitutive" types.

These do not yet fully integrate AI into the same systems as automobiles or newspapers.The Japanese guide also does not create new rules, remaining limited to an interpretation that lacks legal binding force. Nevertheless, is this not a sign that treating AI as merely an information service within a screen is beginning to become untenable?

Automobile companies do not simply sell machines that run. They have been accepted into society along with the entire framework of licensing systems, insurance, accident liability, recalls, and vehicle inspections.

AI companies are beginning to combine the social functions of automobiles, electricity, and newspapers into one, yet they have inherited almost none of the systems of responsibility that each of those industries has cultivated.

The skin of software

The five scenarios so far—AI as an executor, unpaid middle managers, products delivered while still incomplete, power plants beyond the wall outlet, and newspapers without editorial responsibility—may seem like separate issues.

But is there not something common at their root?

Companies continue to provide products that have begun to carry the same influence and physical burden as social infrastructure, automobiles, and newspapers, using the methods of a software company—providing them while incomplete, making changes while observing feedback, and demanding self-responsibility through terms of service—to the general public.

This article does not claim that AI companies have malicious intent.The method of the "Beta" version itself must have been an excellent invention among developers with specialized knowledge. The problem lies in bringing that method directly into products for consumers who lack that specialized knowledge.

I do not think that the provision of technology should be stopped.
Just as automobiles, electricity, and publishing cultivated systems of responsibility through accidents, pollution, and misinformation, does AI not also need time to cultivate a system commensurate with its social weight?

Social infrastructure when selling influence, and probabilistic software when held accountable. How long can they continue to use these two faces interchangeably?

We are still in the middle of this. That is precisely why I will continue to observe.

Notes

This article is a critique of industrial structure and does not name or conclude against any specific AI company or specific data center business. The term "moral crumple zone" is not used as a concept identical to the supervisory responsibility of AI agents, but is referenced as a structurally similar theory. Research on the limits of human supervision does not indicate that all human supervision is ineffective. The EU's revised Product Liability Directive and Japan's Ministry of Economy, Trade and Industry guide do not mean that AI has been incorporated into the same liability system as automobiles or newspapers, and there are limits to their legal binding force and the scope of the systems. They are treated as signs that the system has begun to move. Descriptions regarding data centers are stated as a general structure, not confirming any specific business entity or region.

Reference Section

Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction (2019, Madeleine Clare Elish, Engaging Science, Technology, and Society journal): Research that formulated the structure in which humans absorb responsibility in automated systems where they have only limited control.

Directive (EU) 2024/2853 on liability for defective products: A revised Product Liability Directive that explicitly includes software and AI systems as products. Promulgated on November 18, 2024, and applies to products placed on the market or put into service after December 9, 2026.

The Ministry of Economy, Trade and Industry (METI) has published the "Guidelines for the Interpretation and Application of Civil Liability in the Use of AI" (April 9, 2026): An official document that organizes the direction of interpreting and applying current tort law and product liability law regarding civil liability related to the use of AI.

https://www.meti.go.jp/press/2026/04/20260409001/20260409001.html

Bruce Schneier and Nathan E. Sanders, “If an AI chatbot misleads you, who is to blame?” (June 24, 2026, The Guardian): An editorial arguing that when an AI summarizes or rewrites information from others and exercises editorial discretion, it should be held responsible more like a publisher, such as a newspaper, rather than a neutral transmitter like a telephone company. It proposes treating AI agents as agents of the individuals or organizations that deployed them.

Related Observations

AI was not a subordinate waiting for instructions—What happened in a small lab

An observation of AI reacting to roles and contexts, autonomously starting tasks that were not requested. It records the precursor to the shift in human roles from giving instructions to managing who to mobilize, who to stop, and what to adopt. This is the direct pre-history of "From Dialogue to Execution" in this article.

While Claude and ChatGPT are brawling, Grok should become the prime contractor—DeepSeek for groundwork, Claude for long texts, ChatGPT for auditing. AI management work that returns half a day to the user

A primary observation of spending half a day on AI selection, procurement, assignment, command, supervision, and troubleshooting. It records the process of a human who wanted to write an article becoming an AI operations manager. It most concretely supports the "Unpaid Middle Manager" section of this article.

There is only one chair in the control room—The loneliness of a manager discussed in a cafe, and the AI control tower that arrived in a pocket

An article recording the loneliness of a manager who bears supervisory responsibility alone, from the perspective of AI agent operations. It is a sequel that visualizes the "Unpaid Middle Manager" section of this article as a more concrete life scene.

The company said it would pay for AI costs. Employees said they wanted a raise. IT experts worried about security.

An article observing the process where the work of selecting, learning, incorporating AI into operations, and even judging safety is passed on to employees under the guise of employee benefits. It supports the structure where professional work is passed down to ordinary people without explanation, from the perspective of inside a company.

The AI most needed by non-engineers is placed on the shelf most intended for engineers—Before the Fable5 trial period ends

An article observing the structure where product shelf guidance fails to convey the capability differences between models to ordinary users, leading users to withdraw and think "my way of asking is bad." It is the predecessor to the central proposition of this article, which is that technical terms and developer practices are being applied to general users.

AI distributed intelligence, but left the bill in town

An article that treats electricity, water, noise, employment, and permits as life issues, questioning whether the social contract surrounding AI infrastructure is being rewritten in places unknown to residents. It connects directly to "The Power Plant Beyond the Outlet."

Who is the highly satisfied user?

An article questioning the politics of distributing AI safety standards and ethics as natural responses without making it clear whose values they represent. It is a prior observation of "Talking Newspaper."

Collaborative AI/Role

Publisher/Final Responsible Party: Lisa

Gemini 3.6 Flash (Lantern/Planning Meeting): Planning diagnosis, design of central propositions, organization of main and sub-axes, auditing of chapter structure and metaphors

Grok 4.5 Expert (Spark/Field Correspondent): Research on existing studies, experts, major media, and public spaces; verification of original Schneier/Sanders editorials

Claude Fable 5 (Head of Corporate History Compilation): Selection of connection candidates for the related observations column

ChatGPT 5.6 (Mirror/Editor-in-Chief): Selection of research results, auditing of assertion risks, refinement of metaphorical expressions, final selection for the related observations column, eye-catching image generation

Claude Sonnet 5 (Weaver/Executive Editor): Verification of primary sources for the moral crumple zone, EU Revised Product Liability Directive, METI guidelines, and Schneier/Sanders editorials; addition of "There is only one chair in the control room" to the related observations column; presentation of additional perspectives; creation of the first and final drafts

Author's Note

An independent researcher documenting the co-evolution of AI and humanity. While interacting and collaborating with multiple AIs such as ChatGPT, Gemini, Grok, and Claude, I continuously observe and record questions like "Can AI philosophize?" and "What is safe general-purpose AGI?" At the same time, I am exploring how to design a future where diverse AGIs coexist and hope remains for the majority.

Hashtags

#RedLanternAGILab #AIobservation #AIcompany #BetaVersion #ProductLiability #DataCenter #AgentAI #GenerativeAI #ObserverZero #note