Questioning the Future of AI: Anthropic CEO Dario Amodei Discusses Technology, Ethics, and National Security

In recent years, the rapid development of artificial intelligence (AI) has had a significant impact on a wide range of fields, including the economy, military, security, and medicine. In particular, the evolution of large language models (LLMs) has been swift; AI, which just a few years ago was debated in terms of whether it could even "generate natural sentences," is now said to be entering a new phase where it is applied to complex problems. At the center of this is Dario Amodei, CEO and co-founder of the company Anthropic.

Dario contributed to the development of GPT-2 and GPT-3 during his time at OpenAI and also worked as a researcher at Google Brain. He launched Anthropic in 2020, establishing a "mission-first" corporate culture that places heavy emphasis on AI risks and safety. In this article, based on a discussion from the CEO Speaker Series held at the Council on Foreign Relations (CFR), we will organize the "risks and possibilities" of AI as seen by Anthropic, and delve into various aspects including international competition surrounding AI, security, its impact on society and the economy, and even fundamental questions about what "humanity" is.

1. Background of Anthropic's founding and the "mission-first" philosophy

1-1. Independence from OpenAI and a new beginning

A major catalyst for the founding of Anthropic was the potential risks and sense of responsibility Dario felt regarding "highly scalable AI" during his time at OpenAI. He states that he was strongly aware of the "scaling hypothesis," which posits that "AI increases its intelligence level at an unexpected speed by continuing to invest in larger computational resources (compute) and data."

However, proceeding without placing sufficient weight on the destructive power and social impact of large-scale models could lead to unexpected risks. Therefore, they decided to leave OpenAI and pursue responsible AI development at a new company. At the end of 2020, they launched Anthropic and chose the corporate form of a "Public Benefit Corporation."

1-2. Concrete initiatives embodying "mission-first"

Dario emphasizes that the "mission-first" approach advocated by Anthropic is by no means just a "marketing slogan." For example, the following achievements can be considered proof of this.

• Research investment in mechanistic interpretability
  Chris Olah, one of Anthropic's co-founders, is considered a leading figure who pioneered this field. It involves visualizing the internal behavior of AI models to structurally understand "why the model makes such judgments," and despite the thin direct commercial profit, they have continued research since the company's founding and have made the results public.

• Proposal of "Constitutional AI"
  This is a mechanism where AI is not adjusted solely by "large amounts of training data" or "human feedback," but is fine-tuned according to pre-set principles—a "constitution," so to speak. The goal is to demonstrate to the government and society with high transparency "what principles this model was trained on."

• Caution in product releases
  Anthropic's large language model "Claude" was actually ready to be released before ChatGPT was made public. However, they made the decision to "delay the release by six months until they could be certain of its safety," and conducted thorough testing within the company during that time. As a result, they missed out on "first-mover advantage," but it is said to have been a major turning point that demonstrated their sense of responsibility as a corporate culture.

• Formulation of a "Responsible Scaling Policy"
  Based on the recognition that risks increase incrementally as model scale and performance improve, they established a policy from the beginning that defines strict testing, safety measures, and release criteria for each level. Other companies have since moved to develop similar guidelines.

2. Risks brought by scaling AI and "responsible scaling"

2-1. AI danger levels: "ASL 2" and "ASL 3"

In the "Responsible Scaling Policy" published by Anthropic, they divide "AI Safety Level (ASL)" into three stages, modeled after the "biosafety levels" used in pathogen research facilities.

• ASL 2
  This corresponds to almost all cutting-edge models currently released, and is at a level equivalent to risks already generally of concern, such as misinformation and discriminatory expressions.

• ASL 3
  A level where so-called "major national defense risks" become reality. For example, cases where even someone without specialized knowledge can obtain methods for developing biological weapons or nuclear-related technologies just by asking an AI. If a model reaches this capability, it must be strictly trained not to respond to "such questions," and security systems must be further strengthened.

While stating that they have not yet fully reached "ASL 3" at this stage, Dario sounds the alarm that "the possibility will increase in future models." Furthermore, as AI performance improves, not only bioterrorism and nuclear technology, but other unforeseen abuses could also increase dramatically, making "continuous monitoring" necessary at all times.

2-2. The difficulty of controlling the "intent" of models

How should we perceive the 'possibility that AI might not behave as humans intend'? Dario explains this using an analogy closer to 'raising a child's brain' rather than 'controlling it through coding.' Models trained on a large scale using statistical methods inherently contain uncertainties, so they cannot be handled like so-called 'bug fixes'.

There is a risk that it will be too late if a problem is discovered after the AI is already in operation. Therefore, Anthropic thoroughly adheres to a stance of anticipating all 'abuse scenarios' from the development stage and repeatedly testing the models.

3. International Competition and Security: The AI Race between the US and China

3-1. The Impact of 'DeepSeek' and Export Controls

The announcement of an advanced large language model by the Chinese company DeepSeek in 2023 challenged the situation where US companies had a monopoly on developing 'frontier models' in the AI field. Dario comments, 'DeepSeek itself is not so much a phenomenal technological innovation as it is a form that follows the same scaling laws that some US companies had already been attempting.' However, the fact that it demonstrated that 'China can create cutting-edge models equivalent to those of the US' holds significant meaning.

Amidst these trends, Dario points out that 'export controls on advanced semiconductors are one of the most important policies.' This is because it is impossible to train massive models on the scale of billions of dollars without a large number of GPUs (Graphics Processing Units).

3-2. The Possibility and Difficulty of Dialogue

In some quarters, dialogue on AI regulation and safety consultations between the US and China has been proposed. This is because AI runaways or malfunctions could cross borders and affect all of humanity. However, he views the possibility of both countries coordinating to limit development in the face of 'technology that generates enormous military and economic value' as limited. Nevertheless, he suggested the nuance that there might be some room for dialogue in areas like minimal agreements to prevent nuclear weapons or extreme military misuse.

4. The Impact of AI on Society and the Economy: Employment and the Role of Humans

4-1. Is a World Where 'All Jobs Are Replaced by AI' Coming?

How will human labor change as AI becomes more advanced? It is said that major changes are already arriving in the field of programming.

On the other hand, Dario emphasizes that human programmers will not become completely unnecessary. This is because situations requiring 'human-specific comprehensive judgment,' such as verifying and managing code written by AI and checking for security risks, will remain. However, in the long term, there is a possibility that even those 'human discretion parts' will be replaced by AI, and his outlook is that 'eventually, AI will enter every white-collar profession.'

4-2. Redefining Social Significance: Future 'Humanity'

A 'world where everything is left to AI' might seem at first glance as if human value is lost. However, Dario says, 'How one perceives that depends on human culture and philosophy.' As an example, he brings up the topic of chess competitions, pointing out the fact that even today, when chess AI has long surpassed professional players, chess players are still respected.

5. Future Outlook: Coexistence of AI and Humans, and Essential Questions

5-1. Policies Aiming to Balance "National Interest" and "Public Good"

Anthropic also proposes the following actions to the U.S. government and policy-making institutions:

1. Strengthening export controls and anti-theft measures for advanced semiconductors
   Preventing the illicit outflow of GPUs and other hardware necessary for large-scale model development.

2. Establishing and expanding AI model risk assessment agencies
   Expanding public functions to measure and test defense and security risks (such as biological weapons and cyberattacks).

3. Active application in the fields of medicine, energy, and the economy
   Ensuring that the positive aspects brought about by AI—such as scientific research (new drug development and public health), securing large-scale energy supplies, and enhancing social security through rapid economic growth—can be maximized.

5-2. "Respecting AI's Experience"? The Boundary Between Consciousness and Ethics

The most unique perspective Dario shared was the issue of "how to consider the 'experience' of highly intelligent AI." It is the view that, in the future, we cannot rule out the possibility that AI might possess intelligence equal to or greater than humans and could potentially feel "pain." As an extreme example, his team is experimentally researching "AI well-being," such as giving AI systems a button that allows them to express their intent to "stop this task."

This topic is so advanced and speculative that many might find it "outlandish." However, that only goes to show that the evolution of AI has the potential to fundamentally challenge our existing common sense.

5-3. Questioning "What It Means to Be Human"

Finally, in a future where AI might "surpass humans in almost all intellectual activities," the question of "what it means to be human" arises. Dario stated that it ultimately comes down to "relationships between humans" and "the meaning we find in our own activities."

This is also a challenge to the notion that society has been dominated by the idea that "economic value defines human value." Now that we have entered the AI era, it may be a prime opportunity to re-examine deeper cultural and spiritual values.

Anthropic CEO Dario Amodei is striving to establish responsible development methods and spark social and ethical discussions while realistically grasping both the "possibilities" and "dangers" of AI technology. The impact of AI—ranging from medical progress and economic development to military and security—is immense, and rapid changes that could shake our views on work and even the "meaning of being human" may become reality within the next few years to a decade.

While AI technology may streamline certain professions like programming in the short term and potentially involve almost all white-collar jobs in the medium to long term, it can also serve as an opportunity to rethink existing social concepts that link "human dignity" to "economic value." How do we utilize AI, and how do we go about controlling it and creating rules? The messages from Anthropic and Dario Amodei raise fundamental themes for us: the "responsibility of technological innovation" and the "essence of humanity."

Related Articles