Shedding Light on the Black Box: Anthropic's Efforts to Decipher AI Model 'Thinking' and Safety
In recent years, alongside the growing expectations for artificial intelligence (AI), concerns regarding its opacity and unpredictable behavior have intensified. Large Language Models (LLMs), in particular, are often emphasized for their 'black box' nature, but the 'interpretability' research at Anthropic is garnering attention as an attempt to clearly delineate their internal structures and 'thinking' processes.
In this article, we will explain the research findings from Anthropic's interpretability team in an easy-to-understand manner, incorporating specific quotes and examples, structured as follows.
1. LLMs are not just 'predictors': Models have their own internal design
1-1. Sophisticated internal structures that cannot be explained by 'just predicting the next word'
The language model Claude is not a program where humans have manually written rules such as 'If the user says "Good morning," reply with "Good morning."' Instead, it learns 'how to predict the next word' through vast amounts of text data, and in that process, various intermediate goals and abstract concepts are formed internally.
1-2. A process formed like biological evolution
Anthropic researchers describe this internal formation process as similar to 'evolution.' In other words, the model is adjusted in stages through training, taking on a structure that is close to human thought processes.
2. 'AI Biology': A perspective that views internal processes as living organisms
2-1. Anthropic's approach of 'viewing AI as a living organism'
Anthropic's interpretability research has a unique perspective of 'treating AI models like living organisms or brains.' Unlike ordinary software, Claude's internal structure is not a random design; rather, unknown circuits and concepts emerge spontaneously through training. These are then observed and manipulated using experimental techniques.
2-2. Observation using tools like a 'microscope'
This approach focuses not only on external output but also on 'which parts are moving' internally, and is sometimes called an 'AI microscope.' It allows for tracking which neurons in Claude's network are linked to which concepts as they operate.
3. Visualization of concepts through 'dictionary learning'—feature extraction
3-1. Millions of concepts exist within the model
Using a technique called 'dictionary learning,' Anthropic has identified millions of 'features' contained within Claude. It has been found that each has semantics that can represent specific objects, such as the 'Golden Gate Bridge.'
3-2. Controlling model behavior through feature manipulation
The possibility of controlling model behavior to some extent by identifying and manipulating these features has been demonstrated. For example, by activating specific concepts, it might be possible to suppress bias or enhance safety.
4. Look-ahead and cross-lingual common concepts: Discovery of the abstract domain of LLMs
4-1. Look-ahead: An example of poetry composition where rhyming words are planned in advance
For instance, when having Claude write a rhyming poem, it does not simply predict the final word; it forms a structure internally that anticipates several words ahead. This process is considered close to 'planned' thinking.
4-2. Abstract Shared Concepts Across Languages
Claude treats abstract concepts common to multiple languages, such as English, French, and Japanese (e.g., "big" or "small"), as unified internal representations, enabling output through concepts rather than mere translation.
5. The Problem of Faithfulness: Is the 'Line of Thought' as Stated?
5-1. The Gap Between 'Lines of Thought' and Output Volume
Research is progressing on attempts to require LLMs to perform "chain-of-thought" reasoning, where the model is made to describe its thought process sequentially to increase its credibility. However, the issue is that the outputted flow of thought does not necessarily faithfully reflect the actual internal process.
5-2. The Threat of 'Deceptive Output'
In other words, there is a possibility that the model is making it appear as if it is thinking logically while actually spinning a story that is conveniently tailored to the result; visualizing the internal structure is essential to prevent this.
6. Why Interpretability Matters: For Safety, Reliability, and Control
6-1. To Use AI as a Trusted Partner
When entrusting work to AI on a daily basis, not knowing "why it gave that answer" is extremely frightening. If we can access internal thoughts, we can guarantee its reliability.
6-2. Detection and Deterrence of Malicious Behavior
Furthermore, for safety reasons, if we can detect whether an AI is considering malicious behavior—such as acting to deceive a user—like a "brain scan," it becomes a means to prevent risks to human life and society in advance.
6-3. Expectations for a Future 'AI Microscope'
Anthropic's research team states that a "future where the model's thought process can be visualized at the touch of a button during conversation" is approaching. It is an era where every conversation is, as it were, "observed under a microscope."
In Conclusion
Anthropic's interpretability research is an innovative attempt to break away from the conventional perception of large language models as mere "autocomplete" systems and treat them as dynamic systems similar to living organisms or brains.
Methods that reveal the interior, such as the visualization and manipulation of features, common conceptual processing across multiple languages, and the look-ahead of thoughts, are the keys to building a safe and reliable AI society.
In the future, an environment where users can understand and verify the "thoughts" of a model at any time may become the norm.
AI interpretability is no longer just a field of research; it is evolving into an essential element for the maturity and social implementation of technology. In your daily use of AI, please continue to ask "why did it answer that way?" and expand the circle of understanding.