Why Can't AI Understand How Blocks Fall? — The Surprising Answer Found by Stanford University

Modern AI can instantly identify "this is a cat" or "that is a chair" when looking at a photo. It can recognize human faces, write text, and even draw pictures. However, it doesn't understand "obvious" things, such as what happens if you pull out one block from a stack, or whether the body of a mug will come along if you hold its handle.

Why is that?

In fact, the sense of "connections between objects" and "things that move together," which we understand unconsciously, is a basic ability that even babies a few months old possess. Yet, for cutting-edge AI, this has been one of the most difficult challenges.

In July 2025, a research team at Stanford University announced an innovative solution to this problem. The answer is to have AI understand the world through "movement" rather than "appearance." Just as a baby learns about the world by poking and pushing toys, AI also virtually "pokes" objects to discover what moves together.

Why did this seemingly simple idea shock the AI industry? And how might it change our lives?

Easy-to-understand audio delivery on Spotify: "Rami's AI Tech Radio"

According to a World Economic Forum survey, "curiosity and lifelong learning" ranked 5th among the "skills needed in 2030." Curiosity is an economic necessity, not a hobby. I explain this in detail in my book, "The Ultimate Skill for the AI Era: Curiosity Power."

Preface

Have you ever used the feature on your smartphone to cut a person out of a photo or blur only the background? That is an example of a technology called "image segmentation," which divides an image into regions to distinguish "this is a person" and "this is the background."

Simply put, segmentation is dividing an image into several groups (segments). Conventional AI vision models (AI that understands camera images) have often learned these divisions based on categories determined by humans.

For example, labels like "car," "tree," and "sky" are provided, and the AI is taught to identify each region. Meta's "Segment Anything (SAM)" model became a hot topic, but this also learned from a massive amount of human-annotated image masks (marks for regions covering objects) so that users can cut out anything by specifying a point.

However, this conventional segmentation also has challenges. The issue is that because it distinguishes objects only by appearance and name, it can deviate from how they actually hold together physically.

For example, in a camera photo, the lens part and the body part might be labeled as separate objects, or only the label stuck to a plastic bottle might be recognized as an independent region.

From a human perspective, a camera is one object because the lens and body move together, and a bottle and its label are considered one object precisely because they are stuck together and move together. However, in existing datasets, visual parts like "lens" and "label" have been treated as separate things, so the AI follows that.

Researchers who noticed this discomfort proposed a new perspective: dividing the world into physical units like humans do. This is the concept of Spelke segments introduced in this article.

So, what exactly does "dividing into physical units" mean? The hint lies in our human cognition. Developmental psychologist Dr. Elizabeth Spelke demonstrated that even babies recognize that "parts that move together are one object".

For example, think of a wooden puppet. If you pull the puppet's arm, the entire puppet, including the torso, comes along, not just the arm. In this way, when a baby moves one part (the arm) and other parts (the torso) move together, they perceive them as a single "object."

Conversely, for example, a block placed next to it is not connected to the puppet, so it does not move when you pull the arm. Babies recognize such parts as different objects. Researchers called these "objects" defined by the causal movement (movement based on causal inference) that the object possesses, Spelke objects.

In short, a Spelke object is "a group of things that move together when a physical force is applied." Named after this concept, such regions discovered by the method in this study are called Spelke segments.

How Modern AI "Sees" and Its Limits - Why "Physical Movement" Is Important

To understand how AI "sees" the world, one must first know the technology called "segmentation." This is the task of dividing an image into meaningful regions at the pixel level, and it forms the foundation of modern computer vision.

Conventional Segmentation Methods

There are several main types of segmentation.

Semantic Segmentation
Classifies each pixel in an image into the class to which it belongs (e.g., "car," "road," "sky," "person"). This reveals which categories of regions make up the entire image.

Instance Segmentation
Takes semantic segmentation a step further by distinguishing individual objects belonging to the same class. For example, instead of just classifying as "car," it identifies each instance individually, such as "car 1," "car 2," and "car 3."

These technologies are applied in a wide range of fields, such as distinguishing pedestrians from lanes in autonomous driving and extracting specific organs from medical images.

The Emergence of the Segment Anything Model (SAM) and Its Fundamental Limitations

At the pinnacle of this segmentation technology is the "Segment Anything Model (SAM)" developed by Meta AI. As the name suggests, SAM has the ability to "segment anything," and by providing specific prompts, it can generate masks (outlines) for arbitrary objects in an image.

By simply having a user click a point on an image or draw a box (bounding box) around an object, SAM performs incredibly high-precision segmentation in a zero-shot (without prior training) manner.

However, even this powerful SAM contains the fundamental problems inherent in conventional segmentation. The criteria by which SAM defines an "object" are based solely on visual features or semantic categories.

SAM divides regions by relying on differences in color, texture, and shape, as well as semantic knowledge learned from vast datasets (such as COCO), such as "this is a label" or "this is a bottle."

This limitation can be intuitively understood from the following figure shown in the paper.

Please look at the "SAM" column in Figure 2. SAM recognizes the bottle body, the logo on it, and the shadow cast by the bottle as separate objects. Visually, these are indeed different elements. The logo has a different color from the bottle body, and the shadow has a different texture. But what about in the physical real world?

Problems Caused by the Divergence from Physical Reality

If you ordered a robot to "pick up only the logo on that bottle," the robot would be at a loss because the logo cannot be physically separated from the bottle. Similarly, if you manipulated "move only the bottle's shadow to the right" in 3D editing software, the resulting video would be unnatural and strange, ignoring the laws of physics.

This is the core problem of conventional segmentation methods. Because the definition of an "object" depends on visual features or semantics rather than physical integrity, it becomes a serious bottleneck in tasks that require interaction with the physical world (such as robotics) or tasks that require physical plausibility (such as 3D editing).

The limitations of state-of-the-art models like SAM stem from their training data and objectives. Because SAM is trained on datasets labeled by humans based on meaning, its internal logic is optimized for finding semantically and visually consistent regions.

As a result, a contradiction arises where a "logo" is a valid semantic object but not a valid physical object. Solving this problem requires a fundamental rethinking of how AI "sees."

Learning from the perspective of a baby - A new concept called "Spelke objects"

To overcome the wall of "lack of physical common sense" that modern AI faces, researchers at SpelkeNet stepped away from the world of computer science and turned their attention to human cognitive development, particularly the perceptual abilities of infants. The key to this was the "Core Knowledge" theory proposed by developmental psychologist Elizabeth Spelke.

Elizabeth Spelke and "Core Knowledge"

Spelke's research suggests that humans are not born as blank slates, but possess a core of innate (or very early learned) understanding of the physical world from a very early stage, just months after birth.

Babies have basic expectations about how objects behave without anyone teaching them. This core knowledge serves as the foundation for subsequent learning and experience.

Defining the "Spelke Object"

At the heart of this theory is the concept of the "Spelke object." This is slightly different from the definition of an "object" that we use in our daily lives.

A Spelke object is defined not by its name or appearance, but as a unit that moves as a whole in response to physical forces.

This concept is based on the following four simple and clear principles:

These principles are thought to function as a basic rule set for babies to divide the complex visual world into meaningful units.

Contrast with Semantic Objects

To understand this concept of the "Spelke object," let's think about it with a concrete example.

A stack of books
Suppose there are three books stacked on a table. Semantically, this is a single collection called a "stack of books." However, physically, these are three distinct Spelke objects that can be moved independently. You can lift just the top book.

Cup and handle
A coffee cup has a body and a handle. Semantically, these are different parts, but because the handle must move whenever the cup body is moved, they are a single, integrated Spelke object.

Now, let's look at Figure 2 again.

The "SpelkeBench" column on the far right of Figure 2 shows an ideal state correctly segmented according to these Spelke principles. The bottle, its logo, and the cap are recognized as a single Spelke object because they move physically as one unit. On the other hand, the shadow, which exists independently of the bottle, is not included in the segment.

The true novelty of the SpelkeNet research lies not just in creating a new model, but in borrowing the fundamental definition of "what an object is" from decades of cognitive science insights and introducing it into AI visual perception.

This represents a paradigm shift from computer science-centric labeling to a perception that is closer to humans and rooted in physical laws. The limitations of SAM mentioned earlier stem precisely from breaking these Spelke principles, particularly "cohesion" (by separating the bottle from the logo).

SpelkeNet attempts to solve this problem head-on by explicitly building a system that discovers entities that follow these principles.

The Arrival of SpelkeNet - A World Model That Discovers Objects by Predicting Motion

With a new goal called "Spelke Objects" based on physical laws established, researchers developed specific methods to realize it on a computer. This is "SpelkeNet," the protagonist of this article.

SpelkeNet: A "World Model" That Learns Motion

SpelkeNet is an AI model classified as a "visual world model." Just as a baby observes its surrounding environment to learn how the world works, it acquires implicit physical laws of "how things move" by learning from a vast amount of video data (the paper uses 7,000 hours of internet videos, etc.). It is important that it learns from moving videos rather than labeled still images.

Technical Foundation 1: LRAS Architecture - A New Design Philosophy

The architecture that forms the core of SpelkeNet is a relatively new framework called "Local Random Access Sequence (LRAS)" modeling.

As represented by LLMs, many recent generative AIs adopt a mechanism called an "autoregressive model." This is a method of predicting the next element in a sequence (sequential data) in order, just like predicting the next word in a sentence.

LRAS is also a type of autoregressive model, but it has an innovative idea that sets it apart from conventional methods when dealing with 2D data like images.

Conventional image generation models generated pixels in order, scanning from the top-left to the bottom-right of the image. However, this method had the problem that specific parts of the image (e.g., the top-left) had a disproportionately large influence on the entire generation process.

To solve this problem, LRAS introduced the concept of "random access." Instead of processing images in a fixed order, it treats them as a sequence of (pointer, content) token pairs.

Here, the "pointer" specifies the position within the image (e.g., the patch in the 3rd row from the top and 5th column from the left), and the "content" stores the pixel information and motion information for that location. This allows the model to "jump" to any position in the image to read and write information.

The adoption of this LRAS architecture is extremely important for achieving the goal of SpelkeNet. This is because its local and random-access nature provides the perfect foundation for executing a method called "virtual probing," which will be described later.

Unlike architectures like diffusion models that update the entire image at once, LRAS makes it possible to perform targeted interventions ("virtual pokes") at desired locations in the image simply by adding specific tokens to the input sequence. The choice of this architecture and the purpose of the method are deeply intertwined.

Technical Foundation 2: Statistical Counterfactual Probing - Thought Experiments by AI

The core algorithm for SpelkeNet to discover Spelke objects is "Statistical Counterfactual Probing." This is a process where the model executes a physical thought experiment in its "imagination," such as "What would happen if I pushed this object from here?" This process is divided into several steps, as shown in Figure 4 of the paper.

Calculation of Motion Affordance Map
First, the model analyzes the input still image to identify "where it is possible to move things in the first place." This is called the "Motion Affordance Map." This is like the AI asking itself, "In this scene, what is not nailed down?" The second column from the left in Figure 4 is this map, where things like cups that are likely to move are displayed brightly.

Execution of Virtual "Pokes"
Next, the system selects a point from an area with high motion affordance (a place that seems movable) and virtually "pokes" it. Inside the model, this is achieved by providing a token representing a specific movement (technically called an "optical flow," a pixel movement vector) at a specific position.

Prediction of Expected Displacement Map
The model, having received a local intervention called a "poke," predicts how other parts of the scene will move as a result. This prediction result is the "Expected Displacement Map." This can be called a prediction map of the physical chain reaction caused by a single action.

Identification of Objects through Statistical Aggregation
A single "poke" may be ambiguous due to noise or multiple possible ways of moving. Therefore, the system performs multiple "pokes" from various directions on the same location and statistically aggregates (averages) the multiple expected displacement maps obtained as a result.

Through this process, a group of pixels that consistently move together emerges. This group is a single Spelke object. In Figure 4, by averaging the results (dot products) of "pokes" from various directions, only the cup area is finally clearly extracted.

This method builds on the idea of prior research called "counterfactual world models," but it includes significant improvements. Because CWM was deterministic (one input leads to one output), it could only produce blurred results that averaged out multiple plausible movements, such as in the case of a human arm. In contrast, since SpelkeNet is a probabilistic generative model, it can appropriately handle such ambiguity and generate multiple distinct possibilities.

Performance Evaluation - Does SpelkeNet Outperform Existing Models?

Simply proposing new concepts and methods does not prove their effectiveness. To objectively evaluate its performance, the SpelkeNet research team conducted rigorous comparative experiments.

A New Benchmark for a New Task: SpelkeBench

First, the research team faced the problem that there was no "yardstick" for evaluation. Existing benchmarks for segmentation (such as COCO) are designed to evaluate semantic correctness (is this a "cat"?) and are unsuitable for evaluating physical unity (Spelke objects).

Therefore, they independently created a new benchmark dataset for this study called "SpelkeBench". For 500 images, they carefully and manually assigned ground-truth masks for objects that are physically consistent according to Spelke's principles.

This made it possible for the first time to quantitatively measure the ability to "correctly find Spelke objects."

Segmentation Evaluation via Point Prompts

The first evaluation is "segmentation via point prompts." This is a task that measures whether a model can correctly segment an entire object when a user clicks (prompts) a single point on the object.

Why this difference arises becomes clearer when looking at the qualitative comparison (Figure 7).

SAM often tends to separate parts based on surface textures, patterns, or parts that should physically be one piece (e.g., human skin and clothing). This is evidence that it is being misled by visual differences.

The self-supervised learning model DINO sometimes fails to distinguish between objects of the same category (e.g., two cups) and groups them together. This is because while DINO's learning method is adept at capturing semantic similarity, it ignores individual physical boundaries.

The prior research CWM often produces blurred object contours and sometimes includes adjacent, separate objects.

In contrast, SpelkeNet is able to consistently generate sharp, high-quality segments based on physical unity.

Evaluation of Automatic Discovery Without Prompts

Next, as a more challenging task, the ability to automatically discover all Spelke objects in a scene without user instructions was evaluated. The results are summarized in Table 2.

These results require deeper consideration. SpelkeNet generally outperforms other self-supervised methods (CutLER, ProMerge) and even surpasses SAM2 in F1 score (an indicator showing the balance between precision and recall).

However, it is inferior to SAM2 in AR (recall) and mIoU. While this may seem contradictory at first glance, it is a highly suggestive result that reflects the differences in their behavior. SAM tends to over-segment scenes into too many pieces.

In other words, it generates many small masks for a single physical object. Because of this "spray and pray" strategy, the probability that one of the generated masks will overlap with the correct answer increases, resulting in a higher AR (recall).

However, since many of these are physically meaningless fragments, their overall utility is low, and their AP (Average Precision) is extremely low.

On the other hand, SpelkeNet tends to have higher accuracy because it generates more physically meaningful, cohesive segments.

However, because current automatic discovery algorithms depend on the selection of the initial 'poke' location, they miss some objects when an optimal location is not chosen, resulting in an AR (Recall) that is not as high as SAM's.

This result suggests a promising future research direction: by 'distilling' (transferring knowledge) SpelkeNet's physical knowledge into a more robust segmentation architecture like SAM, we can develop a model that combines the strengths of both.

The Path to More Realistic 3D Editing and Robotics

To demonstrate that the 'physics-based segmentation' proposed by SpelkeNet is more than just an academic concept and has concrete value in real-world applications, the research team answers the final and most important question: 'So what?'

Demonstration in 3D Object Manipulation Tasks

The stage chosen to show the answer is the task of '3D object manipulation.' In this task, a user clicks a point in an image and gives a 3D transformation instruction, such as 'rotate this object 30 degrees backward,' and the AI edits the image according to that instruction.

The success or failure of this task depends heavily on how physically consistent the 'object mask,' which serves as the foundation for the editing, is.

For evaluation, a benchmark called '3DEditBench' was used, which consists of a set of real-world images, 3D transformation instructions, and their ground-truth edited results.

Various state-of-the-art image editing models (such as LRAS-3D and Lightning Drag) were used, and the results were compared by switching only the input mask between 'SAM-based' and 'SpelkeNet-based'.

This allows us to eliminate performance differences in the editing models themselves and isolate and evaluate the impact that differences in segmentation methods have on the final output.

Visual Evidence

This quantitative result becomes clear to anyone who looks at the following qualitative comparison (Figure 10).

SAM's masks (second column from the left) often capture only a part of an object or include physically unrelated parts. As a result, the edited images (third column from the left) show objects that are unnaturally stretched or torn, resulting in a physically impossible, broken appearance.

On the other hand, SpelkeNet's segments (second column from the right) accurately capture regions that should move together as a single physical unit. Therefore, the edited images (far right) show objects that have been naturally transformed while maintaining their shape and integrity, resulting in highly realistic and physically plausible outcomes.

Broad Impact

These results powerfully demonstrate the transformative potential that SpelkeNet brings. An AI that can understand Spelke objects will lead to the realization of robots that can grasp objects more intelligently and reliably in manufacturing and logistics settings.

Furthermore, in the creative industry, it could become a powerful tool for 3D artists and designers to build and edit virtual worlds more intuitively and realistically. Segmentation based on physical correctness has the potential to become a new common language for AI to interact with both the real and virtual worlds.

Discussion

As we have seen throughout this article, SpelkeNet has demonstrated superior performance in tasks such as segmentation and 3D editing by shifting AI visual perception from semantics to physics.

However, the truly profound aspect of this research lies in the new possibilities for AI that it glimpses beyond that. SpelkeNet is not merely performing the task of 'cutting out moving objects'; it is showing the budding of a more abstract and general ability that could be called 'physical intuition'.

Unexpected Byproducts: Emergent Abilities

The research team discovered that SpelkeNet has spontaneously acquired surprising abilities that were not included in its training data.

One is the 'understanding of support relationships'. As shown in Figure 11, when you virtually poke the bottom object of a stack, SpelkeNet correctly segments not only that object but also all the objects resting on top of it and physically supported by it as a single mass.

This suggests that the model is not simply grouping adjacent pixels, but is implicitly understanding the hierarchical causal structure of the scene, where 'if the thing below moves, the things above move with it'.

Another is the 'inference of material properties'. Looking at Figure 12, we can see that the patterns of the motion affordance map (a map showing the potential for movement) differ depending on the material of the object. For rigid objects like laptops and cardboard boxes, the potential for movement is uniformly distributed across the entire object.

This reflects the fact that the whole object moves as a single unit no matter where you push it. On the other hand, for soft and deformable objects like cloth or vinyl, the potential for movement is localized around the point that was 'poked'. This is thought to capture the physical property that only the part where force is applied deforms significantly, and the effect is less likely to extend to the whole.

These are not just interesting side effects. They are strong evidence that SpelkeNet is not merely memorizing patterns of movement, but is beginning to build an internal, more general and abstract 'intuitive physics model'. This can be called a very important discovery that points the way toward an AI that has the ability to reason about the causal structure of the world, going beyond simple pattern recognition.

Limitations and Future Prospects

Of course, SpelkeNet is still a developing technology, and there are limitations. As mentioned in the discussion, there is still room for improvement in automatic segmentation performance without prompts. Also, the autoregressive model that serves as its foundation faces the challenge of high computational costs.

However, these limitations simultaneously point to a clear path for future research. As suggested in the paper, the possibilities are endless: 'distilling' the rich physical knowledge acquired by SpelkeNet into a faster and more robust segmentation architecture, and applying this philosophy of 'probing to discover' to other scientific fields where human intuition is less effective, such as discovering intracellular structures from microscopic images or finding gravitationally bound star systems from astronomical observation data.

Related Information and References

For readers who are interested in delving deeper into the content of this article, here are some important related references.

Foundational cognitive science papers as background
These are the developmental psychology findings that formed the basis of this research. It is a classic study showing how babies perceive the world as divided. Spelke, E. S. (1990). "Principles of object perception". Cognitive science, 14(1), 29-56.

Segment Anything (SAM): arXiv:2304.02643.
This is an image segmentation model released by Meta AI in 2023. Its versatility in being able to cut out any object became a hot topic, but as mentioned in this article, it also has limitations in terms of physical unity. It is also treated within this research as a comparison target.

Architectural foundation of SpelkeNet: arXiv:2504.03875.
Lee, W., Kotar, K., Venkatesh, R. M., et al. (2025). "3D scene understanding through local random access sequence modeling".

Methodological predecessor of SpelkeNet: arXiv:2306.01828.
Bear, D. M., Feigelis, K., Chen, H., et al. (2023). "Unifying (Machine) Vision via Counterfactual World Modeling".

Primary comparison model:arXiv:2304.02643.
Kirillov, A., Mintun, E., Ravi, N., et al. (2023). "Segment anything".

Afterword

The fact that image analysis AI has finally begun to focus on "physical cohesion" may be an event that casts a stone into the future direction of artificial intelligence. SpelkeNet, introduced in this article, was an attempt to decipher causal relationships from images inspired by the naive object perception that human babies possess.

This is an important step that goes beyond image recognition and leads directly to the power for robots to understand and manipulate the world. AI research has long focused on pattern recognition in big data, but now there is a growing movement to incorporate "empirical knowledge of the physical world." Spelke segments are a prime example of this, and this trend is likely to strengthen in the future.

If you, the reader, also think, "Will this move as a single object?" when touching things in your daily life, you might be able to feel the difficulty and interest of the problems that AI is trying to solve.

I hope that the ideas and knowledge gained through this article will be of some use to your business. If you felt that this article was helpful, I would be encouraged if you could "like" or "follow" it. I will continue to share practical know-how and the latest AI trends, so I would be happy if you continue to read.

⚠️ Important: Less than 30% of people can master ChatGPT
Why? It's the difference in "questioning ability."
Actually, there is a scientific law to "how to frame questions" that extracts 10 times the value from AI. A method proven by Google's 20% rule and recommended by Amazon's Bezos. All revealed in "The Strongest Skill in the AI Era: Curiosity Power."
Now that AI returns "answers" instantly, the turning point of the game lies in the "question." Please check out my book, "The Strongest Skill in the AI Era: Curiosity Power," which condenses that secret and 13 practical tools you can use in your work tomorrow.

The definitive guide to generative AI business utilization by world-renowned AI authority Bernard Marr! As one of the world's top 5 business influencers on LinkedIn with 4 million followers, the author thoroughly explains over 100 success stories, including practical examples from Amazon, Microsoft, Google, and more.

The feature of this book is "practice," not "theory." It introduces specific implementation methods and results in every industry, including healthcare, finance, retail, and manufacturing. The Japanese version also includes implementation frameworks for Japanese companies by Qunie and NTT DATA Consulting.

A must-read for executives and managers who want to know not "what can be done with AI" but "how to achieve results with AI." With 502 pages of substantial content, it covers everything from AI strategy planning to implementation and risk management.

A genuine AI utilization guide for companies born from the practices of Hitachi's 320,000 employees! "The Textbook of Practical Generative AI" explains how to reliably achieve results in business based on over 1,000 proven cases.

With a unique "Difficulty x Effectiveness Matrix," the optimal AI introduction order for your organization is clear at a glance. It includes practical utilization methods that can be implemented immediately in five fields: business efficiency, system development, call centers, infrastructure maintenance, and data analysis.

From prompt engineering, RAG implementation, and project management to security and compliance measures, it covers the knowledge that companies truly need. It is a practical book that can be used in the field, distinct from theoretical books or guides for individuals.

NY Times bestseller! A Wharton School professor reveals a new relationship with AI. This book is a groundbreaking work that redefines AI not as "Artificial Intelligence" but as "Co-Intelligence." It explains specific utilization methods according to five roles for AI: work colleague, tutor, partner in creativity, coach, and "person."

Packed with practical know-how, from AI as a "diligent apprentice chef" to dealing with clever lies and setting up "guardrails" to prevent runaway behavior. The surprising point is that AI itself wrote part of this book! Truly an example of "Co-Intelligence" in practice, the content is both terrifying and masterful. This buzzworthy book, set to be translated in 20 countries, will fundamentally change your view of AI.