掩码视觉动作：面向统一世界建模

一、掩码视觉动作：面向统一世界建模

核心贡献

论文提出了 Masked Visual Actions（掩码视觉动作），一种将动作直接表达在像素空间中的控制接口，使预训练视频模型能够同时充当前向动力学模型和逆向动力学模型：

• 前向建模：将机器人运动轨迹以掩码形式输入模型，模型预测场景（物体）的响应。
• 逆向建模：将期望的物体运动轨迹以掩码形式输入，模型恢复出实现该目标的机器人运动。

该方法的亮点在于：

• 像素对齐、与形态无关：动作表示为图像空间中的部分时空轨迹，不依赖特定机器人本体（embodiment-agnostic）。
• 数据高效：仅使用约 15 小时的真实与仿真掩码视频数据进行 LoRA 微调。
• 统一框架：单一模型检查点即可同时支持前向与逆向推理，无需为不同任务训练不同架构。

方法概述

1. 数据集构建：结合 DROID（真实世界）和 Robocasa（仿真）数据，通过两种方式生成掩码条件：

   • 基于分割：使用 Segment Anything 分割出机器人臂作为掩码。
   • 基于渲染：利用 URDF 模型和相机外参渲染机器人网格，生成掩码条件。

2. 模型训练：以 Wan-Fun-Control 2.1（14B）为基座模型，采用 LoRA（秩256）微调，将掩码视频与参考帧拼接作为条件输入。

实验与应用

论文在仿真和真实机器人场景中验证了三个下游应用：

实验表明，该方法在视觉保真度和可控性上优于现有基线（如 Ctrl-World、Wan-move 等），并且能够泛化到训练时未见的机器人形态（如 BEHAVIOR 数据集中的双臂机器人）。

Figure 1: Masked Visual Actions. We finetune a video model to condition on masked trajectories of robots, representing robot actions as pixel-space masked motions. Efficiently finetuned on only 15 hours of data, a single checkpoint of the model can act as an action-conditioned forward model to simulate robotic interactions with diverse and unseen embodiments. By conditioning it on object motion, it can also act as an inverse model that synthesizes the robot motion needed to achieve the desired outcome. We showcase the efficacy of our model for policy evaluation, model-based-planning, and action extraction where the video model acts as a policy.
图 1：掩码视觉动作。我们对视频模型进行微调，以条件化机器人的掩码轨迹，将机器人动作表示为像素空间的掩码运动。该模型仅用 15 小时的数据进行高效微调，单个检查点即可作为动作条件的前向模型，模拟具有多样化且未见过具身能力的机器人交互。通过以物体运动为条件，它还可以作为逆向模型，合成实现期望结果所需的机器人运动。我们展示了该模型在策略评估、基于模型的规划和动作提取方面的有效性，其中视频模型充当策略。

摘要

Video models absorb rich priors over how the visual world moves, interacts, and responds to contact, making them promising substrates for robotic world modeling. The central challenge is how to communicate action to such models in a form aligned with the visual space in which they learned these interaction priors, yet still grounded in physical manipulation.
视频模型能够吸收关于视觉世界如何运动、交互以及对接触做出反应的丰富先验知识，这使得它们成为机器人世界建模的有希望的基础。核心挑战在于如何以一种与模型学习这些交互先验知识的视觉空间相一致，但又与物理操作相联系的形式，将动作传达给这类模型。

We introduce Masked Visual Actions, a pixel-space control interface that expresses action as a partially revealed trajectory of an arbitrary entity in a video. Revealing robot motion makes the model act as a forward dynamics model that predicts the scene’s response to low-level robot actions, while revealing desired object motion makes the same model recover robot behavior consistent with that outcome.
我们提出了掩码视觉动作（Masked Visual Actions），一种像素空间控制接口，它将动作表示为视频中任意实体的部分可见轨迹。揭示机器人运动时，模型充当前向动力学模型，预测场景对底层机器人动作的响应；揭示期望的物体运动时，同一模型则能恢复出与该结果一致的机器人行为。

Finetuned with only 15 hours of masked examples from real videos and simulation, a single checkpoint achieves strong visual fidelity and controllability across diverse scenes and multiple embodiments. In downstream manipulation settings, the model produces imagined rollouts whose outcomes correlate with real-world execution for policy evaluation, improves decision making by ranking candidate futures in model-based planning, and supports inverse modeling by synthesizing robot motion from desired object motion.
通过对来自真实视频和模拟的掩码示例进行仅15小时的微调，单个检查点在各种场景和多种具身中实现了强大的视觉保真度和可控性。在下游操作设置中，该模型生成的想象性回放其结果与用于策略评估的真实世界执行相关联，通过在基于模型的规划中对候选未来进行排序来改进决策制定，并通过从期望的对象运动合成机器人运动来支持逆向建模。

1 引言

Purposeful interaction requires connecting an agent’s actions to their effects in the world, and vice versa. In sensorimotor control, skilled behavior is often described as coupling a forward model that anticipates the sensory consequences of movement with an inverse model that recovers the movement needed to realize a desired state [67, 68, 69].
有目的的交互需要将智能体的行为与其在世界上的效应联系起来，反之亦然。在感觉运动控制中，熟练的行为通常被描述为将预测运动的感官后果的前向模型与实现期望状态所需的运动的逆向模型耦合起来[67，68，69]。

Robotic world models should similarly support both directions of reasoning in a single predictive framework.
机器人世界模型应在单一预测框架内同时支持两个方向的推理。

Recent advances in video models offer a promising route to this ambition. Trained on large-scale observation, they accumulate remarkably broad priors over motion, contact, persistence, deformation, and change, far beyond what can usually be distilled from robot data alone. However, most still remain passive observers rather than tools for intervention.
近期视频模型的研究进展为实现这一宏伟目标提供了一条充满希望的途径。这些模型在大规模观测数据上进行训练，积累了对运动、接触、持久性、形变和变化等方面极其广泛的先验知识，远远超出了通常仅从机器人数据中能够提炼出的范畴。然而，大多数模型仍然是被动的观察者，而非可用于干预的工具。

Existing models condition generation on text [53, 63], tracks [13, 20, 56, 58, 82], forces [21, 22, 44], keypoints [66, 70], or motor commands [19, 25], signals that are often sparse, embodiment-specific, or misaligned with the model’s pre-trained visual experience.
现有模型将生成条件化于文本 [53, 63]、轨迹 [13, 20, 56, 58, 82]、力 [21, 22, 44]、关键点 [66, 70] 或运动指令 [19, 25]，这些信号通常是稀疏的、体现特定的或与模型的预训练视觉经验不匹配的。

What remains missing is an action representation expressed directly in the visual space where pretrained video models learned their interaction priors. Once action is expressed visually, the same model can complete different parts of an interaction depending on the trajectory revealed: revealing robot motion prompts a scene response, while revealing object motion prompts robot behavior.
所缺失的是一种直接在视觉空间中表达的行为表征，预训练的视频模型在该空间中学习了它们的交互先验。一旦行为以视觉方式表达，相同的模型就可以根据揭示的轨迹完成交互的不同部分：揭示机器人运动会触发场景响应，而揭示物体运动会触发机器人行为。

To realize this vision, we introduce Masked Visual Actions, a method that recasts action as a visual primitive directly within the pre-trained model’s native representation. We finetune a pre-trained video model [63] to ingest actions as partially revealed spatiotemporal patterns in pixel space—a masked trajectory of an entity in the scene.
为了实现这一愿景，我们引入了掩码视觉动作（Masked Visual Actions），一种将动作直接作为视觉原语嵌入预训练模型原生表示中的方法。我们对预训练的视频模型 [63] 进行微调，使其能够将动作作为像素空间中部分显露的时空模式进行摄入——即场景中实体的掩码轨迹。

When the revealed entity is the robot, the model predicts the scene’s response and acts as a forward dynamics model; when the revealed entity is instead an object or desired object motion, the same model acts as an inverse model to recover robot behavior consistent with that outcome.
当揭示的实体是机器人时，模型预测场景的响应并充当前向动力学模型；当揭示的实体是物体或期望的物体运动时，相同的模型充当逆向模型，以恢复与该结果一致的机器人行为。

In this view, active and passive roles are not properties of separate architectures, but different queries to the same interaction prior. While conceptually simple, this interface is pixel-aligned, embodiment-agnostic, native to video, and efficient to inject into a pretrained model through lightweight adaptation.
在此视角下，主动和被动角色并非不同架构的属性，而是对同一交互先验的不同查询。尽管该接口在概念上很简单，但它与像素对齐、无关具身、原生于视频，并且可以通过轻量级适配高效地注入到预训练模型中。

In addition to superior visual fidelity compared to prior work, we validate our framework across three applications in robot manipulation, both in simulation and in the real world: policy evaluation, model-based planning, and inverse modeling, where the video model is used as part of a robot policy.
与先前的工作相比，除了卓越的视觉保真度外，我们还在机器人操作的三个应用中验证了我们的框架，包括仿真和真实世界：策略评估、基于模型的规划和逆向建模，其中视频模型被用作机器人策略的一部分。

All experiments use a single checkpoint of the model finetuned on as few as 15 hours of robot interaction data. In policy evaluation, the model’s imagined rollouts exhibit consistent correlation with real-world outcomes, so that simulated performance serves as a useful proxy for actual execution.
所有实验均使用在仅 15 小时机器人交互数据上微调的模型的单个检查点。在策略评估中，模型的想象回滚与现实世界结果表现出持续的相关性，因此模拟性能可作为实际执行的有用代理。

In model-based planning, these same predictive capabilities are used to simulate the effects of different action trajectories and select the best one for execution, leading to consistent gains across diverse tasks and policy architectures. The same checkpoint can also be used in reverse: given desired object motion, it synthesizes robot motion that achieves the goal, and a learned inverse dynamics model extracts the resulting actions.
在基于模型的规划中，这些相同的预测能力被用于模拟不同动作轨迹的效果，并选择最佳轨迹执行，从而在各种任务和策略架构中获得一致的收益。同一个检查点也可以反向使用：给定期望的目标运动，它可以合成实现该目标的机器人运动，并且学习到的逆动力学模型提取由此产生的动作。

We summarize our contributions as follows: (1) we introduce Masked Visual Actions, a pixel-space control interface for pretrained video models, together with an efficient adaptation recipe based on masked examples from real and simulated data; (2) we show that forward and inverse robot world-modeling problems can be cast as complementary conditional prediction problems of the same video model, obtained by revealing different entities in the scene; and (3) we validate this framework in both simulation and the real world across three applications in robot manipulation: policy evaluation, model-based planning, and inverse modeling.
我们将我们的贡献总结如下：(1) 我们引入了掩码视觉动作（Masked Visual Actions），一个用于预训练视频模型、在像素空间进行控制的接口，并结合了基于真实和模拟数据中掩码样本的高效适配方法；(2) 我们展示了，通过揭示场景中的不同实体，可以将正向和逆向的机器人世界建模问题转化为同一视频模型的互补条件预测问题；以及 (3) 我们在模拟和真实世界中，通过机器人操作的三个应用场景：策略评估、基于模型的规划和逆向建模，验证了这个框架。

2 相关工作

Controllable video generation as a robotic interface. Video models become simulators once a control signal is added: physical forces [21, 22], warped flow [5], hand poses [23, 32, 70], point or trajectory tracks [13, 20, 56, 58], and goal images [27]. None of these signals is dense, pixelaligned, or shareable across embodiments. Closer to us, visual prompting via inpainting [1] and world modeling as conditional inference [35] show that varying which region of an image is provided enables a generic task parametrization in visual domains; we extend this idea to robotic world modeling by using masked frames as the control interface.
作为机器人接口的可控视频生成。一旦添加了控制信号，视频模型就变成了模拟器：物理力 [21, 22]、扭曲流 [5]、手部姿势 [23, 32, 70]、点或轨迹 [13, 20, 56, 58] 和目标图像 [27]。这些信号都不是密集、像素对齐或可在不同载体之间共享的。更接近我们的是，通过图像修复进行视觉提示 [1] 和将世界建模作为条件推理 [35] 表明，改变提供图像的哪个区域可以实现视觉域中通用任务的参数化；我们通过使用掩码帧作为控制接口，将这一思想扩展到机器人世界建模。

Figure 2: Comparing action representations for learning. Low-dimensional robot actions are compact, but embodiment-specific and not aligned with the image observations used by video models. End-effector poses or robot skeletons are more visual, but remain sparse and require the model to infer geometry, contact, and interaction effects. Our masked visual actions provide dense, image-aligned conditioning, making robot motion and action directly visible, yielding a more learnable representation across embodiments and object interaction.
图 2：比较学习中的动作表征。低维度的机器人动作紧凑，但具有特定于载体的特性，并且与视频模型使用的图像观测不匹配。末端执行器姿态或机器人骨架更具视觉性，但仍然稀疏，并要求模型推断几何、接触和交互效果。我们提出的掩码视觉动作提供了密集、与图像对齐的条件，使得机器人运动和动作直接可见，从而在不同载体和物体交互中产生更易于学习的表征。

Pixel-grounded action conditioning for robot world models. Most robotic video world models communicate actions through embodiment-specific channels: end-effector poses [25, 55, 73, 82], joint vectors [19, 25], or skeletons [60, 66]. A growing line replaces them with pixel-grounded signals. BridgeV2W [10] and Kinema4D [71] render the robot through its URDF and inject the resulting masks or pointmaps via ControlNet; Action Images [79] encodes 7-DoF actions as multi-view Gaussian heatmaps; ORV [74] conditions on 4D occupancy; Mask2IV [40] conditions on predicted mask trajectories; Mask World Model [47] predicts semantic masks as the output.
面向机器人世界模型的像素锚定动作条件.大多数机器人视频世界模型通过特定于载体的通道来传递动作：末端执行器姿态 [25, 55, 73, 82]、关节向量 [19, 25] 或骨架 [60, 66]。一个不断增长的系列用像素锚定信号取代了它们。BridgeV2W [10] 和 Kinema4D [71] 通过 URDF 渲染机器人，并通过 ControlNet 注入生成的掩码或点图；Action Images [79] 将 7-DoF 动作编码为多视图高斯热图；ORV [74] 以 4D 占用作为条件；Mask2IV [40] 以预测的掩码轨迹作为条件；Mask World Model [47] 将语义掩码作为输出进行预测。

A complementary line treats masks as a data-editing tool: Shadow [9, 36], Phantom [38], Masquerade [37], and EmbodiSwap [14] composite or render the robot onto human videos for cross-embodiment policy transfer. All of these works treat the robot as the active entity during training and run forward only.
一条补充性思路将掩码视为一种数据编辑工具：Shadow [9, 36]、Phantom [38]、Masquerade [37] 和 EmbodiSwap [14] 将机器人合成或渲染到人类视频上，以实现跨体态策略迁移。所有这些工作都将机器人视为训练过程中的主动实体，并且仅进行前向传播。

We expose the same masking interface to any subset of entities, so one model serves as a forward, inverse, or unconditional generator without retraining.
我们向任何实体子集公开相同的掩码接口，因此一个模型无需重新训练即可充当前向、逆向或无条件生成器。

Unified video-action models and downstream uses. UVA [42], UWM [81], AIM [18], XWAM [24], and MotuBrain [50] unify forward dynamics, inverse dynamics, policy, and video generation in one model by masking modality channels (action vector vs. video) or by manipulating diffusion timesteps; large platforms such as Cosmos [53], Genie Envisioner [43], and DreamGen [29] package similar capabilities at foundation-model scale.
统一的视频-动作模型及下游应用。UVA [42]、UWM [81]、AIM [18]、XWAM [24] 和 MotuBrain [50] 通过掩码模态通道（动作向量 vs. 视频）或操纵扩散时间步长，在单一模型中统一了前向动力学、逆向动力学、策略和视频生成；Cosmos [53]、Genie Envisioner [43] 和 DreamGen [29] 等大型平台以基础模型规模集成了类似功能。

Because the masking is over modality channels, actions remain low-dimensional vectors and the unification does not transfer across embodiments. Our masking is spatial: active and passive entities live on the same pixel canvas, so the same forward/inverse switch also bridges the embodiment gap.
由于掩码作用于模态通道，动作保持为低维向量，并且这种统一性无法跨越具身迁移。我们的掩码是空间性的：主动实体和被动实体生活在同一个像素画布上，因此相同的正向/反向开关也弥合了具身差距。

The forward direction is exercised for policy evaluation [60, 65, 75], policy improvement [25], planning [7, 16, 17, 28, 61, 78], and direct video-as-policy [26, 33]; the inverse direction extracts robot motion through point tracking [3, 30], object flow [34, 41, 80], predicted object pose [59], or learned IDMs [8, 16, 17, 54, 64, 76].
前向用于策略评估 [60, 65, 75]、策略改进 [25]、规划 [7, 16, 17, 28, 61, 78] 和直接视频即策略 [26, 33]；反向通过点跟踪 [3, 30]、物体流 [34, 41, 80]、预测物体姿态 [59] 或学习到的 IDM [8, 16, 17, 54, 64, 76] 来提取机器人运动。

Critically, our inverse pipeline reuses the same backbone as the forward simulator, while prior work trains a separate IDM head or flow predictor.
关键在于，我们的逆向流程复用了与正向模拟器相同的骨干网络，而先前的工作则训练了单独的IDM头或流预测器。