PointWorld: 动作条件的全场景3D点flow 预测（用同一种3D物理空间模态来表示状态和动作）

Figure 1: Figure 1. PointWorld is a large pre-trained 3D world model that predicts full-scene 3D point flows from a static point cloud and an embodiment-agnostic description of robot actions, represented also as 3D point flows. We curate a
large-scale 3D dynamics modeling dataset, spanning single-arm, bimanual, whole-body, mobile manipulation interactions
in real and simulated domains. Through careful evaluations, we rigorously study the recipe for scaling up 3D world models.
Pretrained on diverse data, a single model enables diverse manipulation behaviors on physical hardware, given only a single RGB-D image captured in the wild, without additional data or finetuning.

图1。PointWorld 是一个大规模预训练 3D 世界模型，它根据静态点云和与具身形态无关的机器人动作描述来预测完整场景的3D点流，而这些动作描述本身也表示为3D点流。我们整理了一个大规模3D动力学建模数据集，涵盖真实与模拟领域中的单臂，双臂，全身，移动操作交互。通过细致的评估，我们系统而严格地研究了扩展3D世界模型的方法。通过在多样化数据上进行预训练，单一模型仅需一张在真实环境中采集的RGB-D图像、无需额外数据或微调，即可在物理硬件上实现多样化的操作行为。

摘要

Humans anticipate, from a glance and a contemplated action of their bodies, how the 3D world will respond, a capability that is equally vital for robotic manipulation. We introduce PointWorld, a large pretrained 3D world model that unifies state and action in a shared 3D space as 3D point flows: given one or few RGB-D images and a sequence of low-level robot action commands, PointWorld forecasts per-pixel displacements in 3D that respond to the given actions. By representing actions as 3D point flows instead of embodiment-specific action spaces (e.g., joint positions), this formulation directly conditions on physical geometries of robots, while seamlessly integrating learning across embodiments. To train our 3D world model, we curate a large-scale dataset spanning real and simulated robotic manipulation in open-world environments, enabled by recent advances in 3D vision and simulated environments, totaling about 2M trajectories and 5oo hours across a single-arm Franka and a bimanual humanoid. Through rigorous, largescale empirical studies of backbones, action representations, learning objectives, partial observability, data mixtures, domain transfers, and scaling, we distill design principles for large-scale 3D world modeling. With a real-time ( 0.1s ) inference speed, PointWorld can be efficiently integrated in the model predictive control (MPC) framework for manipulation. We demonstrate that a single pre-trained checkpoint enables a realworld Franka robot to perform rigid-body pushing, deformable and articulated object manipulation, and tool use, without requiring any demonstrations or post-training and all from a single image captured in-the-wild. Code, dataset, and pre-trained checkpoints will be open-sourced.

人类只需一瞥并设想身体将要执行的动作，便能预判3D世界将如何响应；这种能力对于机器人操作同样至关重要。

我们提出PointWorld，这是一种大规模预训练的3D world model，它将状态与动作统一到共享的3D 空间中，并表示为3D point flows：

• 给定一张或少量RGB-D图像，以及一系列底层机器人动作指令，PointWorld能够预测响应这些给定动作的逐像素3D位移。

通过将动作表示为3D point flows，而非与具体 embodiment 绑定的动作空间（例如关节位置），这种表述能够直接基于机器人的物理几何结构进行条件建模，同时无缝整合跨 embodiment 的学习。

为训练我们的 3D world model，我们构建了一个大规模数据集，涵盖开放世界环境中的真实与模拟机器人操作，这得益于3D vision 和模拟环境的最新进展；该数据集总计约 2M 条轨迹和 500 小时数据，覆盖单臂Franka 和双臂humanoid。

通过对backbones、动作表示、学习目标、部分可观测性、数据混合、领域迁移和扩展进行严格的大规模实证研究，我们提炼出大规模3D世界建模的设计原则。

凭借实时（0.1s）Inference速度，PointWorId可以高效集成到用于操作的模型预测控制（model predictive control / MPC）框架中。

我们证明，单个预训练Checkpoint即可使真实世界中的Franka 机器人完成刚体推动、可变形物体和关节物体操作以及工具使用，无需任何演示或后训练，且全部仅基于一张在自然环境中捕获的图像。

代码、数据集和预训练Checkpoint 将开源。

1. Introduction

1. 引言

World modeling in unstructured environments is imperative for general-purpose robots: predicting how the world evolves from what the robot sees and intends to do with its body. Humans do this from a glance and a grasp, forecasting deformation, articulation, stability, and contact, revealing how much a world-modeling objective captures when conditioned on a contemplated action in 3D (Figure 3). Actions unfold where physics lives, in space and time: our aim is a predictive model that makes such spatially grounded, actionconditioned predictions from only perceptual inputs in openworld settings, a pinnacle goal of spatial intelligence [1].

非结构化环境中的世界建模对于通用机器人至关重要：根据机器人所看到的内容以及它打算如何用身体执行动作，来预测世界将如何演化。

人类只需一瞥与一次抓取就能做到这一点，预测形变、关节运动、稳定性和接触，这揭示了当以3D中设想的动作作为条件时，世界建模目标能够捕捉到多少信息（图3）。

动作是在物理规律存在的空间与时间中展开的：我们的目标是在开放世界场景中，仅从感知输入出发，构建一个能够做出这种具有空间锚定、以动作为条件的预测模型，这是空间智能的一个巅峰目标[1]。

A large body of work has studied world modeling from complementary angles. Physics-based models [2], while capable of highly accurate predictions, face sim-to-real gaps and require curated, environment-specific modeling. Learning-based dynamics models [3] address this by learning from observed interaction, yet often depend on domainspecific inductive bias (e.g., full observability, objectness priors, or material specification). In parallel, large video generative models trained at

scale [4] are capable of producing photorealistic predictions but lack explicit action conditioning and often fall short on physical consistency. See Ai et al. [5] for a recent survey. Despite progress, a gap remains between what current models predict and what humans can foresee from visual observations in the wild and a contemplated action

大量工作从互补的角度研究了世界建模。

• 基于物理的模型[2] 虽然能够进行高度精确的预测，但面临 sim-to-real 差距，并且需要经过精心整理、针对特定环境的建模。
• 基于学习的动力学模型[3]则通过从观察到的交互中学习来应对这一问题，但往往依赖于特定领域的归纳偏置（例如，完全可观测性、objectness priors或材料规格）。
• 与此同时，经过大规模训练的大型视频生成模型[4]能够生成逼真的预测，但缺乏显式的动作条件控制，并且在物理一致性方面往往表现不足。

关于近期综述，参见 Ai et al. [5]。尽管已经取得了进展，但当前模型所能预测的内容与人类基于野外视觉观察和设想中的动作所能预见的内容之间，仍然存在差距。

Our philosophy is unification for scaling: represent state and action in the same modality of 3D physical space. State is represented by a full-scene 3D point cloud built from RGB-D captures; actions are dense 3D point trajectories instantiated from the agent’s own embodiment, typically known a priori (e.g., a robot description file), and thus forecastable over time. Under this representation, 3D world modeling equates to modeling full-scene 3D point flow under perturbations from a temporal sequence of robot points: given partially observed 3D scene points and those action points, predict per-point scene displacements over a horizon. While conceptually simple, this formulation ties raw sensory observation and an embodiment-agnostic action space in a shared representation through dynamics (what moves, how, and where) and implicitly captures objectness, articulation, and material properties, all through interaction between the robot’s specific geometry (e.g., grippers, fingers) and the partially-observed scene. By modeling the geometries of interaction independent of goals, PointWorld aims to capture the single source of truth of the physical world, while naturally learning from heterogeneous embodiments, tasks, and trajectories (regardless of success or failure), akin to “next-token prediction” [6] but for interaction over 3D space and time. We term our approach PointWorld.

我们的理念是通过统一实现扩展：用同一种3D物理空间模态来表示状态和动作。

• 状态由基于RGB-D 采集构建的全场景3D点云表示；
• 动作则表示为源自智能体自身具身形态的稠密3D点轨迹，这些形态通常可先验获知（例如，机器人描述文件），因此能够随时间进行预测。

在这种表示下，3D世界建模等价于：在由机器人点的时间序列施加扰动的条件下，对全场景3D 点流进行建模——给定部分观测到的 3D 场景点以及这些动作点，预测在一个时间范围内每个点的场景位移。

尽管这一表述在概念上很简单，但它通过动力学（什么在运动、如何运动以及运动到哪里）将原始传感观测与一种与具身形态无关的动作空间联系到共享表示中，并且通过机器人特定几何结构（例如grippers、fingers）与部分观测场景之间的交互，隐式捕获对象属性、关节结构以及材料属性。

通过对独立于目标的交互几何进行建模，PointWorld旨在捕捉物理世界的单一真实来源，同时自然地从异构的具身形态、任务和轨迹中学习（无论成功或失败），类似于“next-token prediction”，但面向 3D 时空中的交互。我们将我们的方法称为 PointWorld。

To provide supervision, we curate a large-scale dataset for 3D dynamics modeling, spanning hundreds of in-the-wild scenes with single-arm, bimanual, and whole-body interactions across both real and simulated domains. The dataset was built from existing robotic manipulation datasets, DROID [7] and BEHAVIOR-1K [8]. Since accurate 3D annotations are crucial for capturing precise contact in physical interactions, significant efforts were spent to build a custom pipeline to extract 3D point flows from the real-world dataset, enabled by recent advances in metric depth estimation [9], camera pose estimation [1o], and point tracking [11]. Leveraging the dataset, we distill important design decisions for large-scale 3D dynamics learning through rigorous investigations of backbone architectures, action representations, objectives, partial observability, data mixtures, scaling laws, and domain transfers under zero-shot and finetuned settings.

为了提供监督，我们整理了一个用于3D 动力学建模的大规模数据集，涵盖数百个现实世界中的-涵盖真实与模拟域中的自然场景，包括单臂、双手和全身交互。

该数据集基于现有的机器人操作数据集DROID[7]和BEHAVIOR-1K[8]构建。

由于精确的3D 标注对于捕捉物理交互中的精确接触至关重要，我们投入了大量工作来构建条自定义流水线，从真实世界数据集中提取3D点流；这一过程得益于近年来在度量深度估计[9]、相机位姿估计[10]和点跟踪[11]方面的进展。

基于该数据集，我们通过对骨干架构、动作表示、目标函数、部分可观测性、数据混合、缩放定律以及零样本和微调设置下的域迁移进行严格研究，提炼出大规模3D 动力学学习中的关键设计决策。

To demonstrate PointWorld’s potential for manipulation, we integrate it with a model-predictive controller (MPC) for action inference on a real robot. As PointWorld predicts scene dynamics jointly over short action chunks in a single forward pass at a real-time latency (o.1 s ), it provides a natural and efficient integration with sampling-based MPC (e.g., MPPI [12]). We show that a single pre-trained checkpoint enables a real-world robot to perform rigid-body pushing, deformable and articulated object manipulation, and tool use, without requiring any demonstrations or posttraining and all from a single image captured in-the-wild.

为了展示PointWorld 在操作任务中的潜力，我们将其与模型预测控制器（MPC）集成，用于真实机器人上的动作Inference。

由于PointWorld 能够以实时 Latency（0.1 s）在一次前向传播中对短动作片段上的场景动态进行联合预测，因此它能够与基于采样的MPC（例如MPPI[12]）实现自然且高效的集成。

我们表明，仅需一个预训练的Checkpoint，就能使真实世界中的机器人完成刚体推移、可变形物体与关节物体操作以及工具使用，而无需任何演示或后训练，并且这一切都仅基于一张在自然环境中拍摄的图像。

Contributions. (i) We introduce a large pre-trained 3D world model, PointWorld, that unifies state and action in a shared representation of 3D point flows, and present rigorous studies of its modeling recipe. (ii) We curate and open-source a large-scale high-quality 3D interaction dataset used for training PointWorld, totaling ~ 2M trajectories or ~ 500 hours. (iii) We demonstrate a single pre-trained PointWorld enables a real robot to perform diverse manipulation tasks from a single in-the-wild RGB-D capture, without requiring additional demonstrations or training

贡献。

• (i）我们提出了一个大型预训练 3D world model——PointWorld，它在 3D 点流的共享表示中统一了状态与动作，并对其建模方案进行了严格研究。
• （ii）我们整理并开源了一个用于训练 PointWorld的大规模高质量3D交互数据集，总计包含～2M 条轨迹或～500小时。
• (iii) 我们证明，单个预训练的 PointWorld即可使真实机器人仅凭一次来源于真实环境的RGB-D捕获执行多样化操作任务，而无需额外的示范或训练。

2. Related Work

2. 相关工作

World Modeling. World models [13] are predictive models that simulate future states given current state and action, categorized often by their state-action representations. Video models use pixel-space state, trained either with photometric reconstruction [4, 14-29] or joint-embedding predictions [30, 31]. 3D world models instead operate on meshes or explicit surfaces [32-38], radiance fields or Gaussians [39-48], or particles [3, 49-56]. Hybrid approaches additionally reason over hierarchical structures in world modeling [57-63]. Action parameterizations range from low-level joint-space commands [14, 64-70], to camera and navigation motions [71-78], textual prompts [4, 79-83], and 2D cues [84-94]. Robot actions can then be produced by online planning [3, 12, 64, 95-97], offline policy synthesis [64-67, 98-100], or inverse-dynamics models [101-103].

世界建模。
世界模型「13]是一种预测模型，用于在给定当前状态和动作的情况下模拟未来状态，通常按其状态-动作表示进行分类。

• 视频模型使用像素空间状态，训练方式要么是光度重建[4，14-29]，要么是联合Embedding 预测[30,31]。
• 3D 世界模型则在网格或显式表面[32-38]、辐射场或 Gaussian[39-48]，或粒子[3，49-56] 上运行。
• 混合方法还会在世界建模中进一步对分层结构进行推理[57-63]。
• 动作参数化的范围包括从低层关节空间命令[14，64-70]，到相机与导航运动 [71-78]、文本 Prompt [4， 79-83]，以及 2D 线索 [84-94]。
• 随后，机器人动作可以通过在线规划 [3，12，64， 95-97]、离线策略合成 [64-67，98-100]，或逆动力学模型[101-103] 来生成。

PointWorld uses 3D point flow as shared state-action rep-resentation, emphasizing contact and geometry rather than appearance, conditions on 3D actions with specific geometry of given robot/gripper, interaction beyond only visible regions compared to 2D cues, and doing so with one (or sparse) input images (with estimated depth) in a single, real-time forward pass of a large pre-trained model (Figure 3).
PointWorld 使用 3D 点流作为共享状态-动作表征-表示，强调接触和几何而非外观；

基于给定机器人/夹爪的具体几何对3D 动作进行条件化；与2D 线索相比，可在不仅仅是可见区域之外进行交互；并且通过一次大型pre-trained model的单次实时前向传递，使用一张（或稀疏的）输入图像（带估计深度）实现这一点（图3）。

Figure 2: Figure 2. Overview of PointWorld. Given calibrated RGB-D, robot joint-space actions, and a robot description file (URDF), we convert actions to robot flows and concatenate with scene to form a single point cloud serving as an
embodiment-agnostic interaction geometry. Scene points are featurized with a frozen DINOv3 encoder, robot points with temporal embeddings, and a point cloud backbone predicts full-scene 3D point flows

图2：图 2。PointWorld 概览。给定校准后的 RGB-D、机器人关节空间动作以及机器人描述文件（URDF），我们将动作转换为机器人流，并与场景拼接，形成一个单一的点云，作为与embodiment无关的交互几何。场景点使用冻结的DINOv3 encoder 进行特征化，机器人点使用时间 Embedding，而点云骨干网络预测全场景 3D 点流。

Dynamics Models in Robotics. Dynamics models in robotics instantiate world models with robot action spaces. They include physics-based simulators [2,104-108] and learning-based models [3, 5, 33, 49-54, 69, 109]. Crucial for robotics, they support policy learning [110-112], planning [58, 113-116], model-based RL [64-67, 98-100], exploration and online guidance [117-120], safety filtering [121, 122], model-based design and verification [105, 123, 124], and policy evaluation [46, 125-127]. While existing dynamics models often require curated, scene-specific modeling |5], our aim is to pre-train a single dynamics model that generalizes across diverse in-the-wild environments. Using 3D flows as state-action space, it naturally encapsulates many action parameterizations used in prior works in an embodiment-agnostic manner: joint-space commands [69, 109, 128], end-effector actions [53], and motion primitives [3, 129], while operating on partially observable RGB-D image(s) in the wild without scene reconstruction [35], priors on objectness [53] or materials [13o].

机器人中的动力学模型。
机器人中的动力学模型将世界模型实例化为具有机器人动作空间的模型。

它们包括基于物理的模拟器[2,104-108]和基于学习的模型[3， 5,33，49-54,69，109]。

作为机器人领域的关键组成部分，它们支持策略学习[110-112]、规划[58,113-116]、基于模型的 RL[64-67，98-100]、探索与在线引|导[117-120]、安全过滤[121,122]、基于模型的设计与验证[105，123，124]，以及策略评估 $[46,125-127].$ 。

尽管现有动力学模型通常需要经过精心整理、面向特定场景的建模[5]，我们的目标是预训练一个能够在多样化真实环境中泛化的统一动力学模型。

以3Dflows作为状态-动作空间时，它能够以与具身形式无关的方式，自然涵盖先前工作中使用的多种动作参数化：

• 关节空间命令[69，109，128]、
• 末端执行器动作 [53]
• 以及运动原语[3，129]；

同时可直接处理真实环境中部分可观测的 RGB-D图像，而无需进行场景重建[35]，也不依赖于关于物体性[53]或材料[130]的先验。

2D and 3D Flows for Manipulation. Flows (or point tracks), which address correspondences across space and time, provide a powerful interface between perception and control. With advances in point tracking [11, 131, 132], recent works explored them as structured representations for policy learning [133-140], reward modeling [141-144], (sub-)goal specification [145], or as visual servoing targets [103, 146-150]. In this work, we leverage recent advances in 3D vision (depth [9], camera pose estimation [1o], and point tracking [11]) to label 3D scene flows from largescale real-world manipulation dataset [7] (with robot flows obtained from known robot geometry, kinematics, and proprioception), which enables training of a large 3D world models via stable regression losses to capture robotic interactions with diverse objects in open-world environments.

用于操作的2D 和3DFlows。

Flows（或点轨迹）用于处理跨空间和时间的对应关系，为感知与控制之间提供了强大的接口。

随着点跟踪技术的发展[11，131，132]，近期工作探索了将其作为策略学习的结构化表示[133-140]、奖励建模[141-144]、(子）目标指定[145]，或作为视觉伺服目标[103，146-150]。

在这项工作中，我们利用3D视觉的最新进展（深度[9]、相机位姿估计[10] 和点跟踪[11]），从大规模真实世界操作数据集[7]中标注3D 场景流（其中机器人flow基于已知的机器人几何、运动学和本体感觉获得），从而能够通过稳定的回归损失训练大型3D world models，以捕捉开放世界环境中机器人与多样物体的交互。

3. Method

3. 方法

We formulate 3D world modeling as action-conditioned fullscene 3D point flow prediction (Section 3.1; Figure 2). We then describe how PointWorld may be used for action inference and discuss its use case in the framework of model predictive control that we explore in this work (Section 3.2).

我们将 3D world modeling 表述为动作条件的全场景 3D 点 flow 预测（第 3.1 节；图 2）。

随后，我们介绍PointWorld 如何用于动作Inference，并讨论其在我们于本工作中探索的模型预测控制框架中的使用案例（第 3.2节)。

3.1. 3D World Modeling with PointWorld

3.1. 使用 PointWorld 进行 3D world modeling

We model environment dynamics as a neural network ${\mathcal{F}}_{\theta }:\mathcal{S}\times \mathcal{A}\to \mathcal{S}$ parameterized by $\theta$ that predicts next state given current state and robot action, where $\mathcal{S}$ and $\mathcal{A}$ denote state and action spaces. Existing approaches [5] typically formulate this as a single-step update $s_{t+1}={\mathcal{F}}_{\theta }(s_t,a_t)$. In contrast, we adopt a multi-step (chunked) formulation for data-driven modeling [151]: the model predicts future states over a horizon $H$ in a single forward pass ${\mathcal{F}}_{\theta }^H:(s_t,a_{t:t+H-1})\to s_{t+1:t+H}$, which improves temporal consistency and amortizes computation. We use $H=10$ steps and $0.1,\mathrm{s}$ per step.

我们将环境动力学（environment dynamics）建模为一个由参数 $\theta$ 参数化的神经网络（neural network）

$${\mathcal{F}}_{\theta }:\mathcal{S}\times \mathcal{A}\to \mathcal{S}$$

，该网络根据当前状态（current state）和机器人动作（robot action）预测下一状态，其中 $\mathcal{S}$ 和 $\mathcal{A}$ 分别表示状态空间（state space）和动作空间（action space）。

现有方法 [5] 通常将其表述为单步更新（single-step update）

$$s_{t+1}={\mathcal{F}}_{\theta }(s_t,a_t)$$

与此不同，我们针对数据驱动建模（data-driven modeling）[151] 采用多步分块式表述（multi-step/chunked formulation）：模型通过一次前向传播（single forward pass），预测时间范围（horizon）$H$ 内的未来状态，即

$${\mathcal{F}}_{\theta }^H:(s_t,a_{t:t+H-1})\to s_{t+1:t+H}$$

，这种方式能够提高时间一致性（temporal consistency），并摊销计算开销（amortize computation）。我们使用 $H=10$ 个时间步，每个时间步为 $0.1,\mathrm{s}$。

State Representation.
Building a world model requires a deliberate choice of a state space $\mathcal{S}$, with state at time $t$ denoted by $s_t\in \mathcal{S}$. In this work, we use point flows (referred also as particles [3, 53]) as the environment state. Formally, let $s_t={({\mathbf{p}}_{t,i},{\mathbf{f}}_i^S)}_{i=1}^{N_S}$ denote the point flows at time $t$, consisting of $N_S$ points with positions ${\mathbf{p}}_{t,i}\in {\mathbb{R}}^3$ and time-constant features ${\mathbf{f}}_i^S\in {\mathbb{R}}^{D_S}$ of dimension $D_S$ for each point. Compared to alternative representations, point flows offer the following advantages for world modeling in manipulation: (i) emphasis on physical interactions between 3D geometries instead of appearance, akin to the role of physics simulators rather than renderers; (ii) accessibility from any RGB-D captures in partially observable environments [114] while not assuming objectness or material priors; (iii) simple and stable training via L2 losses on displacements, without permutation matching; (iv) expressiveness to capture diverse fine-grained contact dynamics. To obtain the point flows, from one or a few calibrated RGB-D views, we mask robot pixels via forward kinematics (using the URDF and joint configuration) and back-project the remaining pixels to obtain ${\mathbf{p}}_{t,i}$. Note that since the model takes in a static point set from the environment as input, and correspondence is preserved only within the model’s forward pass (i.e., its “imagination”), no separate point tracker is required for inference, and point count may vary between forward passes.

状态表示（State Representation）。
构建世界模型（world model）需要审慎选择状态空间 $\mathcal{S}$，时刻 $t$ 的状态记为 $s_t\in \mathcal{S}$。

在本工作中，我们使用点流（point flows，也称为粒子 particles [3, 53]）作为环境状态（environment state）。

形式化地，令

$$s_t={({\mathbf{p}}_{t,i},{\mathbf{f}}_i^S)}_{i=1}^{N_S}$$

表示时刻 $t$ 的点流，该点流由 $N_S$ 个点组成，每个点的位置为 ${\mathbf{p}}_{t,i}\in {\mathbb{R}}^3$，并具有维度为 $D_S$ 的时间恒定特征（time-constant features）${\mathbf{f}}_i^S\in {\mathbb{R}}^{D_S}$。

与其他表示方式（alternative representations）相比，点流在用于操作任务（manipulation）的世界建模中具有以下优势：

• （i）它强调三维几何体之间的物理交互（physical interactions between 3D geometries），而不是外观，这更类似于物理模拟器（physics simulators）而不是渲染器（renderers）所发挥的作用；
• （ii）在部分可观测环境（partially observable environments）中，它可以从任意 RGB-D 采集数据（RGB-D captures）[114] 中获得，同时不需要假设物体性（objectness）或材料先验（material priors）；
• （iii）它可通过对位移（displacements）使用 L2 损失（L2 losses）实现简单且稳定的训练，并且不需要排列匹配（permutation matching）；
• （iv）它具有足够的表达能力（expressiveness），能够捕获多样且细粒度的接触动力学（fine-grained contact dynamics）。

为了获得点流，我们从一个或少量经过标定的 RGB-D 视图（calibrated RGB-D views）出发，通过正向运动学（forward kinematics，使用 URDF 和关节配置 joint configuration）遮蔽机器人像素（robot pixels），并将剩余像素反投影（back-project）以获得 ${\mathbf{p}}_{t,i}$。

需要注意的是，由于模型以环境中的静态点集（static point set）作为输入，而对应关系（correspondence）仅在模型的一次前向传播过程中得到保持，也就是仅存在于模型的“想象”（imagination）过程中，因此在推理（inference）时不需要单独的点追踪器（point tracker），并且不同前向传播之间的点数量可以发生变化。

Figure 3: Figure 3. Rich Supervision of 3D World Modeling for Physical Interactions, when conditioned on 3D robot point flows and partial observable RGB-D. The 3D world modeling objective enjoys dense pixel-level supervision while encoding a wide range of capabilities central to robotic manipulation. To predict full-scene evolution, the model needs to implicitly segment objects of interest, identify material property and/or articulation structure, perform implicit shape completion for contact reasoning, propagate robot-object interaction for object-object dynamics, and simultaneously considering the effects of gravity, encapsulated all in a single forward pass of the learned model

图3。在以三维机器人点流（3D robot point flows）和部分可观测的 RGB-D 为条件时，用于物理交互（physical interactions）的三维世界建模（3D world modeling）具有丰富监督（rich supervision）。三维世界建模目标能够获得密集的像素级监督（dense pixel-level supervision），同时编码对机器人操作（robotic manipulation）至关重要的广泛能力。为了预测完整场景的演化，模型需要隐式地分割感兴趣的物体，识别材料属性和/或关节结构，针对接触推理执行隐式形状补全，将机器人—物体交互传播至物体—物体动力学，并同时考虑重力的影响；所有这些能力都被封装在学习所得模型的一次前向传播之中。

Action Representation.
To learn from heterogeneous embodiments (different kinematics, gripper geometries, and even different numbers of grippers), we again use 3D point flows. However, unlike scene point flows which are obtained from RGB-D captures, robot point flows are generated by forecasting the robot’s own geometry via forward kinematics using its URDF (known a priori). This is an intentional design for ensuring “imagined actions” are fully, rather than partially, observable while being represented in an embodiment-agnostic way—crucial in cases where contact occurs in occluded regions (e.g., holding and transporting a large box with egocentric view). Specifically, given a sequence of joint configurations ${q_{t+k}}_{k=0}^H$, we sample robot surface points once at time $t$, attach each to its corresponding link, and propagate them with forward kinematics to obtain an ordered set of $N_R$ robot points ${(r_{t+k,j},f_{t+k,j}^R)}_{j=1}^{N_R}$ at each time step $t+k$, where $r_{t+k,j}\in {\mathbb{R}}^3$ denotes the position of point $j$ at time $t+k$ and $f_{t+k,j}^R\in {\mathbb{R}}^{D_R}$ is its time-varying feature vector of dimension $D_R$. We treat this collection as the action at time $t+k$ and denote it by $a_{t+k}$. This yields an embodiment-agnostic description of interaction geometry over the horizon. In practice, most robot surface points never contact the scene; for efficiency, we sample robot point flows from only the grippers (a few hundred points per gripper depending on its geometry). See Section 5.2 for experiments.

动作表示（Action Representation）。
为了从异构具身形态（heterogeneous embodiments）中学习——这些具身形态可能具有不同的运动学结构（kinematics）、夹爪几何形状（gripper geometries），甚至不同数量的夹爪——我们再次使用三维点流（3D point flows）。

然而，与从 RGB-D 采集数据（RGB-D captures）中获得的场景点流（scene point flows）不同，机器人点流（robot point flows）是通过使用其预先已知的 URDF，并借助正向运动学（forward kinematics）预测机器人自身的几何形状而生成的。

这是一项有意的设计，旨在确保“想象动作”（imagined actions）是完全可观测的，而非部分可观测的，同时以一种与具身形态无关（embodiment-agnostic）的方式进行表示；当接触发生在被遮挡区域时，这一点尤为关键，例如机器人在自我中心视角（egocentric view）下抓持并搬运一个大型箱子。

具体而言，给定一段关节构型序列（joint configurations）${q_{t+k}}_{k=0}^H$，我们在时刻 $t$ 对机器人表面点进行一次采样，将每个点附着到其对应的连杆（link）上，并通过正向运动学传播这些点，从而在每个时间步 $t+k$ 得到一个由 $N_R$ 个机器人点组成的有序集合 ${(r_{t+k,j},f_{t+k,j}^R)}_{j=1}^{N_R}$，其中，$r_{t+k,j}\in {\mathbb{R}}^3$ 表示点 $j$ 在时刻 $t+k$ 的位置，$f_{t+k,j}^R\in {\mathbb{R}}^{D_R}$ 表示其维度为 $D_R$ 的时变特征向量（time-varying feature vector）。

我们将这一点集合视为时刻 $t+k$ 的动作，并将其记为 $a_{t+k}$。由此可在整个预测时域（horizon）内得到一种与具身形态无关的交互几何（interaction geometry）描述。

在实际情况中，大多数机器人表面点从不与场景发生接触；为提高效率，我们仅从夹爪上采样机器人点流，每个夹爪根据其几何形状采样数百个点。相关实验见第 5.2 节。

Dynamics Prediction.
Given the above state-action representations, we now have a static full-scene point cloud $s_t$ and a temporal sequence of robot point-flow actions $a_{t:t+H-1}$ as inputs to the model. Instead of designing custom architectures, we deliberately build on top of state-of-the-art point cloud backbones [152] to distill the core principles that enable scalable, large-scale 3D world modeling. Towards this goal, we concatenate the initial scene points with the time-stacked robot points to form a single point cloud processed by the backbone. Scene points are featurized with frozen DINOv3 [153, 154] by projecting them to 2D views, while robot points are featurized with temporal embedding. The point cloud backbone processes the concatenated point cloud and outputs features for all points. A shared MLP head then predicts per-point displacements of the scene points at each step within a chunk of length $H$ in a single forward pass. This chunked formulation delivers extremely efficient inference capable of evaluating many candidate trajectories with a real-time latency ($0.1,\mathrm{s}$ per batched forward pass), which stands in contrast to pixel-based approaches that typically require seconds-long inference [31, 80] due to the use of diffusion objectives.

动力学预测（Dynamics Prediction）。
给定上述状态—动作表示（state-action representations），模型的输入现在包括

• 一个静态的完整场景点云（static full-scene point cloud）$s_t$，
• 以及机器人点流动作（robot point-flow actions）的时间序列 $a_{t:t+H-1}$。

我们并未设计定制架构，而是有意构建在最先进的点云骨干网络（state-of-the-art point cloud backbones）[152] 之上，以提炼出能够实现可扩展、大规模三维世界建模（scalable, large-scale 3D world modeling）的核心原则。

为实现这一目标，我们将初始场景点与沿时间维堆叠的机器人点（time-stacked robot points）拼接起来，形成一个由骨干网络处理的单一点云。

场景点通过将其投影到二维视图，并使用冻结的 DINOv3（frozen DINOv3）[153, 154] 进行特征化；机器人点则使用时间嵌入（temporal embedding）进行特征化。

点云骨干网络处理拼接后的点云，并输出所有点的特征。

随后，一个共享的多层感知机头（shared MLP head）在一次前向传播（single forward pass）中，预测长度为 $H$ 的一个分块（chunk）内每个时间步上各场景点的逐点位移（per-point displacements）。

这种分块式表述（chunked formulation）能够实现极其高效的推理，可在实时延迟下评估大量候选轨迹（candidate trajectories），即每次批量前向传播（batched forward pass）仅需 $0.1,\mathrm{s}$；相比之下，基于像素的方法（pixel-based approaches）由于采用扩散目标（diffusion objectives），通常需要持续数秒的推理时间 [31, 80]。

Training Objective

While the formulation lends itself to standard regression objectives, 3D world modeling introduces two distinctive challenges that require careful design: (i) due to full-scene prediction, the robot often manipulates only a small subset of the scene, so most points are static and standard L2 loss leads to very sparse training signal; (ii) real-world data is noisy, so we need to regularize the model to be robust to this noise.

To address challenge (i), we adopt a weighted regression objective, reweighting each point at each timestep by a soft movement likelihood $m_{k,i}\in [0,1]$ computed from ground-truth motion so as to focus the loss on moving points. Letting ${\delta }_{k,i}\ge 0$ denote the norm of the ground-truth displacement vector for point $i$ at step $k$, we set

$$m_{k,i}=\sigma \left(\kappa \left({\delta }_{k,i}-\tau \right)\right),$$

where $\sigma$ is the logistic sigmoid, and $\tau$ and $\kappa$ are non-negative displacement-threshold and temperature parameters, respectively.

We then normalize these likelihoods to obtain weights

$$w_{k,i}=\frac{m_{k,i}}{{\sum }_{k,i}{m}_{k,i}}$$

for each point $i$ at step $k$.

To address challenge (ii), we adopt aleatoric uncertainty regularization [10, 155, 156] by predicting a scalar log-variance $s_{k,i}$ for each point $i$ at step $k$ and further using a Huber loss on the residual. Formally, the full training objective becomes:

$$\frac{1}{2}\sum _{k,i}^{H,N_S}{w}_{k,i}\left({\rho }_{\delta }\left({\hat{\mathbf{P}}}_{t+k,i}-{\mathbf{P}}_{t+k,i}\right)e^{-s_{k,i}}+s_{k,i}\right).\text{\hspace{1em}(1)}$$

Here, $w_{k,i}$ is the movement weight, ${\rho }_{\delta }$ is the Huber loss on the 3D residual, $e^{-s_{k,i}}$ is the uncertainty weight, and $s_{k,i}$ is the uncertainty regularization term.

The function ${\rho }_{\delta }$ is the elementwise Huber loss, and ${\hat{\mathbf{P}}}_{t+k,i}$ and ${\mathbf{P}}_{t+k,i}$ are the predicted and ground-truth positions of point $i$ at step $k$, respectively. In practice, we also ignore the points that are deemed not visible by the 2D tracker used to provide the pseudo ground-truth (more details in Section 4).

训练目标（Training Objective）

尽管上述形式本身适用于标准回归目标（standard regression objectives），但三维世界建模（3D world modeling）引入了两个需要谨慎设计的独特挑战：

• （i）由于需要进行完整场景预测（full-scene prediction），机器人通常只操作场景中的一个较小子集，因此大多数点都是静止的，而标准 L2 损失（standard L2 loss）会导致训练信号（training signal）非常稀疏；
• （ii）真实世界数据（real-world data）含有噪声，因此我们需要对模型进行正则化（regularize），使其对这种噪声具有鲁棒性（robustness）。

为了解决挑战（i），我们采用加权回归目标（weighted regression objective），根据由真实运动（ground-truth motion）计算得到的软运动似然（soft movement likelihood）$m_{k,i}\in [0,1]$，对每个时间步中的每个点重新赋权，从而使损失集中于运动点（moving points）。令 ${\delta }_{k,i}\ge 0$ 表示点 $i$ 在第 $k$ 步的真实位移向量（ground-truth displacement vector）的范数，我们设定

$$m_{k,i}=\sigma \left(\kappa \left({\delta }_{k,i}-\tau \right)\right),$$

其中，$\sigma$ 是逻辑 Sigmoid 函数（logistic sigmoid），$\tau$ 和 $\kappa$ 分别是非负的位移阈值参数（displacement-threshold parameter）和温度参数（temperature parameter）。

随后，我们对这些似然值进行归一化，以获得点 $i$ 在第 $k$ 步的权重：

$$w_{k,i}=\frac{m_{k,i}}{{\sum }_{k,i}{m}_{k,i}}.$$

为了解决挑战（ii），我们采用偶然不确定性正则化（aleatoric uncertainty regularization）[10, 155, 156]，为第 $k$ 步的每个点 $i$ 预测一个标量对数方差（scalar log-variance）$s_{k,i}$，并进一步对残差（residual）使用 Huber 损失（Huber loss）。形式化地，完整训练目标（full training objective）为：

$$\frac{1}{2}\sum _{k,i}^{H,N_S}{w}_{k,i}\left({\rho }_{\delta }\left({\hat{\mathbf{P}}}_{t+k,i}-{\mathbf{P}}_{t+k,i}\right)e^{-s_{k,i}}+s_{k,i}\right).\text{\hspace{1em}(1)}$$

其中，$w_{k,i}$ 是运动权重（movement weight），${\rho }_{\delta }$ 是作用于三维残差的 Huber 损失（Huber loss on 3D residual），$e^{-s_{k,i}}$ 是不确定性权重（uncertainty weight），$s_{k,i}$ 是不确定性正则项（uncertainty regularization term）。

${\rho }_{\delta }$ 是逐元素 Huber 损失（elementwise Huber loss），${\hat{\mathbf{P}}}_{t+k,i}$ 和 ${\mathbf{P}}_{t+k,i}$ 分别表示点 $i$ 在第 $k$ 步的预测位置（predicted position）和真实位置（ground-truth position）。在实际处理中，我们还会忽略那些被用于提供伪真实值（pseudo ground-truth）的二维追踪器（2D tracker）判定为不可见的点；更多细节见第 4 节（Section 4）。

Figure 4: Figure 4. Movement Weighting and Uncertainty Regularization, where the robot releases and drops a yellow cloth. (Bottom Left) The movement weighting, used in the training objective, effectively biases the training towards scene points that are moving at each timestep, computed with the ground-truth flows. (Bottom Right) The uncertainty value, predicted by the model without any ground-truth, regularizes training to prevent overfitting to points that have unreliable ground-truth. Intriguingly, we observe that it also emerges to capture action-conditioned uncertainty arising from the object’s physical properties (e.g., larger variability along the edge of the cloth).
图 4：运动加权与不确定性正则化（Movement Weighting and Uncertainty Regularization）。 在该示例中，机器人松开并使一块黄色布料下落。（左下）运动加权（movement weighting）用于训练目标（training objective）中，它利用真实点流（ground-truth flows）进行计算，能够有效地使训练偏向于每个时间步中正在运动的场景点（scene points）。（右下）不确定性值（uncertainty value）由模型在没有任何真实值（ground-truth）的情况下预测，用于对训练进行正则化（regularize training），从而防止模型对那些真实值不可靠的点发生过拟合（overfitting）。有趣的是，我们还观察到，该不确定性也会自发地表征由动作条件（action-conditioned）和物体物理属性（physical properties）所引起的不确定性，例如，布料边缘处呈现出更大的变化性（variability）。

3.2. 用于机器人操作的 PointWorld

A pre-trained PointWorld enables diverse use cases in robotics, as discussed in Section 2. In this work, we specifically investigate whether a single pre-trained PointWorld can enable action inference in unseen, in-the-wild real-world environments from only a single RGB-D capture, without any additional demonstrations or post-training at deployment time. To this end, we integrate PointWorld in an MPC framework with a sampling-based planner MPPI [12] that plans a sequence of ${}_T$ end-effector pose targets in SE(3) given a cost function defined in the model’s state space.

如第2节所述，预训练的 PointWorld可支持机器人领域中的多种应用场景。

在这项工作中，我们具体研究：是否仅凭单次 RGB-D采集、无需任何额外示范或部署时的后训练，单个预训练的 PointWorld 就能在未见过的、真实世界的自然环境中实现动作 Inference。

为此，我们将 PointWorld 集成到一个MPC 框架中，并结合基于采样的规划器MPPI[12]，在给定模型状态空间中定义的代价函数的条件下，规划一段由 ${}_T$ 个末端执行器位姿目标组成的SE(3)序列。

Specifically, given a calibrated RGB-D capture, we first form a scene point set as described in Section $3\cdot 1$ , yielding an initial state S0. We then sample K action perturbations ${\ell }_{1:K}$ using a time-correlated (cubic-spline) noise distribution, which are added to a nominal end-effector trajectory. For each sampled trajectory ${\mathbf{E}}_{1:T}^{(\ell )}$ , the corresponding robot point-flow actions ${\mathbf{a}}_{1:T}^{(\ell )}$ are constructed, scene flows are rolled out by PoINTWORLD conditioned on ${\mathbf{a}}_{1:T}^{(\ell )}$ , and a trajectory cost $J^{(\ell )}$ is accumulated. The nominal trajectory is iteratively refined by computing exponentiated ${\mathrm{weights}}^{{\omega }_{\ell }}$ ∝ exp $\left(-J^{(\ell )}/\beta \right)$ over samples and updating the nominal as a weighted average of sampled trajectories, where $\beta$ is non-negative temperature.

具体而言，给定一个经过标定的 RGB-D采集，我们首先按照第3.1 节所述构建场景点集，得到初始状态 ${\mathbf{s}}_0$ 。随后，我们使用时间相关的（三次样条）噪声分布采样K个动作扰动 ${\ell }_{1:K}$ ，并将其加到一个标称末端执行器轨迹上。对于每条采样轨迹 ${\mathbf{E}}_{1:T}^{(\ell )}$ ，构建相应的机器人点流动作 ${\mathbf{a}}_{1:T}^{(\ell )},\ne {\mathbf{a}}_{1:T}^{(\ell )}$ 条件下由PoINTWORLD对场景流进行展开，并累积轨迹代价 $J^{(\ell )}$ 。通过在样本上计算指数权重 ${\omega }_{\ell }\propto \exp \left(-J^{(\ell )}/\beta \right)$ ，并将标称轨迹更新为采样轨迹的加权平均，对标称轨迹进行迭代优化，其中 $\bar{\beta }$ 是非负 Temperature。

To define the cost function, we separate task objectives from control regularization. Let Ttask⊆{1,...,NS}\mathcal { T } _ { \mathrm { t a s k } } \subseteq \{ 1 , . . . , N _ { S } \}Ttask​⊆{1,...,NS​} denote a set of task-relevant scene points, with associated target positions {gi}i∈Tfask\{ \mathbf { g } _ { i } \} _ { i \in \mathcal { T } _ { \mathrm { f a s k } } }{gi​}i∈Tfask​​ . Task cost on a predicted state ${\mathbf{s}}_k$ at time k is $\begin{array}{r}{c}_{\mathrm{task}}\left({\mathbf{s}}_k\right)=\frac{1}{|{\mathcal{T}}_{\mathrm{task}}|}{\hat{\sum }}_{i\in {\mathcal{T}}_{\mathrm{task}}}{\left|\left|{\mathbf{p}}_{k,i}-{\mathbf{g}}_i\right|\right|}_2^2\end{array}$ Such pointwise goal costs apply broadly across rigid, deformable, and articulated objects. Task-relevant points can be specified by either human via GUI or by VLMs $[145]$ . The overall optimization problem is formulated as a global trajectory optimization:

为定义代价函数，我们将任务目标与控制正则化分开。设 Ttask⊆{1,…,NS}\mathcal { T } _ { \mathrm { t a s k } } \subseteq \{ 1 , \dots , N _ { S } \}Ttask​⊆{1,…,NS​} 表示一组与任务相关的场景点，其对应的目标位置为 {gi}i∈Ttask\{ { \bf g } _ { i } \} _ { i \in { \cal T } _ { \mathrm { t a s k } } }{gi​}i∈Ttask​​ 。在时间 $k$ 上，预测状态 ${\mathbf{s}}_k$ 的任务代价为 $\begin{array}{r}{c}_{\mathrm{task}}\left({\mathbf{s}}_k\right)=\frac{1}{|{\mathcal{T}}_{\mathrm{task}}|}{\sum }_{i\in {\mathcal{T}}_{\mathrm{task}}}{\left|\left|{\mathbf{p}}_{k,i}-{\mathbf{g}}_i\right|\right|}_2^2\end{array}$ 。这类逐点目标代价可广泛应用于刚体、可变形物体以及关节物体。与任务相关的点既可以由人类通过GUI指定，也可以由 VLMs[145] 指定。整体优化问题被表述为一个全局轨迹优化问题：

$$\begin{array}{rl}& \underset{{\mathbf{E}}_{0:T}}{\mathrm{argmin}}{\sum }_{k=1}^T\left[c_{\mathrm{task}}\left({\mathbf{s}}_k\right)+c_{\mathrm{ctrl}}\left({\mathbf{E}}_k\right)\right]\\ & \mathrm{s}.\mathrm{t}.{\mathbf{s}}_{1:T}={\mathcal{F}}_{\mathbf{\theta }}^T\left({\mathbf{s}}_0,{\mathbf{a}}_{1:T}\right),{\mathbf{E}}_0={\mathbf{E}}_{\mathrm{measured}},\end{array}$$

where Cctrl subsumes path length and reachability regularization, ${\mathbf{E}}_k$ denotes end-effector pose at step $k,$ and ${\mathbf{E}}_{\mathrm{measured}}$ is the current end-effector pose. Further details in Appendix.

其中， $c_{\mathrm{ctrl}}$ 包含路径长度和可达性正则化， ${\mathbf{E}}_k$ 表示步骤时的末端执行器位姿， ${\mathbf{E}}_{\mathrm{measured}}$ 是当前的末端执行器位姿。更多细节见附录。

2D Image Reference

FS Depth + Optimized Extrinsics (ours)

Sensor Depth + Dataset Extrinsics V2 (original)

FS Depth + Optimized Extrinsics (ours) FS Depth + VGGT Extrinsics Sensor Depth + Dataset Extrinsics V2 Sensor Depth + Dataset Extrinsics V1

Figure 5: Figure 5. 3D Annotation Quality and Comparisons. FS denotes FoundationStereo [9]; Dataset Extrinsics V1 and
V2 are the two DROID extrinsics releases. (Top) Compared to DROID releases, our pipeline yields substantially higher
quality depth and camera pose calibration, resulting in more accurate robot mask overlays and better aligned point clouds
(readers are encouraged to zoom in or check out the interactive visualization on the project website for details). (Bottom)
We further compute depth reprojection loss (differences between analytical and observed depth of robot surface), and F1 scores of point cloud alignment. We observe purely leveraging existing models (FS, VGGT) are insufficient, and V2
extrinsics improve over V1 by filtering out scenes with poor point cloud alignment but result in significantly lower scene counts. In contrast, our annotation pipeline retains substantially more scenes below o.1o depth-loss criterion and dominates all metrics.

图 5：图 5. 3D 标注质量与比较。FS 表示 FoundationStereo [9]；Dataset Extrinsics V1 和 V2 是 DROID extrinsics的两个发布版本。（上）与DROID发布版本相比，我们的流水线生成了显著更高质量的深度和相机位姿标定，从而得到更准确的机器人mask叠加和对齐更好的点云（建议读者放大查看，或访问项目网站上的交互式可视化了解详情）。(下）我们进一步计算了深度重投影损失（机器人表面解析深度与观测深度之间的差异）以及点云Alignment的F1分数。我们观察到，仅依赖现有模型（FS、VGGT）是不够的，而V2 extrinsics 通过过滤掉点云 Alignment 较差的场景，相比V1有所改进，但会导致场景数量显著减少。相比之下，我们的标注流水线在低于0.10深度损失标准的情况下保留了显著更多的场景，并且在所有指标上均表现最佳。

4. Dataset Curation and Evaluation Protocol

4. 数据集整理与评估协议

Accurate, large-scale 3D data is essential for the world model in Section 3 to generalize in the wild. Apart from requiring action labels, the dataset needs to also have accurate spatial perception (i.e., high-fidelity depth), hand-eye calibration (i.e., camera extrinsics in robot base frame), and per-pixel correspondence matching amid occlusions (i.e., point tracking). While large efforts have been made for collecting diverse real-world manipulation datasets [7, 158], obtaining their 3D annotations has previously been challenging. Our key observation is that recent advances in 3D vision-metric depth estimation, camera pose estimation, and dense point tracking-are maturing to provide a markerless offline pipeline that operates purely on recorded data to produce such a dataset of interest (Figure 5 top-left). Photorealistic simulation complements this with ground-truth supervision. Combining both, we curate a dataset of about 2 M trajectories (5oo hours) spanning single-arm, bimanual, and whole-body teleoperated interactions across in-the-wild real scenes and simulated home-scale environments. To the best of our knowledge, this constitutes the largest 3D

准确、大规模的 3D 数据对于第3 节中的世界模型在真实开放环境中的泛化至关重要。除了需要动作标签之外，数据集还需要具备准确的空间感知（即高保真深度）、手眼标定（即机器人基座坐标系下的相机外参），以及遮挡情况下的逐像素对应匹配（即点跟踪）。尽管在收集多样化的真实世界操作数据集方面已投入了大量努力[7，158]，但此前获取其3D标注一直具有挑战性。我们的一个关键观察是，近年来3D视觉方面的进展——度量深度估计、相机位姿估计和稠密点跟踪已逐渐成熟，能够提供一种无标记的离线流程，仅依赖已记录的数据即可生成这类目标数据集（图5左上）。逼真的照片级仿真通过真值监督对其形成补充。结合二者，我们构建了一个约2M条轨迹（500小时）的数据集，涵盖单臂、双臂以及全身遥操作交互，覆盖真实开放场景和仿真的家居尺度环境。据我们所知，这构成了规模最大的3D

dynamics modeling dataset, which we fully open-source.

动力学建模数据集，我们将其完全开源。

3D Annotation for Real-World Data. We leverage DROID [7], a robot manipulation dataset with diverse in-the-wild interactions recorded by two external cameras and a wrist-mounted camera. Although DROID provides sensor depth and camera extrinsics, the depth often degrades in open-world environments and camera poses are inaccurate due to imperfect calibration. Frontier 3D reconstruction models such as VGGT [1o] jointly estimate depth and camera parameters from RGB images and often look visually plausible, but yield overly smoothed depth maps and camera poses that can deviate from ground-truth by tens of centimeters.

真实世界数据的3D 标注。我们利用DROID[7]，这是一个机器人操作数据集，包含由两个外部摄像头和一个腕部安装摄像头记录的多样化真实环境交互。尽管DROID 提供了传感器深度和相机外参，但深度信息在开放世界环境中往往会退化，而由于校准不完善，相机位姿也并不准确。诸如 VGGT[10] 这样的前沿3D 重建模型能够从 RGB 图像中联合估计深度和相机参数，且结果通常在视觉上看起来合理，但会产生过度平滑的深度图，并且相机位姿可能偏离真实值数十厘米。

After extensive experimentation, we adopt a three-stage annotation pipeline that combines several learned models with a dedicated optimization procedure. First, we replace sensor depth with stereo-estimated depth from FoundationStereo [9] which is particularly effective at the close working distances typical of manipulation. Second, we compute camera extrinsics by refining VGGT-initialized camera poses with an optimization procedure that aligns robot depth observations to the known robot mesh. Third, given accurate depth

经过大量实验，我们采用了一个三阶段标注流水线，将多个学习模型与专门的优化过程相结合。首先，我们使用FoundationStereo[9]估计的双目深度替换传感器深度，这在操作任务中典型的近距离工作场景下尤为有效。其次，我们通过一种优化过程对由VGGT初始化的相机位姿进行细化，使机器人深度观测与已知的机器人网格对齐，从而计算相机外参。第三，在获得准确的深度之后

Figure 6: Figure 6. Unseen rollouts from a single pre-trained PointWorld across diverse domains, visualized with Viser [157]. Given RGB-D captures, PointWorld predicts 10 -step point flows conditioned on robot flows. We show first prediction, last prediction, and last ground-truth. Green points in GT mark regions occluded during 2D point tracking, for which we observe model predictions are often more accurate because these points are not being supervised in model training. Due to grid downsampling (1.5 cm ) we apply to all point clouds, all 3D visualization is upsampled to image

图6：图 6。单个预训练 PointWorld 在不同领域上的未见 rollout，使用 Viser [157] 进行可视化。给定 RGB-D 捕获，PointWorld 在机器人 flow 条件下预测 10 步 point flow。我们展示了第一次预测、最后一次预测以及最后的 ground-truth。GT 中的绿色点标记了在 2D point tracking 过程中被遮挡的区域；我们观察到，模型对这些区域的预测往往更准确，因为这些点在模型训练中未受到监督。由于我们对所有 point cloud 都应用了网格下采样（1.5 cm），为保证视觉清晰度，所有 3D 可视化都通过 nearest neighbors 上采样到图像分辨率。交互式可视化见项目网站。

Figure 7: Figure 7. Roadmap for Scaling 3D World Models, measured by ${\ell }_2$ error on moving scene points on DROID test set. Starting from an existing baseline [5], we progressively modernize the backbone, stabilize training objectives, leverage pre-trained features, and scale model size, yielding consistent gains in accuracy. Hatched bars indicate settings that are not adopted in the final model.

图 7：图 7。3D World Models 扩展路线图，采用 DROID 测试集上运动场景点的 ${\ell }_2$ error 进行衡量。从现有基线 [5]出发，我们逐步对 backbone进行现代化改造，稳定训练目标，利用预训练特征，并扩大模型规模，从而带来持续的精度提升。带斜线阴影的柱表示最终模型中未采用的设置。

and extrinsics, we perform per-pixel point tracking using CoTracker3 [11]. CoTracker3 is a 2D point tracker that outputs image-space correspondences and a visibility map; we lift these tracks to 3D using the refined depth and camera poses and carry over the visibility labels so that occluded points are excluded from supervision during model training. With this pipeline, we recover reliable tracked 3D point flows for over 6o% of DROID (nearly 2oo hours of raw human teleoperation) and obtain reconstructed point clouds that both qualitatively and quantitatively improve over both original dataset and alternative annotation methods (Figure 5). To further assess extrinsics accuracy in the absence of ground truth in the real world, we treat the best 1% of scenes under the original dataset extrinsics (as measured by depth reprojection loss) as a proxy for the true extrinsics. Relative to this reference, our optimized extrinsics achieve a median translation and rotation error of 1.8 cm and 1.9 degrees.

以及外参，我们使用 CoTracker3 [11] 执行逐像素点跟踪。CoTracker3 是一个 2D 点跟踪器，输出图像空间对应关系和可见性图；我们利用优化后的深度和相机位姿将这些轨迹提升到3D，并保留可见性标签，从而在模型训练期间将被遮挡的点排除在监督之外。通过这条流程，我们为超过60%的DROID（接近200小时的原始人类遥操作数据）恢复了可靠的跟踪3D点流，并获得了在定性和定量上均优于原始数据集和替代标注方法的重建点云（图5）。为了在真实世界缺乏真值的情况下进一步评估外参精度，我们将原始数据集外参下表现最好的1%场景（以深度重投影损失衡量）视为真实外参的代理。相对于这一参考，我们优化后的外参实现了1.8 cm的平移误差中位数和1.9 度的旋转误差中位数。

Simulation (BEHAVIOR-1K). To complement realworld data, we use BEHAVIOR-1K [8] (B₁K), which provides about 11oo hours of teleoperated (pre-filtering) interaction in photorealistic home-scale environments with bimanual, whole-body, and mobile manipulation. We obtain ground-truth 3D point flows by leveraging known simulation state. Because the dataset focuses on long-horizon activities

仿真（BEHAVIOR-1K）。为补充真实世界数据，我们使用 BEHAVIOR-1K[8]（B1K），该数据集在照片级真实感的家庭尺度环境中，提供了约1100小时通过遥操作获得的（预过滤前）交互数据，涵盖双手、全身以及移动操作。我们利用已知的仿真状态来获取真实值 3D point flows。由于该数据集侧重于长时程活动

while PointWorld focuses on short-horizon interaction dynamics, we filter trajectories using privileged information accessible in simulation. We retain only trajectories with active contacts between robot and objects and those with nonzero object motion. More details are in Appendix.

而 PointWorld 侧重于短时程交互动力学，因此我们使用仿真中可访问的特权信息对轨迹进行过滤。我们仅保留机器人与物体之间存在主动接触且物体运动非零的轨迹。更多细节见附录。

Model Evaluation Protocol. We evaluate predicted point flow from PointWorld and other baselines using a perpoint, pertimestep l2 distance over the prediction horizon. Because most scene points remain static during robot interaction, we focus on the metric on moving points (l2 mover), as measured by ground-truth data and filter the full set of points using the movement likelihood introduced in Section 3. For real-world domains, we further denoise the evaluation data by training a separate expert model only on the held-out test split to flag unreliable flows via the uncertainty objective from Section 3, retaining only the top 80% of points measured by model confidence. All evaluated models are trained exclusively on the imperfect training set and are evaluated on the expert-filtered test set. Details in Appendix

模型评估协议。我们使用在预测时间范围内按点、按时间步的 ${\ell }_2$ 距离来评估 PointWorld 和其他基线方法预测的点流。由于在机器人交互过程中，大多数场景点保持静止，我们将重点放在运动点（ ${\ell }_2$ mover）的指标上，该指标由ground-truth 数据测量，并使用第3节中引入的运动可能性对完整点集进行筛选。对于真实世界域，我们进一步对评估数据进行去噪：仅在保留的测试划分上训练一个单独的 expert 模型，通过第 3 节中的不确定性目标来标记不可靠的流，并仅保留按模型置信度衡量排名前80%的点。所有参与评估的模型都仅在不完美的训练集上训练，并在经过expert 筛选的测试集上进行评估。详见附录。

Interpretation of the Metric. This dense per-point ${\ell }_2$ metric is highly discriminative when comparing methods and reveals systematic differences in rollout fidelity that tasklevel success rates often fail to expose [125]. Because all errors are measured over one-second horizons, absolute metric differences can appear modest, since even large motions move points by only a few centimeters, yet we empirically observe that small numerical differences often correspond to pronounced qualitative gains in rollout fidelity. Given the scale of the evaluation set (approx. 40,0o0 robot trajectories with 10,000 point flows each), standard errors for the ${\ell }_2$ metrics are negligible $(\le 10^{-5}\mathrm{m})$ , so we report means only.

指标的解释。这个密集的逐点 ${\ell }_2$ 指标在比较不同方法时具有很强的判别力，并能揭示rollout 保真度中的系统性差异，而任务级成功率通常无法暴露这些差异[125]。由于所有误差都是在一秒时间范围内测量的，因此绝对指标差异看起来可能并不大，因为即使是大幅运动，点的位移也只有几厘米；然而，我们在经验上观察到，数值上的微小差异往往对应着rollout 保真度在定性上的显著提升。考虑到评估集的规模（约40,000 条机器人轨迹，每条包含10,000 个点流）， ${\ell }_2$ 指标的标准误差可以忽略不计 $(\le 10^{-5}\mathrm{m})$ ，因此我们只报告均值。

5. Experiments

5. 实验

Focusing on real-world data, we chart a roadmap of empirical lessons we learned for scaling 3D world models (Section 5.1, Figure 7) [159, 16o]. We then discuss targeted ablations along complementary design axes for PointWorld (Section 5.2). Using real and simulated data, we quantify in-domain, cross-domain, and held-out generalization under zero-shot and finetuned settings (Section 5.3). Finally, we study PointWorld for MPC-based action inference on a physical robot in the wild without extra demonstrations or finetuning (Section 5.4). All experiments are constructed to isolate a single modeling choice under controlled setups compared to baselines unless otherwise stated.

聚焦真实世界数据，我们总结了在扩展 3D 世界模型过程中获得的经验性结论路线图（第 5.1 节，图 7）[159，160]。随后，我们沿着PointWorld的互补设计维度讨论了有针对性的消融实验（第5.2节）。结合真实数据与模拟数据，我们在zero-shot 和 finetuned 设置下量化了域内、跨域以及留出集上的泛化能力（第 5.3节）。最后，我们研究了PointWorld 在真实物理机器人上的基于 MPC 的动作 Inference，且无需额外演示或 finetuning（第 5.4 节）。除非另有说明，所有实验均在受控设置下构建，用于隔离单一建模选择并与基线进行比较。

5.1. Scaling 3D World Models: A Roadmap

5.1. 扩展 3D 世界模型：路线图

Modern point cloud backbone (PTv3 [152]) is effective, efficient, and scalable for 3D world modeling. Graphbased neural dynamics (GBND) models are widely used for

现代点云骨干网络（PTv3[152]）在3D 世界建模中效果显著、高效且具有可扩展性。基于图的神经动力学（GBND）模型被广泛用于

Figure 8: Figure 8. Real-World Action Inference. PointWorld runs zero-shot with MPC for rigid, deformable, articulated, and tool-use tasks in the wild. Success rates are on top.

图 8：图 8. 真实世界动作 Inference。PointWorld 在真实环境中结合 MPC 以 zero-shot 方式运行，可处理刚体、可变形体、关节物体以及工具使用任务。顶部显示的是成功率。

Figure 9: Figure 9. Scaling Study. Scaling PointWorld in either data or model size yields roughly log-linear gains in prediction accuracy.

图9：图 9. 扩展研究。在数据规模或模型规模上扩展 PointWorld，都会使预测准确率获得近似对数线性的提升。

Backbone主干网络	Params参数量	Mem.内存	FLOPs	Latency	l2 mov.l2 移动	l2 stat.l2静止
- GBND	1.00X	1.00X	1.00X	13.46	0.0390	0.0066
- PointNet	1.03X	0.34X	0.04X	5.93	0.0369	0.0084
- PointNet++	1.07X	0.67X	0.06x	327.08	0.0368	0.0073
- SparseConv	33.31X	7.18x	1.32X	17.70	0.0396	0.0076
- Transformer	41.06x	0.31X	3.38x	30.43	0.0339	0.0071
- PTv3-50M	49.14x	0.30x	0.34X	59.60	0.0331	0.0067
- PTv3-132M	127.22X	0.69x	1.04X	69.60	0.0324	0.0061
- PTv3-411M	398.67x	1.89X	1.90X	102.47	0.0315	0.0059
- PTv3-1B	957.71x	4.30X	3.57X	123.65	0.0312	0.0056

Table 1: Table 1. Backbone Comparisons. PTv3 [152] enables massive parameter scaling while retaining similar memory and efficient inference (latency in milliseconds, PTv3-1B ≈ 0.12 s ).
表 1：表 1。骨干网络对比。PTv3 [152] 在保持相近内存占用和高效 Inference 的同时，实现了大规模 Parameter 扩展(Latency 以毫秒计，PTv3-1B ≈ 0.12 s)。

dynamics modeling due to their relational inductive bias [5]. Scaling a GBND baseline to our dataset reveals two challenges (Table 1). Memory consumption grows rapidly because maintaining high-dim features for all points in a scene is expensive. Purely local message passing struggles under partial observability, since long-range effects must traverse noisy hops. Motivated by these limitations, we study alternative point cloud architectures, moving from PointNet [161], PointNet++ [162], sparse convolutional nets [163] to transformers [164]. Among these, PointTransformerV3 [152] (PTv3) delivers the strongest modeling power. Its point serialization mirrors GBND’s local grouping, while U-net hierarchy enables attention over progressively coarser point sets for long-range modeling and substantial parameter growth. Table 1 shows that it scales to 957× GBND while keeping modest memory and runtime increases. These results motivate PTv3 as the default backbone.

由于其关系归纳偏置[5]，适合用于动力学建模。将 GBND 基线扩展到我们的数据集后，暴露出两个挑战（表1）。内存消耗增长迅速，因为为场景中的所有点维护高维特征代价高昂。在部分可观测条件下，纯局部消息传递表现不佳，因为长程效应必须通过含噪的逐跳传播。受这些局限性的启发，我们研究了替代性的点云架构，从PointNet[161]、PointNet++[162]、稀疏卷积网络[163]到 Transformer [164]。其中，PointTransformerV3[152]（PTv3）展现出最强的建模能力。其点序列化方式与GBND的局部分组相呼应，而U-net层级结构支持在逐渐更粗粒度的点集上进行注意力计算，以实现长程建模和显著的Parameter增长。表1显示，在仅带来适度内存和运行时间增长的情况下，它可扩展到 957× GBND。这些结果表明，PTv3 是默认骨干网络的最佳选择。

Movement weighting, uncertainty regularization, Huber loss stabilize 3D world model learning on real-world data. Discussed in Section 3, naïve l2 loss is hard to optimize because only a fraction of points move (1 — 5% ). Noisy real-world data exacerbates this. We therefore adopt movement weighting, uncertainty regularization, and a Huber loss on 3D residuals. Movement weighting alone over-

运动加权、不确定性正则化和 Huber loss 能够稳定真实世界数据上的 3D world model 学习。如第 3 节所述，朴素的l2 Ioss 难以优化，因为只有一部分点会移动（1－5%）。真实世界数据中的噪声进一步加剧了这一问题。因此，我们在 3D residuals 上采用运动加权、不确定性正则化和 Huber loss。仅使用运动加权会过-

emphasizes noisy signals, but the uncertainty head and robust loss temper the weights and reduce overfitting. Together.
these changes stabilize training and improve accuracy relative to an unweighted l2 baseline.

度强调噪声信号，但 uncertainty head 和 robust loss 能够缓和这些权重并减少过拟合。综合来看，这些改动稳定了训练，并且相较于未加权的l2 baseline 提升了准确性。

Pre-trained 2D features offer critical priors and substantial gains. High-quality pretrained 3D representations remain scarce despite compelling 3D geometry. Methods such as Sonata [16o] make encouraging progress but often lag behind in fine-grained scenes. Following [9, 10], we hypothesize that dense features from DINOv3 [153] provide objectness priors without explicit segmentation. We therefore project points into calibrated cameras and attach features from multiple layers from a frozen DINOv3. This simple addition substantially boosts accuracy over the baseline.

预训练的 2D 特征提供了关键先验和显著增益。尽管3D 几何极具吸引力，高质量的预训练 3D 表征仍然稀缺。Sonata[160] 等方法取得了令人鼓舞的进展，但在细粒度场景中往往仍然落后。遵循[9，10]，我们假设来自 DINOv3 [153]的稠密特征即使没有显式分割，也能提供objectness先验。因此，我们将点投影到经过标定的相机中，并附加来自冻结的DINOv3 多个层的特征。这一简单的改动相较于 baseline 显著提升了准确性。

Model size scaling is necessary to ingest large-scale world modeling data. With architecture, objective, and features in place, we expand depth and width within the same PTv3 blueprint. Aligned with scaling-law observations in vision and language modeling [165], scaling model size from 50 M to 1 B parameters yields smooth, log-linear gains (Figure 9) similarly for 3D world modeling.

模型规模扩展对于摄取大规模世界建模数据是必要的。在架构、目标和特征就位后，我们在相同的PTv3 蓝图内扩展深度和宽度。与视觉和语言建模中的 scaling-law 观察结果一致[165]，将模型规模从 50 M 扩展到 1 B 个参数，会带来平滑的对数线性收益（图9），这在3D世界建模中同样成立。

Taken together, all these levers-backbone, training objective, pre-trained feature, and model scaling-yield substantial gains over the original GBND baseline [5].

综合来看，所有这些杠杆——backbone、训练目标、预训练特征和模型扩展 相较于原始的 GBND 基线 [5] 都带来了显著提升。

5.2. 消融实验

Representing actions as point flows on grippers balances effective, efficient contact reasoning and enables positive transfer across heterogeneous embodiments. In PointWorld, robot actions are dense point flows over grippers with 300 — 500 points per gripper. We compare against four baselines: (i) whole-body point flows with the same number of points (sparser coverage), (ii) whole-body point clouds with 2ooo points (similar density as ours), (iii) 6-DoF endeffector pose and gripper openness, and (iv) joint positions and gripper openness. The last two low-dim variants omit robot points, which the flowbased models concatenate with

将动作表示为 gripper上的点流，在有效、高效的接触推理之间取得了平衡，并能够在异构 embodiment 之间实现正迁移。在 PointWorld 中，机器人动作是 gripper 上的稠密点流，每个 gripper 有 300 一 500 个点。我们与四个基线进行比较：（i）使用相同点数的全身点流（覆盖更稀疏），（ii）使用2000个点的全身点云（与我们相近的密度），(iii)6-DoF endeffector pose 和 gripper openness，以及(iv)关节位置和 gripper openness。最后两种低维变体省略了机器人点，而基于流的模型将其拼接到

Input Observation

GT and From Scratch

DROID Pre-Trained
BEHAVIOR Pre-Trained

D+B Pre-Trained
Figure 10: Figure 10. Zero-Shot and Finetuned Generalization to Held-Out Real-World Scenes, where the robot transports a reflective glass bottle. PointWorld pre-trained on DROID or jointly on DROID and BEHAVIOR (D+B) are capable of zeroshot generalizing to unseen environment and motion from a held-out DROID lab’s scene, closing the gap to the specialist variant trained on that lab’s data. PoIntWorld pre-trained on only simulation data fail to generalize zero-shot. Further finetuning yields more accurate object trajectories of grasped objects.
图 10：图 10. 对保留的真实世界场景进行 Zero-Shot 和 Finetuned 泛化，其中机器人搬运一个反光玻璃瓶。基于DROID 预训练，或在 DROID 和 BEHAVIOR（D+B）上联合预训练的 PointWorld，能够对来自一个保留的 DROID 实验室场景中的未见环境和运动进行zero-shot 泛化，缩小了与在该实验室数据上训练的专用变体之间的差距。仅基于仿真数据预训练的 PoIntWorld 无法进行 zero-shot 泛化。进一步 finetuning 可得到更准确的被抓取物体轨迹。

		In-Domain域内		Cross-Domain跨域		Held-Out Real保留的真实场景			From从
		D → D	B→B	D → B	B → D	D→ H	B→H	D + B → H	Scratch
Zero-Shot零样本	l2 mover ↓	0.0315	0.0087	0.1460	0.0558	0.0305	0.0531	0.0300	0.0293
	l2 static ↓	0.0059	0.0010	0.0050	0.0058	0.0049	0.0057	0.0063	0.0043
Finetuned微调	l2 mover ↓l2移动体↓			0.0107	0.0378	0.0271	0.0299	0.0272	0.0293
	l2 static ↓l2.静态↓			0.0003	0.0086	0.0040	0.0046	0.0040	0.0043

Table 2: Table 2. Generalization of PointWorld across in-domain, cross-domain, heldout real environments under zero-shot and finetuned settings. D denotes DROID, B denotes BiK, H denotes held-out real-world scenes. “From Scratch” denotes specialist trained on the held-out lab’s data. Evaluations are done on unseen samples from the corresponding dataset. PointWorld generalizes within domains, zero-shot transfers to unseen real-world environments, surpasses specialists if finetuned with 2ox fewer updates, and benefits from real-sim co-training.

表2：表2. PointWorld 在零样本和微调设置下，跨域内、跨域以及保留真实环境上的泛化能力。D表示 DROID，B表示 B1K，H表示保留的真实世界场景。“From Scratch”表示在保留实验室的数据上训练的专用模型。评估在对应数据集的未见样本上进行。PointWorld能够在域内泛化，以零样本方式迁移到未见的真实世界环境；如果使用少20倍更新次数进行微调，其表现可超过专用模型；并且能从真实-模拟联合训练中受益。

Figure 11: Figure 11. Action representations. Representing actions as point flows on grippers balances effective, efficient contact reasoning and enables positive transfer across heterogeneous embodiments.

图11：图11。动作表示。将动作表示为 gripper上的点流，在实现有效、高效的接触推理的同时，还能在异构embodiment之间实现正迁移。

scene features. We train all models jointly on both DROID and BiK data, where DROID uses a single-arm Franka and B₁K uses a bimanual humanoid. Results are in Figure 11. On BiK, representing contact spatially lets point-flow actions outperform low-dim alternatives (end-effector poses and joint positions). Sparse whole-body flows underperform gripperonly flows, likely due to insufficient resolution to capture precise contact. Dense whole-body flows help but still lag behind, as gradients must pass through inactive points and incur compute overhead. On real-world DROID, both whole-body pointflow baselines underperform low-dim baselines. A plausible explanation is that extensive robot points obscure alreadysparse learning signals from noisy real-world data. Gripper-only flows address this issue and attain the best performance, underscoring their effectiveness on real-world data and their ability to obtain positive transfer across heterogeneous embodiments in both domains.

场景特征。我们在 DROID 和 B1K数据上联合训练所有模型，其中 DROID 使用单臂Franka，B1K 使用双臂humanoid。结果见图11。在 B1K上，以空间方式表示接触使 point-flow动作优于低维替代方案（end-effectorpose 和 joint position）。稀疏的 whole-body flow 表现不如仅 gripper flow，可能是因为其分辨率不足，无法捕捉精确接触。密集的 whole-bodyflow有所帮助，但仍然落后，因为梯度必须穿过非活动点并带来计算开销。在真实世界的 DROID上，两种 whole-body point-flow 基线都不如低维基线。一个合理的解释是，大量机器人点掩盖了来自含噪真实世界数据中本已稀疏的学习信号。仅gripperflow解决了这一问题，并取得了最佳性能，凸显了其在真实世界数据上的有效性，以及其在两个域中跨异构embodiment 获得正迁移的能力。

Using chunked prediction in both training and inference reduces rollout drift while improving compute efficiency. PointWorld performs 10-step chunked prediction

在训练和 Inference 中同时使用分块预测，既能减少 rollout drift，又能提高计算效率。PointWorld 执行 10 步分块预测

(equivalent to 1 second). We ablate this design choice against two autoregressive baselines: (i) teacher-forcing (GT input each step) and (ii) self-feeding with 1ok warmup steps, plus sliding-window inference (W = 1, 5) using the same chunked model. Results are shown in Figure 12. Teacher-forcing outperforms self-feeding when training and inference strategies are aligned. Evaluating a chunk-trained model with W = 1 (equivalent to self-feeding) incurs the strongest performance degradation; W = 5 recovers some accuracy but degrades after the trained window. Matching chunked prediction in training and testing over the full horizon minimizes drift while amortizing compute with only a single forward pass (vs. 2-10 for autoregressive), highlighting chunking as both more accurate and more compute-efficient design choice.

(相当于1 秒)。我们将这一设计选择与两个自回归基线进行消融对比：(i) teacher-forcing（每一步使用 GT 输入）和(ii) 具有 10k warmup steps 的 self-feeding，此外还包括使用相同分块模型的 sliding-window Inference($\dot{W}=\dot{1},5)$ 。结果如图12 所示。当训练和 Inference 策略一致时，teacher-forcing 的表现优于 self-feeding。使用W= 1（等价于 self-feeding）来评估一个经过分块训练的模型，会导致最明显的性能下降；W= 5 能恢复一部分精度，但在超出训练窗口后性能会下降。在整个时域范围内让训练和测试都采用匹配的分块预测，能够在仅需一次前向传递的情况下最小化drift并分摊计算成本（相比自回归需要2-10次），这表明分块不仅更准确，也是更高计算效率的设计选择。

PointWorld is robust to different levels of partial observability and benefits from additional cameras in both training and inference. We train four variants that differ only in camera count for RGB-D observations: one, two, three, or a random draw of up to three cameras. We then evaluate all models on three settings with up to three

PointWorld 对不同程度的部分可观测性具有良好的鲁棒性，并且在训练和Inference中都能从额外的摄像头中受益。我们训练了四个变体，它们仅在RGB-D观测的摄像头数量上有所不同：一个、两个、三个，或最多三个摄像头的随机采样。随后，我们在最多三个

cameras. Results are in Figure 13. Error on moving points stays sub-centimeter with negligible standard errors, but using more cameras at train time consistently reduces error at test time. Interestingly, models trained with fixed camera count perform better when more cameras are available at inference. The random-view model is most robust across all test camera counts, suggesting that exposure to varied observability helps the model infer objectness and physical properties under

partial observability at inference time.

摄像头的三种设置下评估所有模型。结果见图13。移动点上的误差保持在亚厘米级，标准误差可以忽略不计，但在训练时使用更多摄像头会稳定地降低测试时的误差。有趣的是，在固定摄像头数量下训练的模型，在Inference时可用摄像头更多的情况下表现更好。随机视角模型在所有测试摄像头数量下都最为鲁棒，这表明在训练中接触多样的可观测性，有助于模型在Inference时于部分可观测条件下推断对象性和物理属性。

Prediction error decreases roughly log-linearly with both model size and data. Inspired by scaling laws from language and vision [165-167], we test whether PointWorld follows similar trends. On DROID, we vary model capacity (50M — 1 B) and data fraction (5% — 100%). Each curve sweeps one axis only. In log space we observe approximately linear behavior for both axes (Figure 9), suggesting predictable gains from extra data and capacity.

预测误差会随着模型规模和数据量的增加而近似按对数线性下降。受语言和视觉领域缩放定律[165-167]的启发，我们测试了 PointWorld 是否遵循类似趋势。在 DROID 上，我们改变模型容量(50M — 1 B）和数据占比(5% — 100%)。每条曲线只扫描一个轴。在对数空间中，我们观察到两个轴都呈现近似线性行为（图9），这表明额外的数据和容量能够带来可预测的收益。

5.3. Generalization and Transfer

5.3. 泛化与迁移

We study PointWorld’s generalization across in-domain, cross-domain, and to held-out real-world environments under zero-shot and finetuned settings. Each finetuning uses 1/20 of the original training iterations. Results are in Table 2.

我们研究了PointWorld 在零样本和微调设置下，跨域内、跨域以及迁移到留出的真实世界环境中的泛化能力。每次微调均使用原始训练迭代次数的1/20。结果见表 2。

PointWorld generalizes within domains. We study in-domain transfer on held-out splits of DROID and BiK that are unseen during training. On BiK the model achieves sub-centimeter mover error on held-out trajectories, while DROID performance on held-out remains similar to training despite real-world variations. This indicates that PointWorld does not simply memorize training samples.

PointWorld 能够在域内实现泛化。我们在训练期间未见过的 DROID 和 B1K 留出划分上研究了域内迁移。在 B1K 上，模型在留出轨迹上达到了亚厘米级的 mover误差；而在 DROID上，尽管存在真实世界变化，模型在留出数据上的表现仍与训练时相近。这表明 PointWorld 并非只是记忆训练样本。

Pre-trained PointWorld can be efficiently finetuned (2ox fewer updates) for both real-to-sim and sim-to-real transfer. We study cross-domain transfer by evaluating how a model pre-trained on DROID generalizes to B₁K, and vice versa. Zero-shot transfer between simulation and real domains remains challenging. Yet, finetuning with only 5% of the original training steps rapidly narrows the gap to domainspecific models trained from scratch using 20× more updates. The effect is symmetric: real-to-sim and sim-to-real transfers both benefit. Empirically, we observe training on real-world data provide better transfer than reverse, plausibly due to the higher scene diversity of the real-world data.

预训练的 PointWorld 可以高效地进行微调（更新次数减少 20 倍），用于 real-to-sim 和 sim-to-real 迁移。我们通过评估在DROID上预训练的模型对 B1K的泛化能力，以及反向评估，来研究跨域迁移。模拟与真实域之间的零样本迁移仍然具有挑战性。然而，仅使用原始训练步数的5%进行微调，就能快速缩小与从头开始训练的领域专用模型之间的差距，而后者需要多进行 20×倍的更新。这种效果是对称的：real-to-sim 和 sim-to-real 迁移都能从中受益。根据经验，我们观察到，在真实世界数据上训练比反向迁移提供了更好的迁移效果，这很可能是由于真实世界数据具有更高的场景多样性。

PointWorld zero-shot generalizes to unseen realworld environments, surpasses specialists if finetuned with 2o x fewer updates, and benefits from real-sim cotraining. To study held-out real generalization, we hold out data from the CLVR lab within DROID and evaluate how well a model pre-trained on the remaining DROID data generalizes to that lab. The heldout set is split into 90% train and 10% test. Zero-shot models never see these frames, while finetuned variants access only the 90% subset. PointWorld pre-trained on the remaining DROID data achieves on-par performance with specialists

PointWorld能够零样本泛化到未见过的现实世界环境中；如果使用少 20 x更新次数进行微调，其表现还能超过专用模型，并且能够从 real-sim 联合训练中受益。为了研究对留出的真实环境的泛化能力，我们从DROID 中留出了CLVRlab 的数据，并评估在其余 DROID 数据上预训练的模型对该实验室的泛化效果。留出集被划分为 90% train 和 10%test。零样本模型从未见过这些帧，而微调变体只能访问 90%子集。在其余 DROID 数据上预训练的 PointWorld 实现了与在该数据上训练的专用模型相当的性能

Figure 12: Figure 12. Ablation on Chunked Prediction, where we study different rollout strategies in training and testing. Chunked rollouts at both training and inference time lead to significantly less drift than other baselines while amortizing compute with only a single forward pass of the model.

图 12：图 12。关于 Chunked Prediction 的消融实验，我们研究了训练和测试中的不同 rollout 策略。在训练和Inference时都采用分块 rollout，相比其他基线能够显著减少漂移，同时仅需一次模型前向传递即可摊销计算成本。

Figure 13: Figure 13. Ablation on Partial Observability, where we train variants of PointWorld with varying number of cameras and evaluate them on all settings at test time. PointWorld is robust to different levels of partial observability and benefits from additional cameras in both training and inference. Training with randomized camera counts yields the best performance across all test settings.
图13：关于部分可观测性的消融实验。我们训练了使用不同数量摄像头的 PointWorld变体，并在测试时于所有设置上对其进行评估。PointWorld 对不同程度的部分可观测性都具有鲁棒性，并且在训练和Inference中都会受益于额外的摄像头。使用随机化摄像头数量进行训练，在所有测试设置上都能获得最佳性能。

held-out lab despite changes in background, lighting, object, and possibly motion distribution. With finetuning, it quickly surpasses the specialist. We observe simulation-pretrained models do not outperform scratch baselines yet reach comparable accuracy with finetuning. Finally, a model pre-trained on combined DROID and BiK mix delivers mildly stronger zero-shot performance than DROID-only.

在保留的实验室环境中，尽管背景、光照、物体以及可能的运动分布发生了变化。经过微调后，它很快就超过了专用模型。我们观察到，经过仿真预训练的模型虽然尚未优于从头训练的基线，但在微调后可达到相当的准确率。最后，在DROID 和 B1K 混合数据上预训练的模型，相比仅使用 DROID 预训练，在零样本性能上略强。

5.4. Model-Based Planning with PointWorld

5.4. 使用 PointWorld 的基于模型的规划

Pre-trained on diverse interactions, we test whether PointWORLD can be zero-shot deployed for manipulation on a physical robot in the wild. For evaluation, we use a Franka setup similar to DROID, mounted on a wheeled base and equipped with one RealSense D435 camera. Depth is estimated using FoundationStereo [9]. For each task, we manually draw an object mask and specify target positions through a GUI tool. Each optimization rolls out 3o steps (3 autoregressive forward passes). With only the pre-trained model and a shared MPC framework, PointWorld optimizes actions for realworld tasks: non-prehensile pushing of rigid

基于多样化交互进行 Pre-trained Model后，我们测试 PointWORLD 是否能够在真实环境中的实体机器人上进行零样本部署，以完成操作任务。为进行评估，我们使用了一个与DROID 类似的Franka配置，安装在带轮底座上，并配备了一台 RealSense D435 相机。深度通过 FoundationStereo [9] 进行估计。对于每个任务，我们通过一个 GUI 工具手动绘制对象掩码并指定目标位置。每次优化执行 30 步 rollout（3 次自回归前向传递）。仅使用 pre-trained model 和一个共享的 MPC 框架，PointWorld 为真实世界任务优化动作：对刚性物体进行非抓取式推动

objects (tissue box, book), deformable manipulation (folding a scarf, placing a pillow), articulated manipulation (opening a microwave and closing a drawer, with revolute and prismatic joints), and tool use (sweeping with a duster or broom). Tasks and success rates are shown in Figure 8, indicating the pre-trained PointWorld captures transferable interaction dynamics. including contact reasoning under partial observability (rigid pushing), implicitly inferring articulation and deformation of objects (articulated and deformable manipulation), and object-object interactions (tool use).

物体（纸巾盒、书本）、可变形操作（折叠围巾、放置枕头）、关节操作（打开微波炉和关闭抽屉，涉及旋转关节和移动关节），以及工具使用（用掸子或扫帚清扫）。任务及成功率如图8所示，表明预训练的PointWorld捕获了可迁移的交互动力学，包括部分可观测条件下的接触推理（刚体推动）、对物体关节结构和形变的隐式推断（关节操作和可变形操作），以及物体与物体之间的交互（工具使用）。

6. Conclusion

6. 结论

We introduced PointWorld, a large pre-trained 3D world model, that predicts 3D environment dynamics given in-thewild RGB-D capture(s) and robot actions under a shared representation of 3D point flows. To train the model, we leveraged recent advances in 3D vision and curated a largescale dataset for action-conditioned 3D world modeling, with high-quality depth maps, camera poses, and 3D tracks. Through empirical evaluations, we rigorously studied the recipe for scaling 3D world model training, including backbone designs, action representations, learning objectives, partial observability, data mixtures, domain transfers, and scaling laws. Pre-trained on diverse data, a single PointWorld model enabled practical manipulation behaviors in the real world, including non-prehensile pushing, deformable and articulated object manipulation, and tool use.

我们推出了 PointWorld，一个大规模预训练 3D 世界模型，它在 3D 点流的共享表示下，根据真实环境中的 RGB-D 捕获数据和机器人动作来预测3D环境动态。为了训练该模型，我们利用了3D视觉领域的最新进展，并整理了一个用于动作条件3D世界建模的大规模数据集，其中包含高质量的深度图、相机位姿和3D轨迹。通过实证评估，我们系统而严格地研究了扩展3D世界模型训练的方法，包括骨干网络设计、动作表示、学习目标、部分可观测性、数据混合、领域迁移和缩放定律。在多样化数据上进行预训练后，单个PointWorld模型即可在现实世界中实现实用的操作行为，包括非抓取式推动、可变形和关节物体操作，以及工具使用。

Acknowledgments

致谢

This work is in part supported by the Stanford Institute for Human-Centered AI (HAI), the Schmidt Futures Senior Fellows grant, ONR MURI No0014-21-1-2801, ONR MURI No0014-22-1-2740. We would like to thank Leslie Kaelbling, Tomás Lozano-Pérez, Yunzhu Li, Jiajun Wu, Ruohan Zhang, Jiayuan Mao, Abhishek Gupta, Pieter Abbeel, Pulkit Agrawal, Shenlong Wang, Anirudha Majumdar, Chuang Gan, David Held, Karen Liu, Jeannette Bohg, Balakumar Sundaralingam, Anqi Li, Bowen Wen, Xiaoyang Wu, Hang Gao, Chelsea Ye, Mijiu Mili, the BEHAVIOR team, members of the Stanford Vision and Learning Lab, and members of the Learning & Intelligent Systems Group for fruitful discussions, feedback, and support.

这项工作部分得到了 Stanford Institute for Human-Centered AI (HAI)、Schmidt Futures Senior Fellows grant.ONR MURI N00014-21-1-2801、ONR MURI N00014-22-1-2740 的支持。我们感谢 Leslie Kaelbling、TomásLozano-Pérez、Yunzhu Li、Jiajun Wu、Ruohan Zhang、Jiayuan Mao、Abhishek Gupta、Pieter Abbeel、PulkitAgrawal、Shenlong Wang、Anirudha Majumdar、Chuang Gan、David Held、Karen Liu、Jeannette Bohg、Balakumar Sundaralingam、Angi Li、Bowen Wen、Xiaoyang Wu、Hang Gao、Chelsea Ye、Mijiu MiliBEHAVIOR 团队、Stanford Vision and Learning Lab 的成员以及 Learning & Intelligent Systems Group 的成员，感谢他们富有成效的讨论、反馈与支持。

参考文献

[1] Li Fei-Fei. From words to worlds: Spatial intelligence is ai’s next frontier. https://drfeifei. substack.com / p / from - words - to - worlds -spatial-intelligence, 2025. Substack. 2

[1] Li Fei-Fei. 从词语到世界：空间智能是 ai 的下—个前沿。https://drfeifei. substack.com / p / from - words - to -worlds -spatial-intelligence, 2025. Substack. 2

[2] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5026-5033, 2012. 2, 3

[2] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco:—种用于基于模型控制的物理引擎。见 2012 IEEE/RSJInternational Conference on Intelligent Robots and Systems (IROS)，第 5026-5033 页，2012。 2, 3

[3] Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint arXiv:1810.01566, 2018. 2, 3

[3] Yunzhu Li、Jiajun Wu、Russ Tedrake、Joshua B Tenenbaum 和 Antonio Torralba。用于操纵刚体、可变形物体和流体的粒子动力学学习。arXiv 预印本 arXiv:1810.01566，2018。2，3

[4] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. 2

[4] Tim Brooks、Bill Peebles、Connor Holmes、Will DePue、Yufei Guo、Li Jing、David Schnurr、Joe Taylor、Troy Luhman、Eric Luhman、Clarence Ng、Ricky Wang 和 Aditya Ramesh。作为世界模拟器的视频生成模型。2024。2

[5] Bo Ai, Stephen Tian, Haochen Shi, Yixuan Wang, Tobias Pfaff, Cheston Tan, Henrik I. Christensen, Hao Su, Jiajun Wu, and Yunzhu Li. A review of learning-based dynamics models for robotic manipulation. Science Robotics, 10(106), 2025. 2, 3,8,9

[5] Bo Ai、Stephen Tian、Haochen Shi、Yixuan Wang、Tobias Pfaff、Cheston Tan、Henrik I. Christensen、HaoSu、Jiajun Wu 和 Yunzhu Li。基于学习的机器人操纵动力学模型综述。Science Robotics，10(106)，2025。2，3，8,9

[6] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 2

[6] Tom B. Brown、Benjamin Mann、Nick Ryder、Melanie Subbiah、Jared D. Kaplan、Prafulla DhariwalArvind Neelakantan、Pranav Shyam、Girish Sastry、Amanda Askell 等。语言模型是少样本学习者。arXiv预印本arXiv:2005.14165，2020。2

[7] Alex Khazatsky, Karl Pertsch, Ashvin Nair, Ajay Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Madhav Srirama, Annie Chen, Benjamin Ellis, Patrick Fagan, Joseph Hejna, Maria Itkina, Lucille Lepert, Henry Ma, Alex Miller, Guanzhi Wu, Suneel Belkhale, Anish Dass, Jeongseok Ha, Ayush Jain, Angelica Lee, Youngwoon Lee, Nicole Memmel, Jeongho Park, Ilya Radosavovic, Rose Wang, Xuchen Zhan, Michael Black, Chi Chi, Landon Hatch, Jinqiang Lin, Zhenjia Lu, Jean Mercat, Ali Rehman, Pratyusha Sanketi, Akshara Sharma, Stacey Simpson, Quan Vuong, Pranav Walke, Blake Wulfe, Chih-Yuan Xiao, Brian Yang, Arman Yavary, Tony Zhao, Cem Agia, Parv Baijal, Alec Castro, Huan Chen, Tao Chen, Jen Jen Chung, Joshua Drake, Matthew Foster, Zhe Gao, Andres Herrera, Jeongwon Heo, Andy Hsu, Siyan Hu, Gabriel Jackson, Brian Le, Canyu Li, Hugo Lin, Donghun Ma, Avinash Maddukuri, Mihir Mirchandani, John Morton, Duc Nguyen, Brian O’Neill, Vincent Scalise, Jacob Seale, Doyeon Son, Yeming Tian, Quang Tran, Henry Wang, Andy Wu, Ho Chit Billy Xie, Juncheng Yang, Xiaolong Yin, Wenlong Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Anchit Gupta, Dinesh Jayaraman, Joseph J. Lim, Jitendra Malik, Roberto Martín-Martín, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Kuan-Ting Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. Droid: A large-scale in-the-wild robot manipulation dataset. In Robotics: Science and Systems XX, 2024. 2, 3,6, 22

[7] Alex Khazatsky、Karl Pertsch、Ashvin Nair、Ajay Balakrishna、Sudeep Dasari、Siddharth Karamcheti、Soroush Nasiriany、Madhav Srirama、Annie Chen、Benjamin Ellis、Patrick Fagan、Joseph Hejna、MariaItkina、Lucille Lepert、Henry Ma、Alex Miller、Guanzhi Wu、Suneel Belkhale、Anish Dass、Jeongseok Ha、Ayush Jain、Angelica Lee、Youngwoon Lee、Nicole Memmel、Jeongho Park、Ilya Radosavovic、RoseWang、Xuchen Zhan、Michael Black、Chi Chi、Landon Hatch、Jinqiang Lin、Zhenjia Lu、Jean Mercat、AliRehman、Pratyusha Sanketi、Akshara Sharma、Stacey Simpson、Quan Vuong、Pranav Walke、Blake Wulfe、Chih-Yuan Xiao、Brian Yang、Arman Yavary、Tony Zhao、Cem Agia、Parv Baijal、Alec Castro、Huan Chen、Tao Chen、Jen Jen Chung、Joshua Drake、Matthew Foster、Zhe Gao、Andres Herrera、Jeongwon Heo、Andy Hsu、Siyan Hu、Gabriel Jackson、Brian Le、Canyu Li、Hugo Lin、Donghun Ma、Avinash Maddukuri、Mihir Mirchandani、John Morton、Duc Nguyen、Brian O’Neill、Vincent Scalise、Jacob Seale、DoyeonSon、Yeming Tian、Quang Tran、Henry Wang、Andy Wu、Ho Chit Billy Xie、Juncheng Yang、Xiaolong Yin、Wenlong Zhang、Osbert Bastani、Glen Berseth、Jeannette Bohg、Ken Goldberg、Abhinav Gupta、AnchitGupta、Dinesh Jayaraman,Joseph J. Lim、Jitendra Malik、Roberto Martín-Martín、SubramanianRamamoorthy、Dorsa Sadigh、Shuran Song、Kuan-Ting Wu、Michael C. Yip、Yuke Zhu、Thomas Kollar、Sergey Levine 和 Chelsea Finn。Droid：一个大规模真实环境机器人操作数据集。载于 Robotics: Science andSystems XX, 2024。 2, 3, 6,22

[8] Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martín, Chen Wang, Gabrael Levine, Wensi Ai, Benjamin Martinez, Hang Yin, Michael Lingelbach, Minjune Hwang, Ayano Hiranaka, Sujay Garlanka, Arman Aydin, Sharon Lee, Jiankai Sun, Mona Anvari, Manasi Sharma, Dhruva Bansal, Samuel Hunter, Kyu-Young Kim, Alan Lou, Caleb R Matthews, Ivan VillaRenteria, Jerry Huayang Tang, Claire Tang, Fei Xia, Yunzhu Li, Silvio Savarese, Hyowon Gweon, C. Karen Liu, Jiajun Wu, and Li Fei-Fei. Behavior-1k: A human-centered, embodied ai benchmark with 1,000 everyday activities and

[8] Chengshu Li、Ruohan Zhang、Josiah Wong、Cem Gokmen、Sanjana Srivastava、Roberto Martín-Martín、Chen Wang、Gabrael Levine、Wensi Ai、Benjamin Martinez、Hang Yin、Michael Lingelbach、Minjune Hwang、Ayano Hiranaka、Sujay Garlanka、Arman Aydin、Sharon Lee、Jiankai Sun、Mona AnvariManasi Sharma、Dhruva Bansal、Samuel Hunter、Kyu-Young Kim、Alan Lou、Caleb R Matthews、IvanVillaRenteria、Jerry Huayang Tang、Claire Tang、Fei Xia、Yunzhu Li、Silvio Savarese、Hyowon Gweon、C.Karen Liu、Jiajun Wu 和 Li Fei-Fei。Behavior-1k：一个以人为中心的具身 ai 基准，包含 1,000 个日常活动和

realistic simulation. arXiv preprint arXiv:2403.09227, 2024. 2, 8, 24

逼真的仿真。arXiv 预印本 arXiv:2403.09227，2024。2，8， 24

[9] Bowen Wen, Matthew Trepte, Joseph Aribido, Jan Kautz, Orazio Gallo, and Stan Birchfield. Foundationstereo: Zeroshot stereo matching. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 5249-5260, 2025. 2, 3, 6, 9, 11, 22, 30

[9] Bowen Wen、Matthew Trepte、Joseph Aribido、Jan Kautz、Orazio Gallo 和 Stan Birchfield。Foundationstereo:零样本立体匹配。载于 Proceedings of the Computer Vision and Pattern RecognitionConference，页 5249-5260， 2025。 2, 3, 6, 9, 11, 22, 30

[1o] Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. 2, 3, 5, 6, 9, 22

[10] Jianyuan Wang、Minghao Chen、Nikita Karaev、Andrea Vedaldi、Christian Rupprecht 和 DavidNovotny。Vggt:视觉几何基础的 Transformer。载于 Proceedings of the IEEE/CVF Conference on ComputerVision and Pattern Recognition, 2025。 2, 3, 5, 6, 9, 22

[11] Nikita Karaev, Yuri Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker3: Simpler and better point tracking by pseudolabelling real videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6013-6022, 2025. 2, 3, 8, 23, 26

[11] Nikita Karaev、Yuri Makarov、Jianyuan Wang、Natalia Neverova、Andrea Vedaldi 和 ChristianRupprecht。Cotracker3：通过对真实视频进行伪标签标注，实现更简单且更好的点跟踪。载于 Proceedings of theIEEE/CVF International Conference on Computer Vision，页 6013-6022，2025。 2, 3, 8, 23, 26

[12] Grady Williams, Andrew Aldrich, and Evangelos A. Theodorou. Model predictive path integral control: From theory to

parallel computation. Journal of Guidance, Control, and Dynamics, 40(2):344-357, 2017. 2, 5, 21

[12] Grady Williams、Andrew Aldrich 和 Evangelos A. Theodorou。模型预测路径积分控制：从理论到并行计算。Journal of Guidance, Control, and Dynamics, 40(2):344-357, 2017。 2, 5, 21

[13] David Ha and J"urgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018. 2

[13] David Ha 和 J"urgen Schmidhuber。世界模型。arXiv 预印本 arXiv:1803.10122，2018。2

[14] Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. Advances in neural information processing systems, 29, 2016. 2

[14] Chelsea Finn、Ian Goodfellow 和 Sergey Levine。通过视频预测进行物理交互的无监督学习。Advances inneural information processing systems, 29, 2016。 2

[15] Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804.01523, 2018.

[15] Alex X. Lee、Richard Zhang、Frederik Ebert、Pieter Abbeel、Chelsea Finn 和 Sergey Levine。随机对抗视频预测。arXiv 预印本 arXiv:1804.01523，2018。

[16] Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L. Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. In Advances in Neural Information Processing Systems, 2015.

[16] Junhyuk Oh、Xiaoxiao Guo、Honglak Lee、Richard L. Lewis 和 Satinder Singh. 在 Atari 游戏中使用深度网络进行动作条件视频预测. 收录于 Advances in Neural Information Processing Systems, 2015.

[17] William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. arXiv preprint arXiv:1605.08104, 2016.

[17] William Lotter、Gabriel Kreiman 和 David Cox. 用于视频预测和无监督学习的深度预测编码网络. arXiv 预印本arXiv:1605.08104, 2016.

[18] Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In Proceedings of the 35th International Conference on Machine Learning, 2018.

[18] Emily Denton 和 Rob Fergus. 使用学习到的先验进行随机视频生成. 收录于 Proceedings of the 35thInternational Conference on Machine Learning, 2018.

[19] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157, 2021.

[19] Wilson Yan、Yunzhi Zhang、Pieter Abbeel 和 Aravind Srinivas. VideoGPT :使用 vq-vae 和 Transformer 进行视频生成. arXiv 预印本 arXiv:2104.10157， 2021.

[2o] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458, 2022.

[20] Jonathan Ho、Tim Salimans、Alexey Gritsenko、William Chan、Mohammad Norouzi 和 David J. Fleet。视频扩散模型。arXiv 预印本 arXiv:2204.03458，2022 年。

[21] Uriel Singer, Adam Polyak, Timothy Hayes, Sagie Benaim, Oron Gafni, Oron Ashual, Yuval Atzmon, Tomer Shalev, Devi Parikh, Yaniv Taigman, Ronen Banner, and Eliya Nachmani. Make-a-video: Text-to-video generation without textvideo data. arXiv preprint arXiv:2209.14792, 2022.

[21] Uriel Singer、Adam Polyak、Timothy Hayes、Sagie Benaim、Oron Gafni、Oron Ashual、Yuval Atzmon、Tomer Shalev、Devi Parikh、Yaniv Taigman、Ronen Banner 和 Eliya Nachmani。Make-a-video:无需 text-video 数据的文本到视频生成。arXiv 预印本 arXiv:2209.14792，2022 年。

[22] Jonathan Ho, William Chan, Carl Doersch, Suman Ravuri, David J. Fleet, Mohammad Norouzi, and Tim Salimans,

Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022.

[22] Jonathan Ho、William Chan、Carl Doersch、Suman Ravuri、David J. Fleet、Mohammad Norouzi 和 TimSalimans。Imagen video：使用扩散模型生成高清视频。arXiv 预印本 arXiv:2210.02303，2022 年。

[23] Yuwei Guo, Zhiyu Chen, Chenyang Lei, Chengyue Gao, Fang Lu, and Yi Zhang. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023.

[23] Yuwei Guo、Zhiyu Chen、Chenyang Lei、Chengyue Gao、Fang Lu 和 Yi Zhang。Animatediff :无需特定微调即可为你个性化的文本到图像扩散模型制作动画。arXiv 预印本 arXiv:2307.04725，2023 年。

[24] Yichao Yin, Lei Qi, Hong Zhu, Jiakai Qin, Salman Khan, Luc Van Gool, Li Liu, and Philip Torr. Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv preprint arXiv:2310.12190, 2023.

[24] Yichao Yin、Lei Qi、Hong Zhu、Jiakai Qin、Salman Khan、Luc Van Gool、Li Liu 和 Philip Torr。Dynamicrafter：利用视频扩散先验为开放域图像制作动画。arXiv 预印本 arXiv:2310.12190，2023。

[25] Jiajun Chen, Yu Li, Zhaohui Fan, Fan Zhang, Yu Qiao, and Bo Dai. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprint arXiv:2303.19712, 2023. [26] Danila Kondratyuk, Uriel Singer, Willi Menapace, Reza Mahjourian, Xiaohua Zhai Sun, et al., and Agrim Gupta. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125, 2023.

[26] Danila Kondratyuk、Uriel Singer、Willi Menapace、Reza Mahjourian、Xiaohua Zhai Sun等，以及 AgrimGupta。Videopoet：—个用于零样本视频生成的 Large Language Model。arXiv 预印本 arXiv:2312.14125，2023。

[27] Andreas Blattmann, Willi Menapace, Yiyi He, Arash Vahdat, Robin Rombach, et al., and Agrim Gupta. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662, 2023.

[27] Andreas Blattmann、Willi Menapace、Yiyi He、Arash Vahdat、Robin Rombach等，以及 Agrim Gupta。使用扩散模型进行照片级真实感视频生成。arXiv 预印本 arXiv:2312.06662，2023。

[28] Haoyu Zhen, Qiao Sun, Hongxin Zhang, Junyan Li, Siyuan Zhou, Yilun Du, and Chuang Gan. Tesseract: learning 4d embodied world models. arXiv preprint arXiv:2504.20995, 2025. [29] Klemen Kotar, Wanhee Lee, Rahul Venkatesh, Honglin Chen, Daniel Bear, Jared Watrous, Simon Kim, Khai Loong Aw, Lilian Naing Chen, Stefan Stojanov, et al. World modeling with probabilistic structure integration. arXiv preprint arXiv:2509.09737, 2025. 2

[29] Klemen Kotar、Wanhee Lee、Rahul Venkatesh、Honglin Chen、Daniel Bear、Jared Watrous、SimonKim、Khai Loong Aw、Lilian Naing Chen、Stefan Stojanov 等。通过概率结构集成进行世界建模。arXiv 预印本arXiv:2509.09737，2025。2

[3o] Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. arXiv preprint arXiv:2404.08471, 2024. 2

[30] Adrien Bardes、Quentin Garrido、Jean Ponce、Xinlei Chen、Michael Rabbat、Yann LeCun、MahmoudAssran 和 Nicolas Ballas. 重新审视用于从视频中学习视觉表征的特征预测。arXiv preprint arXiv:2404.08471，2024。2

[31] Mahmoud Assran, Adrien Bardes, Xinlei Chen, Yann LeCun, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985, 2025. 2, 4

[31] Mahmoud Assran、Adrien Bardes、Xinlei Chen、Yann LeCun 等. V-jepa 2：自监督视频模型实现理解、预测和规划。arXiv preprint arXiv:2506.09985，2025。2，4

[32] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689, 2021. 2

[32] Peng Wang、Lingjie Liu、Yuan Liu、Christian Theobalt、Taku Komura 和 Wenping Wang. NeuS:通过用于多视图重建的体渲染学习神经隐式表面。arXiv preprint arXiv:2106.10689，2021。2

[33] Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W. Battaglia. Learning mesh-based simulation with graph networks. arXiv preprint arXiv:2010.03409, 2020. 3

[33] Tobias Pfaff、Meire Fortunato、Alvaro Sanchez-Gonzalez 和 Peter W. Battaglia. 使用图网络学习基于网格的仿真。arXiv preprint arXiv:2010.03409，2020。3

[34] Alberta Longhini et al. Edo-net: Learning elastic properties of deformable objects from graph dynamics. arXiv preprint arXiv:2209.08996, 2022.

[34] Alberta Longhini 等人。Edo-net：从图动力学中学习可变形物体的弹性属性。arXiv 预印本 arXiv:2209.08996，2022。

[35] Hanxiao Jiang, Hao-Yu Hsu, Kaifeng Zhang, Hsin-Ni Yu, Shenlong Wang, and Yunzhu Li. Phystwin: Physicsinformed reconstruction and simulation of deformable objects from videos. arXiv preprint arXiv:2503.17973, 2025. 3, 21

[35] Hanxiao Jiang、Hao-Yu Hsu、Kaifeng Zhang、Hsin-Ni Yu、Shenlong Wang 和 Yunzhu Li。 Phystwin :基于物理信息的从视频中重建与模拟可变形物体。arXiv 预印本 arXiv:2503.17973，2025。3，21

[36] Hongchi Xia, Entong Su, Marius Memmel, Arhan Jain, Raymond Yu, Numfor Mbiziwo-Tiapo, Ali Farhadi, Abhishek Gupta, Shenlong Wang, and Wei-Chiu Ma. Drawer: Digital

[36] Hongchi Xia、Entong Su、Marius Memmel、Arhan Jain、Raymond Yu、Numfor Mbiziwo-Tiapo、AliFarhadi、Abhishek Gupta、Shenlong Wang 和 Wei-Chiu Ma。Drawer:数字化

reconstruction and articulation with environment realism. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 21771-21782, 2025

重建与具有环境真实性的关节化。在 Proceedings of the Computer Vision and Pattern Recognition Conference中，第21771-21782页，2025。

[37] Hongchi Xia, Zhi-Hao Lin, Wei-Chiu Ma, and Shenlong Wang. Video2game: Real-time interactive realistic and browser-compatible environment from a single video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578-4588, 2024.

[37] Hongchi Xia、Zhi-Hao Lin、Wei-Chiu Ma 和 Shenlong Wang。Video2game：从单个视频生成实时交互、逼真且兼容浏览器的环境。见 Proceedings of the IEEE/CVF Conference on Computer Vision and PatternRecognition，页 4578-4588，2024。

[38] Haitao Qiu, Xiaoyu Zhou, Liyang Zhang, Jiaqi Yang, Kailun Zhou, et al. Physgen3d: Crafting a miniature interactive world from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.2

[38] Haitao Qiu、Xiaoyu Zhou、Liyang Zhang、Jiaqi Yang、Kailun Zhou 等。Physgen3d:从单张图像构建—个 微型交互世界。见 Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025。2

[39] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. arXiv preprint arXiv:2003.08934, 2020. 2, 21

[39] Ben Mildenhall、Pratul P. Srinivasan、Matthew Tancik、Jonathan T. Barron、Ravi Ramamoorthi 和 RenNg。Nerf：将场景表示为用于视图合成的神经辐射场。arXiv 预印本 arXiv:2003.08934，2020。2，21

[4o] Bernhard Kerbl, Georgios Kopanas, Thomas LeGendre, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023. 21

[40] Bernhard Kerbl、 Georgios Kopanas、Thomas LeGendre 和 George Drettakis。 3d gaussian splatting :用于实时辐射场渲染的 3D 高斯泼溅。ACM Transactions on Graphics，42(4)，2023。21

[41] Danny Driess, Zhiao Huang, Yunzhu Li, Russ Tedrake, and Marc Toussaint. Learning multi-object dynamics with compositional neural radiance fields. In Conference on robot learning, pages 1755-1768. PMLR, 2023.

[41] Danny Driess、Zhiao Huang、Yunzhu Li、Russ Tedrake 和 Marc Toussaint. 使用组合式 neural radiancefields 学习多物体动力学. In Conference on robot learning, pages 1755-1768. PMLR, 2023.

[42] Jad Abou-Chakra et al. Physically embodied gaussian splatting: A realtime correctable world model for robotics. arXiv preprint arXiv:2406.10788, 2024.

[42] Jad Abou-Chakra et al. 具身化物理 gaussian splatting:一种用于机器人学的实时可校正世界模型. arXivpreprint arXiv:2406.10788, 2024.

[43] Jad Abou-Chakra et al. Real-is-sim: Bridging the sim-to-real gap with a dynamic digital twin. arXiv preprint arXiv:2504.03597, 2025.

[43] Jad Abou-Chakra et al. Real-is-sim:通过动态数字孪生弥合 sim-to-real 差距. arXiv preprint arXiv:2504.03597, 2025.

[44] Tianyuan Zhang, Hong-Xing Yu, Rundi Wu, Brandon Y. Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, and William T. Freeman. Physdreamer: Physics-based interaction with 3d objects via video generation. arXiv preprint arXiv:2404.13026, 2024.

[44] Tianyuan Zhang、Hong-Xing Yu、Rundi Wu、Brandon Y. Feng、Changxi Zheng、Noah Snavely、JiajunWu 和 William T. Freeman. Physdreamer：通过视频生成实现基于物理的 3d 物体交互. arXiv preprintarXiv:2404.13026, 2024.

[45] Guanxing Lu, Baoxiong Jia, Puhao Li, et al. Gwm: Towards scalable gaussian world models for robotic manipulation. arXiv preprint arXiv:2508.17600, 2025

[45] Guanxing Lu、Baoxiong Jia、Puhao Li 等人。Gwm：迈向可扩展的用于机器人操作的高斯世界模型。arXiv 预印本 arXiv:2508.17600，2025。

[46] Kaifeng Zhang, Shuo Sha, Hanxiao Jiang, Matthew Loper, Hyunjong Song, Guangyan Cai, Zhuo Xu, Xiaochen Hu, Changxi Zheng, and Yunzhu Li. Real-to-sim robot policy evaluation with gaussian splatting simulation of soft-body interactions. arXiv preprint arXiv:2511.04665, 2025. 3

[46] Kaifeng Zhang、Shuo Sha、Hanxiao Jiang、Matthew Loper、Hyunjong Song、Guangyan Cai、Zhuo Xu、Xiaochen Hu、Changxi Zheng 和 Yunzhu Li。基于软体交互高斯 splatting 仿真的 real-to-sim 机器人策略评估。arXiv 预印本 arXiv:2511.04665，2025。3

[47] Guangyuan Xiang, Juntong Wang, Linhan Qiu, Jing Yang, Zhihao Liu, Jiahui Zhang, Zhen Li, et al. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.

[47] Guangyuan Xiang、Juntong Wang、Linhan Qiu、Jing Yang、Zhihao Liu、Jiahui Zhang、Zhen Li 等人。Physgaussian：用于生成动力学的物理集成 3D 高斯。发表于 IEEE/CVF Conference on Computer Vision andPattern Recognition 论文集，2024。

[48] Zizhang Li, Hong-Xing Yu, Wei Liu, Yin Yang, Charles Herrmann, Gordon Wetzstein, and Jiajun Wu. Wonderplay: Dynamic 3d scene generation from a single image and actions. arXiv preprint arXiv:2505.18151, 2025. 2

[48] Zizhang Li、Hong-Xing Yu、Wei Liu、Yin Yang、Charles Herrmann、Gordon Wetzstein 和 Jiajun Wu。Wonderplay：从单张图像和动作生成动态 3D 场景。arXiv 预印本 arXiv:2505.18151，2025。2

[49] Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende, and Koray Kavukcuoglu. Interaction networks for learning about objects, relations and physics. arXiv preprint arXiv:1612.00222, 2016. 2, 3

[49] Peter W. Battaglia、Razvan Pascanu、Matthew Lai、Danilo Rezende 和 Koray Kavukcuoglu.用于学习对象、关系和物理的交互网络。arXiv 预印本 arXiv:1612.00222，2016。2，3

[50] Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter W. Battaglia. Learning to simulate complex physics with graph networks. arXiv preprint arXiv:2002.09405, 2020.

[50] Alvaro Sanchez-Gonzalez、Jonathan Godwin、Tobias Pfaff、Rex Ying、Jure Leskovec 和 Peter W.Battaglia. 使用图网络学习模拟复杂物理。arXiv 预印本 arXiv:2002.09405，2020。

[51] Jad Abou-Chakra et al. Particlenerf: A particle-based encoding for online neural radiance fields. arXiv preprint arXiv:2211.04041, 2022.

[51] Jad Abou-Chakra 等. Particlenerf：一种用于在线神经辐射场的基于粒子的编码。arXiv 预印本arXiv:2211.04041，2022。

[52] Suning Huang, Qianzhong Chen, Xiaohan Zhang, Jiankai Sun, and Mac Schwager. Particleformer: A 3d point cloud world model for multi-object, multi-material robotic manipulation. arXiv preprint arXiv:2506.23126, 2025.

[52] Suning Huang、Qianzhong Chen、Xiaohan Zhang、Jiankai Sun 和 Mac Schwager. Particleformer:—种用于多对象、多材料机器人操作的 3d 点云世界模型。arXiv 预印本 arXiv:2506.23126，2025。

[53] Kaifeng Zhang, Baoyu Li, Kris Hauser, and Yunzhu Li. Particle-grid neural dynamics for learning deformable object models from rgb-d videos. arXiv preprint arXiv:2506.15680, 2025.3

[53] Kaifeng Zhang、Baoyu Li、Kris Hauser 和 Yunzhu Li. 用于从 rgb-d 视频中学习可变形物体模型的粒子-网格神经动力学. arXiv 预印本 arXiv:2506.15680, 2025.3

[54] Kaifeng Zhang, Baoyu Li, Kris Hauser, and Yunzhu Li. Adaptigraph: Material-adaptive graph-based neural dynamics for robotic manipulation. arXiv preprint arXiv:2407.07889, 2024. 3

[54] Kaifeng Zhang、Baoyu Li、Kris Hauser 和 Yunzhu Li. Adaptigraph：用于机器人操作的材料自适应基于图的神经动力学. arXiv 预印本 arXiv:2407.07889, 2024.3

[55] Zihao He, Bo Ai, Tongzhou Mu, Yulin Liu, Weikang Wan, Jiawei Fu, Yilun Du, Henrik I Christensen, and Hao Su. Scaling cross-embodiment world models for dexterous manipulation. arXiv preprint arXiv:2511.01177, 2025

[55] Zihao He、Bo Ai、Tongzhou Mu、Yulin Liu、Weikang Wan、Jiawei Fu、Yilun Du、Henrik I Christensen 和Hao Su. 用于灵巧操作的跨 embodiment 世界模型扩展. arXiv 预印本 arXiv:2511.01177， 2025.

[56] William F Whitney, Jacob Varley, Deepali Jain, Krzysztof Choromanski, Sumeet Singh, and Vikas Sindhwani. Modeling the real world with high-density visual particle dynamics. arXiv preprint arXiv:2406.19800, 2024. 2

[56] William F Whitney、Jacob Varley、Deepali Jain、Krzysztof Choromanski、Sumeet Singh 和 VikasSindhwani. 用高密度视觉粒子动力学对真实世界进行建模. arXiv 预印本 arXiv:2406.19800, 2024.2

[57] Constructions Aeronautiques, Adele Howe, Craig Knoblock, ISI Drew McDermott, Ashwin Ram, Manuela Veloso, Daniel Weld, David Wilkins Sri, Anthony Barrett, Dave Christianson, et al. Pddl-the planning domain definition language. Technical Report, Tech. Rep., 1998. 2

[57] Constructions Aeronautiques、Adele Howe、Craig Knoblock、ISI Drew McDermott、Ashwin RamManuela Veloso、Daniel Weld、David Wilkins Sri、Anthony Barrett、Dave Christianson 等。Pddl——规划领域定义语言。技术报告，Tech. Rep.，1998。2

[58] Leslie Pack Kaelbling and Tomás Lozano-Pérez. Hierarchical task and motion planning in the now. In 2011 IEEE international conference on robotics and automation, pages 1470-1477. IEEE, 2011. 3

[58] Leslie Pack Kaelbling 和 Tomás Lozano-Pérez. 当下的分层任务与运动规划。见 2011 IEEE internationalconference on robotics and automation，页 1470-1477。 IEEE，2011。 3

[59] Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv:1904.12584, 2019.

[59] Jiayuan Mao、Chuang Gan、Pushmeet Kohli、Joshua B Tenenbaum 和 Jiajun Wu。神经符号概念学习器:从自然监督中解释场景、词语和句子。arXiv 预印本 arXiv:1904.12584，2019。

[60] Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1):1-62, 2022.

[60] Yann LeCun。迈向自主机器智能的路径 0.9 版。2，2022-06-27。Open Review，62(1):1-62，2022。

[61] Eric Xing, Mingkai Deng, Jinyu Hou, and Zhiting Hu. Critiques of world models. arXiv preprint arXiv:2507.05169, 2025.

[61] Eric Xing、Mingkai Deng、Jinyu Hou 和 Zhiting Hu。世界模型批判。arXiv 预印本 arXiv:2507.05169，2025年。

[62] Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention. arXiv preprint arXiv:2402.08268, 2024

[62] Hao Liu、Wilson Yan、Matei Zaharia 和 Pieter Abbeel。基于 blockwise ringattention 的百万长度视频与语言世界模型。arXiv 预印本 arXiv:2402.08268，2024 年。

[63] Qineng Wang, Wenlong Huang, Yu Zhou, Hang Yin, Tianwei Bao, Jianwen Lyu, Weiyu Liu, Ruohan Zhang, Jiajun Wu, Li Fei-Fei, et al. Enact: Evaluating embodied cognition with world modeling of egocentric interaction. arXiv preprint arXiv:2511.20937, 2025. 2

[63] Qineng Wang、Wenlong Huang、Yu Zhou、Hang Yin、Tianwei Bao、Jianwen Lyu、Weiyu Liu、RuohanZhang、Jiajun Wu、Li Fei-Fei 等。Enact：通过对第一人称交互进行世界建模来评估具身认知。arXiv 预印本arXiv:2511.20937，2025年。2

[64] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning

[64] Danijar Hafner、Timothy Lillicrap、Ian Fischer、Ruben Villegas、David Ha、Honglak Lee 和 James Davidson。学习

latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551, 2018. 2, 3

用于从像素进行规划的潜在动力学。arXiv 预印本 arXiv:1811.04551，2018。2， 3

[65] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019. 21

[65] Danijar Hafner、Timothy Lillicrap、Jimmy Ba 和 Mohammad Norouzi。 Dream to control：通过潜在想象学习行为。arXiv 预印本 arXiv:1912.01603，2019。21

[66] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020.

[66] Danijar Hafner、Timothy Lillicrap、Mohammad Norouzi 和 Jimmy Ba。使用离散世界模型掌握 Atari。arXiv预印本 arXiv:2010.02193，2020。

[67] Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16828, 2023. 2, 3

[67] Nicklas Hansen、Hao Su 和 Xiaolong Wang。Td-mpc2：用于连续控制的可扩展、鲁棒世界模型。arXiv 预印本arXiv:2310.16828，2023。2, 3

[68] Han Qi, Haocheng Yin, Aris Zhu, Yilun Du, and Heng Yang. Strengthening generative robot policies through predictive world modeling. arXiv preprint arXiv:2502.00622, 2025.

[68] Han Qi、Haocheng Yin、Aris Zhu、Yilun Du 和 Heng Yang。通过预测性世界建模强化生成式机器人策略。arXiv 预印本 arXiv:2502.00622，2025。

[69] Yanjiang Guo, Lucy Xiaoyang Shi, Jianyu Chen, and Chelsea Finn. Ctrl-world: A controllable generative world model

for robot manipulation. arXiv preprint arXiv:2510.10125, 2025. 3

[69] Yanjiang Guo、Lucy Xiaoyang Shi、Jianyu Chen 和 Chelsea Finn。Ctrl-world:—种用于机器人操作的可控生成式世界模型。arXiv 预印本 arXiv:2510.10125，2025。3

[7o] Chenhao Li, Andreas Krause, and Marco Hutter. Robotic world model: A neural network simulator for robust policy optimization in robotics. arXiv preprint arXiv:2501.10100, 2025. 2

[70] Chenhao Li、Andreas Krause 和 Marco Hutter。Robotic world model:—种用于机器人领域稳健策略优化的神经网络模拟器。arXiv 预印本 arXiv:2501.10100，2025。2

[71] Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024. 2

[71] Jake Bruce、Michael D Dennis、Ashley Edwards、Jack Parker-Holder、Yuge Shi、Edward Hughes、Matthew Lai、Aditi Mavalankar、Richie Steigerwald、Chris Apps 等。Genie:生成式交互环境。见 Forty-firstInternational Conference on Machine Learning, 2024。 2

[72] Jensen Zhou, Hang Gao, Vikram Voleti, Aaryaman Vasishta, Chun-Han Yao, Mark Boss, Philip Torr, Christian Rupprecht, and Varun Jampani. Stable virtual camera: Generative view synthesis with diffusion models. arXiv preprint arXiv:2503.14489, 2025.

[72] Jensen Zhou、Hang Gao、Vikram Voleti、Aaryaman Vasishta、Chun-Han Yao、Mark Boss、Philip Torr、Christian Rupprecht 和 Varun Jampani。Stable virtual camera：使用 diffusion models 的生成式视图合成。arXiv预印本 arXiv:2503.14489，2025。

[73] Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101, 2024.

[73] Hao He、Yinghao Xu、Yuwei Guo、Gordon Wetzstein、Bo Dai、Hongsheng Li 和 Ceyuan Yang。Cameractrl：为 text-to-video generation 提供相机控制。arXiv 预印本 arXiv:2404.02101，2024。

[74] Xianglong He, Chunli Peng, Zexiang Liu, Boyang Wang, Yifan Zhang, Qi Cui, Fei Kang, Biao Jiang, Mengyin An, Yangyang Ren, et al. Matrix-game 2.o: An open-source, real-time, and streaming interactive world model. arXiv preprint arXiv:2508.13009, 2025.

[74] Xianglong He、Chunli Peng、Zexiang Liu、Boyang Wang、Yifan Zhang、Qi Cui、Fei Kang、Biao Jiang、Mengyin An、Yangyang Ren 等。Matrix-game 2.0：一个开源、实时且流式的交互式 world model。arXiv 预印本arXiv:2508.13009，2025。

[75] Jiaqi Li, Junshu Tang, Zhiyong Xu, Longhuang Wu, Yuan Zhou, Shuai Shao, Tianbao Yu, Zhiguo Cao, and Qinglin Lu. Hunyuan-gamecraft: High-dynamic interactive game video generation with hybrid history condition. arXiv preprint, 2025. arXiv preprint.

[75] Jiaqi Li、Junshu Tang、Zhiyong Xu、Longhuang Wu、Yuan Zhou、Shuai Shao、Tianbao Yu、Zhiguo Cao和 Qinglin Lu。Hunyuan-gamecraft：基于混合历史条件的高动态交互式游戏视频生成。arXiv 预印本，2025。arXiv预印本。

[76] Guangcong Zheng, Teng Li, Rui Jiang, Yehao Lu, Tao Wu, and Xi Li. Cami2v: Camera-controlled image-to-video diffusion model. arXiv preprint arXiv:2410.15957, 2024. [77] Chonghyuk Song, Michal Stary, Boyuan Chen, George Kopanas, and Vincent Sitzmann. Generative view stitching. arXiv preprint arXiv:2510.24718, 2025.

[77] Chonghyuk Song、Michal Stary、Boyuan Chen、George Kopanas 和 Vincent Sitzmann。生成式视图拼接。arXiv 预印本 arXiv:2510.24718，2025 年。

[78] Siyuan Zhou, Yilun Du, Yuncong Yang, Lei Han, Peihao Chen, Dit-Yan Yeung, and Chuang Gan. Learning 3d persistent embodied world models. arXiv preprint arXiv:2505.05495, 2025. 2 [79] Sherry Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Leslie Kaelbling, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. arXiv preprint arXiv:2310.06114, 2023. 2

[79] Sherry Yang、Yilun Du、Kamyar Ghasemipour、Jonathan Tompson、Leslie Kaelbling、Dale Schuurmans和 Pieter Abbeel。学习交互式真实世界模拟器。arXiv 预印本 arXiv:2310.06114，2023 年。2

[8o] Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025.4

[80] Niket Agarwal、Arslan Ali、Maciej Bala、Yogesh Balaji、Erik Barker、Tiffany Cai、PrithvijitChattopadhyay、Yongxin Chen、Yin Cui、Yifan Ding 等。用于 Physical AI的 Cosmos world Foundation Model平台。arXiv 预印本 arXiv:2501.03575，2025 年。4

[81] Hassan Abu Alhaija, Jose Alvarez, Maciej Bala, Tiffany Cai, Tianshi Cao, Liz Cha, Joshua Chen, Mike Chen, Francesco Ferroni, Sanja Fidler, et al. Cosmos-transfer1: Conditional world generation with adaptive multimodal control. arXiv preprint arXiv:2503.14492, 2025.

[81] Hassan Abu Alhaija、Jose Alvarez、Maciej Bala、Tiffany Cai、Tianshi Cao、Liz Cha、Joshua Chen、MikeChen、Francesco Ferroni、Sanja Fidler 等。Cosmos-transfer1：具有自适应多模态控制的条件世界生成。arXiv 预印本 arXiv:2503.14492，2025 年。

[82] Ruijie Zheng, Jing Wang, Scott Reed, Johan Bjorck, Yu Fang, Fengyuan Hu, Joel Jang, Kaushil Kundalia, Zongyu Lin Loic Magne, Avnish Narayan, You Liang Tan, Guanzhi Wang, Qi Wang, Jiannan Xiang, Yinzhen Xu, Seonghyeon Ye, Jan Kautz, Furong Huang, Yuke Zhu, and Linxi Fan. Flare: Robot learning with implicit world modeling. arXiv preprint arXiv:2505.15659, 2025.

[83] Joel Jang, Seonghyeon Ye, Zongyu Lin, Jiannan Xiang, Johan Bjorck, Yu Fang, Fengyuan Hu, Spencer Huang, Kaushil Kundalia, Yen-Chen Lin, Loic Magne, Ajay Mandlekar, Avnish Narayan, You Liang Tan, Guanzhi Wang, Jing Wang, Qi Wang, Yinzhen Xu, Xiaohui Zeng, Kaiyuan Zheng, Ruijie Zheng, Ming-Yu Liu, Luke Zettlemoyer, Dieter Fox, Jan Kautz, Scott Reed, Yuke Zhu, and Linxi Fan. Dreamgen: Unlocking generalization in robot learning through video world models arXiv preprint arXiv:2505.12705, 2025. 2

[84] Ryan Burgert, Yuancheng Xu, Wenqi Xian, Oliver Pilarski, Pascal Clausen, Mingming He, Li Ma, Yitong Deng, Lingxiao Li, Mohsen Mousavi, et al. Go-with-the-flow: Motioncontrollable video diffusion models using real-time warped noise. arXiv preprint arXiv:2501.08331, 2025. 2

[84] Ryan Burgert、Yuancheng Xu、Wenqi Xian、Oliver Pilarski、Pascal Clausen、Mingming He、Li Ma、Yitong Deng、Lingxiao Li、Mohsen Mousavi 等人。Go-with-the-flow：使用实时扭曲噪声的运动可控视频扩散模型。arXiv 预印本 arXiv:2501.08331，2025。2

[85] Muyao Niu, Xiaodong Cun, Xintao Wang, Yong Zhang, Ying Shan, and Yinqiang Zheng. Mofa-video: Controllable image animation via generative motion field adaptions in frozen image-to-video diffusion model. arXiv preprint arXiv:2405.20222, 2024.

[85] Muyao Niu、Xiaodong Cun、Xintao Wang、Yong Zhang、Ying Shan 和 Yinqiang Zheng。Mofa-video :在冻结的 image-to-video 扩散模型中通过生成式运动场自适应实现可控图像动画。arXiv预印本 arXiv:2405.20222，2024。

[86] Daniel Geng, Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Tatiana Lopez-Guevara, Carl Doersch, Yusuf Aytar, Michael Rubinstein, et al. Motion prompting: Controlling video generation with motion trajectories. arXiv preprint arXiv:2412.02700, 2024.

[86] Daniel Geng、Charles Herrmann、Junhwa Hur、Forrester Cole、Serena Zhang、Tobias Pfaff、TatianaLopez-Guevara、Carl Doersch、Yusuf Aytar、Michael Rubinstein等人。Motion prompting：使用运动轨迹控制视频生成。arXiv 预印本 arXiv:2412.02700，2024。

[87] Nate Gillman, Charles Herrmann, Michael Freeman, Daksh Aggarwal, Evan Luo, Deqing Sun, and Chen Sun. Force prompting: Video generation models can learn and generalize physics-based control signals. arXiv preprint, 2025. arXiv preprint.

[87] Nate Gillman、Charles Herrmann、Michael Freeman、Daksh Aggarwal、Evan Luo、Deqing Sun 和 ChenSun。Force prompting：视频生成模型可以学习并泛化基于物理的控制信号。arXiv 预印本，2025。arXiv 预印本。

[88] Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. arXiv preprint arXiv:2311.17117, 2023.

[88] Li Hu、Xin Gao、Peng Zhang、Ke Sun、Bang Zhang 和 Liefeng Bo。Animate anyone：用于角色动画的—致且可控的图像到视频合成。arXiv 预印本 arXiv:2311.17117，2023。

[89] Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling. In ACM SIGGRAPH Conference Papers, 2024.

具有显式运动建模的可控图像到视频生成。见 ACM SIGGRAPH Conference Papers，2024。

[9o] Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Li, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH Conference Papers, 2024.

[90] Zhouxia Wang、Ziyang Yuan、Xintao Wang、Yaowei Li、Tianshui Chen、Menghan Xia、Ping Li、PingLuo 和 Ying Shan。Motionctrl：用于视频生成的统一且灵活的运动控制器。见 ACM SIGGRAPH ConferencePapers,2024。

[91] Weijia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. Draganything: Motion control for anything using entity representation. In European Conference on Computer Vision, 2024.

[91] Weijia Wu、Zhuang Li、Yuchao Gu、Rui Zhao、Yefei He、David Junhao Zhang、Mike Zheng Shou、YanLi、Tingting Gao 和 Di Zhang。Draganything：使用实体表示实现对任意对象的运动控制。见 EuropeanConference on Computer Vision, 2024.

[92] Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308.08089, 2023.

[92] Shengming Yin、Chenfei Wu、Jian Liang、Jie Shi、Houqiang Li、Gong Ming 和 Nan Duan。 Dragnuwa :通过集成文本、图像和轨迹实现视频生成中的细粒度控制。arXiv 预印本 arXiv:2308.08089，2023。

[93] Zhenghao Zhang, Junchao Liao, Menghao Li, Zuozhuo Dai, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Tora: Trajectory-oriented diffusion transformer for video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.

[93] Zhenghao Zhang、Junchao Liao、Menghao Li、Zuozhuo Dai、Bingxue Qiu、Siyu Zhu、Long Qin 和Weizhi Wang。Tora：面向视频生成的轨迹导向扩散 Transformer。见 Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition, 2025。

[94] Jinbo Xing, Hanyuan Liu, Menghan Xia, Yong Zhang, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Tooncrafter: Generative cartoon interpolation. ACM Transactions on Graphics, 2024. 2

[94] Jinbo Xing、Hanyuan Liu、Menghan Xia、Yong Zhang、Xintao Wang、Ying Shan 和 Tien-Tsin Wong。Tooncrafter：生成式卡通插值。ACM Transactions on Graphics，2024。2

[95] Pieter-Tjerk de Boer, Dirk P. Kroese, Shie Mannor, and Reuven Y. Rubinstein. A tutorial on the cross-entropy method. Annals of Operations Research, 134(1):19-67, 2005. 2

[95] Pieter-Tjerk de Boer、Dirk P. Kroese、Shie Mannor 和 Reuven Y. Rubinstein。交叉熵方法教程。Annals ofOperations Research，134(1):19-67，2005。 2

[96] Matthew Kelly. An introduction to trajectory optimization: How to do your own direct collocation. SIAM Review, 59 (4):849-904, 2017.

[96] Matthew Kelly. 轨迹优化导论：如何自己进行 direct collocation。SIAM Review，59 (4):849-904，2017。

[97] Joelle Pineau, Geoffrey Gordon, and Sebastian Thrun. Pointbased value iteration: An anytime algorithm for pomdps. In Proceedings of the 18th International Joint Conference on Artificial Intelligence (IJCAI), 2003. 2

[97] Joelle Pineau、Geoffrey Gordon 和 Sebastian Thrun。基于点的价值迭代：—种用于 pomdps 的 anytime 算法。见 Proceedings of the 18th International Joint Conference on Artificial Intelligence (IJCAI)， 2003。 2

[98] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. arXiv preprint arXiv:1906.08253, 2019. 2, 3

[98] Michael Janner、Justin Fu、Marvin Zhang 和 Sergey Levine。何时信任你的模型：基于模型的策略优化。arXivpreprint arXiv:1906.08253，2019。 2，3

[99] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. arXiv preprint arXiv:1805.12114, 2018.

[99] Kurtland Chua、Roberto Calandra、Rowan McAllister 和 Sergey Levine。利用概率动力学模型在少量试验中进行深度强化学习。arXiv preprint arXiv:1805.12114，2018。

[1oo] Danijar Hafner, Wilson Yan, and Timothy Lillicrap. Training agents inside of scalable world models. arXiv preprint arXiv:2509.24527, 2025. 2, 3

[100] Danijar Hafner、Wilson Yan 和 Timothy Lillicrap. 在可扩展世界模型内部训练智能体。arXiv 预印本arXiv:2509.24527，2025。2，3

[1o1] Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. Advances in neural information processing systems, 29, 2016. 2

[101] Pulkit Agrawal、Ashvin V Nair、Pieter Abbeel、Jitendra Malik 和 Sergey Levine. 通过戳刺来学习戳刺：直觉物理的体验式学习。Advances in neural information processing systems，29，2016。2

[1o2] Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in neural information processing systems, 36:9156- 9172, 2023.

[102] Yilun Du、Sherry Yang、Bo Dai、Hanjun Dai、Ofir Nachum、Josh Tenenbaum、Dale Schuurmans 和Pieter Abbeel. 通过文本引导的视频生成学习通用策略。Advances in neural information processing systems，36:9156-9172，2023。

[1o3] Jie Zhang, Yilun Du, et al. Learning to act from actionless videos through dense correspondences. arXiv preprint arXiv:2310.08576, 2023. 2, 3

[103] Jie Zhang、Yilun Du 等. 通过密集对应关系从无动作视频中学习行动。arXiv 预印本 arXiv:2310.08576，2023。2,3

[104] Erwin Coumans. Bullet physics simulation. In ACM SIGGRAPH 2015 Courses, 2015. 3

[104] Erwin Coumans. Bullet 物理仿真。见 ACM SIGGRAPH 2015 Courses，2015。3

[1o5] Russ Tedrake and the Drake Development Team. Drake: A planning, control, and analysis toolbox for nonlinear systems. Technical Report, MIT CSAIL, https: / / drake. mit.edu, 2019. 3

[105] Russ Tedrake 和 Drake Development Team。Drake：一个用于非线性系统的规划、控制与分析工具箱。技术报告，MIT CSAIL，https: / / drake. mit.edu，2019。3

[1o6] Viktor Makoviychuk, Ilya Wawrzyniak, Sergii Gupta, Anurag Narang, Maciej Kayastha, Yashraj Henry, Guanyang Li, Igor Ling, Andreas Merentitis, Alexey Makoviychuk, Kaichun Tsai, Gabe State, Dieter Fox, and Ankur Handa. Isaac gym: High performance gpu based physics simulation for robot learning. arXiv preprint arXiv:2108.10470, 2021.

[106] Viktor Makoviychuk、Ilya Wawrzyniak、Sergii Gupta、Anurag Narang、Maciej Kayastha、YashrajHenry、Guanyang Li、Igor Ling、Andreas Merentitis、Alexey Makoviychuk、Kaichun Tsai、Gabe State、Dieter Fox 和 Ankur Handa。Isaac gym：面向机器人学习的高性能基于 GPU 的物理仿真。arXiv 预印本arXiv:2108.10470，2021。

[1o7] Nathan Koenig and Andrew Howard. Design and use paradigms for gazebo, an open-source multi-robot simulator. In 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2149-2154, 2004

[107] Nathan Koenig 和 Andrew Howard。gazebo 的设计与使用范式：一个开源多机器人模拟器。见 2004IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)，第 2149-2154 页，2004。

[1o8] Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Frédo Durand. Difftaichi: Differentiable programming for physical simulation. arXiv preprint arXiv:1910.00935, 2020. 3

[108] Yuanming Hu、Luke Anderson、Tzu-Mao Li、Qi Sun、Nathan Carr、 Jonathan Ragan-Kelley 和 FrédoDurand。Difftaichi：用于物理仿真的可微分编程。arXiv 预印本 arXiv:1910.00935，2020。3

[1o9] Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Daydreamer: World models for physical robot learning. In Conference on robot learning, pages 2226-2240. PMLR, 2023. 3

[109] Philipp Wu、Alejandro Escontrela、Danijar Hafner、Pieter Abbeel 和 Ken Goldberg。 Daydreamer :用于物理机器人学习的世界模型。见 Conference on robot learning，第 2226-2240 页。PMLR，2023。3

[11o] Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2o18 IEEE international conference on robotics and automation (ICRA), pages 38033810. IEEE, 2018. 3

[110] Xue Bin Peng、Marcin Andrychowicz、Wojciech Zaremba 和 Pieter Abbeel。通过动力学随机化实现机器人控制从仿真到现实的迁移。见 2018 IEEE international conference on robotics and automation (ICRA)，第38033810页。IEEE，2018。3

[111] OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.

[111] OpenAI:Marcin Andrychowicz、Bowen Baker、Maciek Chociej、Rafal Jozefowicz、Bob McGrew、JakubPachocki、Arthur Petron、Matthias Plappert、Glenn Powell、Alex Ray等。学习灵巧的手内操纵。TheInternational Journal of Robotics Research, 39(1):3-20， 2020。

[112] Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.04034, 2021. 3

[112] Ashish Kumar、Zipeng Fu、Deepak Pathak 和 Jitendra Malik。Rma：用于腿式机器人的快速运动适应。arXiv 预印本 arXiv:2107.04034，2021。3

[113] Tomas Lozano-Perez. Spatial planning: A configuration space approach. IEEE transactions on computers, 32(o2): 108-120, 1983.3

[113] Tomas Lozano-Perez。空间规划：—种构型空间方法。IEEE Transactions on Computers，32(02)：108-120，1983。3

[114] Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134, 1998.3

[114] Leslie Pack Kaelbling、Michael L Littman 和 Anthony R Cassandra。部分可观测随机域中的规划与行动。Artificial Intelligence，101(1-2):99-134，1998。3

[115] Marc Toussaint. Logic-geometric programming: An optimization-based approach to combined task and motion planning. In IJCAI, pages 1930-1936, 2015.

[115] Marc Toussaint。逻辑-几何规划：一种基于优化的任务与运动联合规划方法。见 IJCAI，页 1930-1936，2015。

[116] Tobia Marcucci, Jack Umenberger, Pablo Parrilo, and Russ Tedrake. Shortest paths in graphs of convex sets. SIAM Journal on Optimization, 34(1):507-532, 2024. 3

[116] Tobia Marcucci、Jack Umenberger、Pablo Parrilo 和 Russ Tedrake. 凸集图中的最短路径. SIAM Journal onOptimization, 34(1):507-532, 2024. 3

[117] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pages 2778-2787. PMLR, 2017. 3

[117] Deepak Pathak、Pulkit Agrawal、Alexei A Efros 和 Trevor Darrell. 通过自监督预测驱动的好奇心探索. 见International conference on machine learning, 第 2778-2787 页. PMLR, 2017. 3

[118] Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International

[118] Ramanan Sekar、Oleh Rybkin、Kostas Daniilidis、Pieter Abbeel、Danijar Hafner 和 Deepak Pathak. 通过自监督世界模型进行探索规划. 见 International

conference on machine learning, pages 8583-8592. PMLR, 2020.

conference on machine learning, 第 8583-8592 页. PMLR, 2020.

[119] Igor Mordatch and Emo Todorov. Combining the benefits of function approximation and trajectory optimization. In Robotics: Science and Systems, page 23, 2014.

[119] Igor Mordatch 和 Emo Todorov。结合函数逼近与轨迹优化的优势。见 Robotics: Science and Systems，第23页，2014年。

[12o] Maximilian Du and Shuran Song. Dynaguide: Steering diffusion polices with active dynamic guidance. arXiv preprint arXiv:2506.13922, 2025. 3

[120] Maximilian Du 和 Shuran Song。Dynaguide:通过主动动态引导控制 diffusion polices。arXiv preprintarXiv:2506.13922，2025 年。3

[121] Aaron D Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada. Control barrier functions: Theory and applications. In 2019 18th European control conference (ECC), pages 3420-3431. Ieee, 2019.3

[121] Aaron D Ames、Samuel Coogan、Magnus Egerstedt、Gennaro Notomista、Koushil Sreenath 和 PauloTabuada。控制屏障函数：理论与应用。见 2019 18th European control conference (ECC)，第 3420-3431 页。Ieee，2019年。3

[122] Zhanyi Sun and Shuran Song. Latent policy barrier: Learning robust visuomotor policies by staying in-distribution. arXiv preprint arXiv:2508.05941, 2025. 3

[122] Zhanyi Sun 和 Shuran Song。Latent policy barrier：通过保持分布内学习鲁棒的 visuomotor 策略。arXivpreprint arXiv:2508.05941，2025年。3

[123] Alexandre Donze. Breach, a toolbox for verification and parameter synthesis of hybrid systems. In Computer Aided Verification, pages 167-170. Springer, 2010. 3

[123] Alexandre Donze. Breach，一个用于混合系统验证和参数综合的工具箱。见 Computer Aided Verification，第167-170 页。Springer，2010。3

[124] Xiaomeng Xu, Huy Ha, and Shuran Song. Dynamics-guided diffusion model for robot manipulator design. CoRR, 2024.3

[124] Xiaomeng Xu、Huy Ha 和 Shuran Song. 用于机器人机械臂设计的动力学引导扩散模型。CoRR，2024。3

[125] Jose Barreiros, Andrew Beaulieu, Aditya Bhat, Rick Cory, Eric Cousineau, Hongkai Dai, Ching-Hsin Fang, Kunimatsu Hashimoto, Muhammad Zubair Irshad, Masha Itkina, et al. A careful examination of large behavior models for multitask dexterous manipulation. arXiv preprint arXiv:2507.05331, 2025. 3, 8

[125] Jose Barreiros、Andrew Beaulieu、Aditya Bhat、Rick Cory、Eric Cousineau、Hongkai Dai、Ching-HsinFang、Kunimatsu Hashimoto、Muhammad Zubair Irshad、Masha Itkina 等。对用于多任务灵巧操作的大型行为模型的审慎考察。arXiv 预印本 arXiv:2507.05331，2025。3，8

[126] Sherry Yang et al. Evaluating robot policies in a world model. arXiv preprint arXiv:2506.00613, 2025.

[126] Sherry Yang 等。 在世界模型中评估机器人策略。arXiv 预印本 arXiv:2506.00613，2025。

[127] Gemini Robotics Team, Coline Devin, Yilun Du, Debidatta Dwibedi, Ruiqi Gao, Abhishek Jindal, Thomas Kipf, Sean Kirmani, Fangchen Liu, Anirudha Majumdar, et al. Evaluating gemini robotics policies in a veo world simulator. arXiv preprint arXiv:2512.10675, 2025. 3

[127] Gemini Robotics Team、Coline Devin、Yilun Du、Debidatta Dwibedi、Ruiqi Gao、Abhishek Jindal、Thomas Kipf、Sean Kirmani、Fangchen Liu、Anirudha Majumdar 等。在 veo world simulator 中评估 geminirobotics 策略。arXiv 预印本 arXiv:2512.10675，2025。3

[128] Yuang Wang, Chao Wen, Haoyu Guo, Sida Peng, Minghan Qin, Hujun Bao, Xiaowei Zhou, and Ruizhen Hu. Precise action-to-video generation through visual action prompts. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12713-12724, 2025. 3

[128] Yuang Wang、Chao Wen、Haoyu Guo、Sida Peng、Minghan Qin、Hujun Bao、Xiaowei Zhou 和Ruizhen Hu。通过视觉动作 Prompt 实现精确的动作到视频生成。见 Proceedings of the IEEE/CVF InternationalConference on Computer Vision，页 12713-12724，2025。 3

[129] Haonan Chen, Yilong Niu, Kaiwen Hong, Shuijing Liu, Yixuan Wang, Yunzhu Li, and Katherine Rose Driggs-Campbell. Predicting object interactions with behavior primitives: An application in stowing tasks. In 7th Annual Conference on Robot Learning, 2023. 3

[129] Haonan Chen、Yilong Niu、Kaiwen Hong、Shuijing Liu、Yixuan Wang、Yunzhu Li 和 Katherine RoseDriggs-Campbell。使用行为基元预测物体交互：在收纳任务中的应用。见第 7 届 Annual Conference on RobotLearning，2023。3

[13o] D. Sulsky, Z. Chen, and H. L. Schreyer. A particle method for history-dependent materials. Technical report, U.S. Department of Energy, OSTI, 1993. 3

[130] D. Sulsky、Z. Chen 和 H.L. Schreyer。一种用于历史相关材料的粒子方法。技术报告，美国能源部，OSTI，1993。3

[131] Carl Doersch, Pauline Luc, Yi Yang, Dilara Gokay, Skanda Koppula, Ankush Gupta, Joseph Heyward, Ignacio Rocco, Ross Goroshin, Joao Carreira, et al. Bootstap: Bootstrapped training for tracking-any-point. In Proceedings of the Asian Conference on Computer Vision, pages 3257-3274, 2024. 3

[131] Carl Doersch、Pauline Luc、Yi Yang、Dilara Gokay、Skanda Koppula、Ankush Gupta、JosephHeyward、Ignacio Rocco、Ross Goroshin、Joao Carreira等人。Bootstap：用于 tracking-any-point 的自举训练。载于 Proceedings of the Asian Conference on Computer Vision，页码 3257-3274，2024。3

[132] Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20406-20417, 2024. 3

[132] Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou.Spatialtracker：在 3d 空间中跟踪任意 2d 像素。见 IEEE/CVF Conference on Computer Vision and PatternRecognition 论文集，页 20406-20417，2024。3

[133] Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.00025, 2023. 3

[133] Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. 用于策略学习的任意点轨迹建模。arXiv 预印本 arXiv:2401.00025，2023。3

[134] Siddhant Haldar and Lerrel Pinto. Point policy: Unifying observations and actions with key points for robot manipulation. arXiv preprint arXiv:2502.20391, 2025.

[134] Siddhant Haldar and Lerrel Pinto. Point policy：使用关键点统一观测与动作以进行机器人操作。arXiv 预印本arXiv:2502.20391，2025。

[135] Thomas Weng, Sujay Bajracharya, Yufei Wang, Khush Agrawal, and David Held. Fabricflownet: Bimanual cloth manipulation with a flow-based policy. arXiv preprint arXiv:2111.05623, 2021.

[135] Thomas Weng, Sujay Bajracharya, Yufei Wang, Khush Agrawal, and David Held. Fabricflownet : 基于流的策略实现双手布料操作。arXiv 预印本 arXiv:2111.05623，2021。

[136] Daniel Seita, Yufei Wang, Sarthak J Shetty, Edward Yao Li, Zackory Erickson, and David Held. Toolflownet: Robotic manipulation with tools via predicting tool flow from point clouds. arXiv preprint arXiv:2211.09oo6, 2022.

[136] Daniel Seita、Yufei Wang、Sarthak J Shetty、Edward Yao Li、Zackory Erickson 和 David Held.Toolflownet：通过从点云预测工具流实现使用工具的机器人操作。arXiv preprint arXiv:2211.09006，2022。

[137] Shengjie Wang, Jiacheng You, Yihang Hu, Jiongye Li, and Yang Gao. Skil: Semantic keypoint imitation learning for generalizable data-efficient manipulation. In Robotics: Science and Systems (RSS), 2025.

[137] Shengjie Wang、Jiacheng You、Yihang Hu、Jiongye Li 和 Yang Gao. Skil：用于可泛化且数据高效操作的语义关键点模仿学习。见 Robotics: Science and Systems (RSS)，2025。

[138] Jun Guo, Xiaojian Ma, Yikai Wang, Min Yang, Huaping Liu, and Qing Li. Flowdreamer: A rgb-d world model with flow-based motion representations for robot manipulation. arXiv preprint arXiv:2505.10075, 2025.

[138] Jun Guo、Xiaojian Ma、Yikai Wang、Min Yang、Huaping Liu 和 Qing Li. Flowdreamer :—种用于机器人操作的 RGB-D 世界模型，具有基于流的运动表征。arXiv preprint arXiv:2505.10075，2025。

[139] Bardienus P. Duisterhof, Zhao Mandi, Yunchao Yao, JiaWei Liu, Jenny Seidenschwarz, Mike Zheng Shou, Deva Ramanan, Shuran Song, Stan Birchfield, Bowen Wen, and Jeffrey Ichnowski. Deformgs: Scene flow in highly deformable scenes for deformable object manipulation. arXiv preprint arXiv:2312.00583, 2024.

[139] Bardienus P. Duisterhof、Zhao Mandi、Yunchao Yao、JiaWei Liu、Jenny Seidenschwarz、Mike ZhengShou、Deva Ramanan、Shuran Song、Stan Birchfield、Bowen Wen 和 Jeffrey Ichnowski. Deformgs:用于可变形物体操作的高可变形场景中的场景流。arXiv preprint arXiv:2312.00583，2024。

[14o] Zhao-Heng Yin, Sherry Yang, and Pieter Abbeel. Objectcentric 3d motion field for robot learning from human videos. arXiv preprint arXiv:2506.04227, 2025. 3

[140] Zhao-Heng Yin、Sherry Yang 和 Pieter Abbeel. 以对象为中心的 3D 运动场：从人类视频中进行机器人学习.arXiv 预印本 arXiv:2506.04227, 2025.3

[141] Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gordon Wetzstein, Manuela Veloso, and Shuran Song. Flow as the cross-domain manipulation interface. arXiv preprint arXiv:2407.15208, 2024. 3

[141] Mengda Xu、Zhenjia Xu、Yinghao Xu、Cheng Chi、Gordon Wetzstein、Manuela Veloso 和 ShuranSong. Flow 作为跨域操作接口. arXiv 预印本 arXiv:2407.15208， 2024.3

[142] Shivansh Patel, Xinchen Yin, Wenlong Huang, Shubham Garg, Hooshang Nayyeri, Li Fei-Fei, Svetlana Lazebnik, and Yunzhu Li. A real-to-sim-to-real approach to robotic manipulation with vlm-generated iterative keypoint rewards. arXiv preprint arXiv:2502.08643, 2025.

[142] Shivansh Patel、Xinchen Yin、Wenlong Huang、Shubham Garg、Hooshang Nayyeri、Li Fei-Fei、Svetlana Lazebnik 和 Yunzhu Li. —种 real-to-sim-to-real 的机器人操作方法，使用由 VLM 生成的迭代关键点奖励.arXiv 预印本 arXiv:2502.08643, 2025.

[143] Irmak Guzey, Yinlong Dai, Georgy Savva, Raunaq Bhirangi, and Lerrel Pinto. Bridging the human to robot dexterity gap through object-oriented rewards. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 3344-3351. IEEE, 2025

[143] Irmak Guzey、Yinlong Dai、Georgy Savva、Raunaq Bhirangi 和 Lerrel Pinto. 通过面向对象的奖励弥合人类 与机器人之间的灵巧性差距. 载于 2025 IEEE International Conference on Robotics and Automation (ICRA)，第 3344-3351 页. IEEE, 2025.

[144] Junyao Shi, Joshua Smith, Jianing Qian, and Dinesh Jayaraman. Points2reward: Robotic manipulation rewards from just one video. 2025.3

[144] Junyao Shi、Joshua Smith、Jianing Qian 和 Dinesh Jayaraman. Points2reward :仅通过—个视频获得机器人操作奖励。2025.3

[145] Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation. arXiv preprint arXiv:2409.01652, 2024. 3, 5, 21

[145] Wenlong Huang、Chen Wang、Yunzhu Li、Ruohan Zhang 和 Li Fei-Fei. Rekep：用于机器人操作的关系关键点约束的时空推理。arXiv preprint arXiv:2409.01652，2024. 3， 5， 21

[146] Ben Eisner, Harry Zhang, and David Held. Flowbot3d: Learning 3d articulation flow to manipulate articulated objects. In Robotics: Science and Systems (RSS), 2022. 3

[146] Ben Eisner、Harry Zhang 和 David Held. Flowbot3d：学习 3d articulation flow 以操作铰接物体。见 Robotics: Science and Systems (RSS), 2022. 3

[147] Homanga Bharadhwaj, Roozbeh Mottaghi, Abhinav Gupta, and Shubham Tulsiani. Track2act: Predicting point tracks

[147] Homanga Bharadhwaj、 Roozbeh Mottaghi、Abhinav Gupta 和 Shubham Tulsiani. Track2act :预测点轨迹

from internet videos enables generalizable robot manipulation. In European Conference on Computer Vision, pages 3o6- 324. Springer, 2024.

来自互联网视频使机器人操作具备可泛化能力。发表于 European Conference on Computer Vision，页码 306-324。Springer，2024。

[148] Shivansh Patel, Shraddhaa Mohan, Hanlin Mai, Unnat Jain, Svetlana Lazebnik, and Yunzhu Li. Robotic manipulation by imitating generated videos without physical demonstrations. arXiv preprint arXiv:2507.00990, 2025.

[148] Shivansh Patel、Shraddhaa Mohan、Hanlin Mai、Unnat Jain、Svetlana Lazebnik 和 Yunzhu Li. 通过模仿生成视频而无需物理演示的机器人操作。arXiv 预印本 arXiv:2507.00990，2025。

[149] Hongyu Li, Lingfeng Sun, Yafei Hu, Duy Ta, Jennifer Barry, George Konidaris, and Jiahui Fu. Novaflow: Zero-shot manipulation via actionable flow from generated videos. arXiv preprint arXiv:2510.08568, 2025.

[149] Hongyu Li、Lingfeng Sun、Yafei Hu、Duy Ta、Jennifer Barry、George Konidaris 和 Jiahui Fu. Novaflow :通过来自生成视频的可操作流实现零样本操作。arXiv 预印本 arXiv:2510.08568，2025。

[15o] Karthik Dharmarajan, Wenlong Huang, Ruohan Zhang, Jiajun Wu, and Li Fei-Fei. Dream2flow: Bridging video generation and open-world manipulation with 3d object flow, 2025. Preprint. 3

[150] Karthik Dharmarajan、Wenlong Huang、Ruohan Zhang、Jiajun Wu 和 Li Fei-Fei. Dream2flow :通过 3dobject flow 连接视频生成与开放世界操作，2025。预印本。3

[151] Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023. 3

[151] Tony Z Zhao、Vikash Kumar、Sergey Levine 和 Chelsea Finn. 使用低成本硬件学习细粒度双手操作。arXivpreprint arXiv:2304.13705，2023。3

[152] Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point transformer v3: Simpler, faster, stronger. arXiv preprint arXiv:2312.10035, 2023. 4, 8, 9 [153] Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien Mairal, Hervé Jégou, Patrick Labatut, and Piotr Bojanowski. Dinov3. arXiv preprint arXiv:2508.10104, 2025.4, 9

[153] Oriane Siméoni、Huy V. Vo、Maximilian Seitzer、Federico Baldassarre、Maxime Oquab、Cijo Jose、Vasil Khalidov、Marc Szafraniec、Seungeun Yi、Michaël Ramamonjisoa、Francisco Massa、Daniel Haziza、Luca Wehrstedt、Jianyuan Wang、Timothée Darcet、Théo Moutakanni、Leonel Sentana、Claire Roberts、Andrea Vedaldi、Jamie Tolan、John Brandt、Camille Couprie、Julien Mairal、Hervé Jégou、Patrick Labatut 和Piotr Bojanowski. Dinov3. arXiv preprint arXiv:2508.10104，2025。 4，9

[154] Bardienus P. Duisterhof, Jan Oberst, Bowen Wen, Stan Birchfield, Deva Ramanan, and Jeffrey Ichnowski. Rayst3r: Predicting novel depth maps for zero-shot object completion. arXiv preprint arXiv:2506.05285, 2025. 4

[154] Bardienus P. Duisterhof、Jan Oberst、Bowen Wen、Stan Birchfield、Deva Ramanan 和 Jeffrey Ichnowski.Rayst3r：用于零样本物体补全的新深度图预测。arXiv preprint arXiv:2506.05285，2025。4

[155] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017. 5

[155] Alex Kendall 和 Yarin Gal. 在用于计算机视觉的 bayesian deep learning 中我们需要哪些不确定性？ Advancesin neural information processing systems, 30, 2017。 5

[156] David Novotny, Diane Larlus, and Andrea Vedaldi. Learning 3d object categories by looking around them. In Proceedings of the IEEE international conference on computer vision, pages 5218-5227, 2017. 5

[156] David Novotny、Diane Larlus 和 Andrea Vedaldi。通过环绕观察学习 3d 物体类别。见 Proceedings of theIEEE international conference on computer vision，页 5218-5227，2017。 5

[157] Brent Yi, Chung Min Kim, Justin Kerr, Gina Wu, Rebecca Feng, Anthony Zhang, Jonas Kulhanek, Hongsuk Choi, Yi Ma, Matthew Tancik, and Angjoo Kanazawa. Viser: Imperative, web-based 3d visualization in python, 2o25.7

[157] Brent Yi、Chung Min Kim、Justin Kerr、Gina Wu、Rebecca Feng、Anthony Zhang、Jonas Kulhanek、Hongsuk Choi、Yi Ma、Matthew Tancik 和 Angjoo Kanazawa。Viser:基于命令式、Web 的 python 3d 可视化，2025。7

[158] Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x -embodiment: Robotic learning datasets and rt — x models: Open x-embodiment collaboration o. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6892-6903. IEEE, 2024. 6

[158] Abby O’Neill、Abdul Rehman、Abhiram Maddukuri、Abhishek Gupta、Abhishek Padalkar、AbrahamLee、Acorn Pooley、Agrim Gupta、Ajay Mandlekar、Ajinkya Jain 等。Open x-embodiment：机器人学习数据集和 rt — x models : Open x-embodiment collaboration 0。 见 2024 IEEE International Conference on Roboticsand Automation (ICRA)，页 6892-6903。IEEE，2024。6

[159] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 202os. In Proceedings of the IEEE/CVF conference on com-

[159] Zhuang Liu、Hanzi Mao、Chao-Yuan Wu、Christoph Feichtenhofer、Trevor Darrell 和 Saining Xie。面向2020 年代的 convnet。 见 Proceedings of the IEEE/CVF conference on com-

puter vision and pattern recognition, pages 11976-11986, 2022. 8

puter vision and pattern recognition，第 11976-11986 页，2022。8

[16o] Xiaoyang Wu, Daniel DeTone, Duncan Frost, Tianwei Shen, Chris Xie, Nan Yang, Jakob Engel, Richard Newcombe, Hengshuang Zhao, and Julian Straub. Sonata: Selfsupervised learning of reliable point representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. 8, 9

[160] Xiaoyang Wu, Daniel DeTone, Duncan Frost, Tianwei Shen, Chris Xie, Nan Yang, Jakob Engel, Richard Newcombe, Hengshuang Zhao, and Julian Straub. Sonata: 可靠点表示的自监督学习。见 Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025。 8, 9

[161] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-66o, 2017.9

[161] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: 用于 3d 分类与分割的点集深度学习。 见 Proceedings of the IEEE conference on computer vision and pattern recognition，第 652-660 页，2017。9

[162] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017. 9

[162] Charles Ruizhongtai Qi, Li Yi, Hao Su， and Leonidas J Guibas. Pointnet+ +: 度量空间中点集的深度分层特征 学习。 Advances in neural information processing systems，30，2017。9

[163] Spconv Contributors. Spconv: Spatially sparse convolution library. https://github.com/traveller59/ spconv, 2022. 9

[163] Spconv Contributors. Spconv：空间稀疏卷积库。https://github.com/traveller59/ spconv，2022。9

[164] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 9

[164] Ashish Vaswani、Noam Shazeer、Niki Parmar、Jakob Uszkoreit、Llion Jones、Aidan N Gomez、Łukasz Kaiser 和 Illia Polosukhin. Attention is all you need。 Advances in neural information processing systems,30, 2017。9

[165] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. 9, 11

[165] Jared Kaplan、Sam McCandlish、Tom Henighan、Tom B. Brown、Benjamin Chess、Rewon Child、ScottGray、Alec Radford、Jeff Wu 和 Dario Amodei. 神经语言模型的缩放定律。arXiv preprint arXiv:2001.08361，2020。9，11

[166] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. In International Conference on Machine Learning, 2022.

[166] Aakanksha Chowdhery、Sharan Narang、Jacob Devlin、Maarten Bosma、Gaurav Mishra、AdamRoberts、Paul Barham、Hyung Won Chung、Charles Sutton、Sebastian Gehrmann 等. Palm:使用 Pathways扩展语言建模。载于 International Conference on Machine Learning，2022。

[167] Jean-Baptiste Alayrac, Dustin Donato, Antoine Ortega, Jacob Menick, Aidan Clark, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022. 11

[167] Jean-Baptiste Alayrac、Dustin Donato、Antoine Ortega、Jacob Menick、Aidan Clark 等人。Flamingo :—种用于 Few-shot Learning 的视觉语言模型。arXiv 预印本 arXiv:2204.14198，2022。11

[168] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, page 1, 2004. 21

[168] Pieter Abbeel 和 Andrew Y Ng。通过逆强化学习进行学徒学习。见 Proceedings of the twenty-firstinternational conference on Machine learning，第 1 页，2004。 21

[169] Sizhe Lester Li, Annan Zhang, Boyuan Chen, Hanna Matusik, Chao Liu, Daniela Rus, and Vincent Sitzmann. Controlling diverse robots by inferring jacobian fields with deep networks. Nature, pages 1-7, 2025. 21

[169] Sizhe Lester Li、Annan Zhang、Boyuan Chen、Hanna Matusik、Chao Liu、Daniela Rus 和 VincentSitzmann。通过使用深度网络推断jacobian 场来控制多样化机器人。Nature，第1-7页，2025。21

[17o] Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, pages 226-231, 1996. 23

[170] Martin Ester、Hans-Peter Kriegel、Jörg Sander 和 Xiaowei Xu。—种基于密度的算法，用于在含噪声的大型空间数据库中发现聚类。见 Proceedings of the Second International Conference on Knowledge Discovery andData Mining，第 226-231 页，1996。23

[171] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023. 30 [172] Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. arXiv preprint arXiv:2402.10329, 2024. 30

[172] Cheng Chi、Zhenjia Xu、Chuer Pan、Eric Cousineau、Benjamin Burchfiel、Siyuan Feng、Russ Tedrake和 Shuran Song。Universal manipulation interface：无需真实野外机器人即可进行真实场景机器人教学。arXiv 预印本 arXiv:2402.10329，2024。30

[173] Yifeng Zhu, Abhishek Joshi, Peter Stone, and Yuke Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors. arXiv preprint arXiv:2210.11339, 2022. 30

[173] Yifeng Zhu、Abhishek Joshi、Peter Stone 和 Yuke Zhu。Viola：用于基于视觉操作的模仿学习，结合对象提议先验。arXiv 预印本 arXiv:2210.11339，2022。30

[174] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, ChaoYuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 30

[174] Nikhila Ravi、Valentin Gabeur、Yuan-Ting Hu、Ronghang Hu、Chaitanya Ryali、Tengyu Ma、HaithamKhedr、Roman Rädle、Chloe Rolland、Laura Gustafson、Eric Mintun、Junting Pan、Kalyan Vasudev Alwala、Nicolas Carion、ChaoYuan Wu、Ross Girshick、Piotr Dollár 和 Christoph Feichtenhofer。Sam 2:在图像和视频中分割任何内容。arXiv 预印本 arXiv:2408.00714，2024。30

[175] Kevin Zakka, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, Erik Frey, Koushil Sreenath, Lueder A Kahrs, et al. Mujoco playground. arXiv preprint arXiv:2502.08844, 2025. 30

[175] Kevin Zakka、Baruch Tabanpour、Qiayuan Liao、Mustafa Haiderbhai、Samuel Holt、Jing Yuan Luo、Arthur Allshire、Erik Frey、Koushil Sreenath、Lueder A Kahrs 等。Mujoco playground。arXiv 预印本arXiv:2502.08844，2025。30

A.附录

Appendix Contents

附录目录

A.1. Extended Discussions on Limitations … 21 A.1. 关于局限性的扩展讨论…. 21

A. 2 DROID 3D Annotation Pipeline … 22 A. 2 DROID 3D 标注流程… 22

A.2.1. Depth Estimation … 22A.2.1. 深度估计 … 22

A.2.2. Camera Pose Estimation … 22 A.2.2. 相机位姿估计 …. 22

A.2.3. Benchmark Metrics for 3D Annotation … 23 A.2.3. 用于 3D 标注的基准指标… 23

A.2.4. Occlusion-Aware Tracking in 3D … 23

A.3 BEHAVIOR-1K Data Generation … 24A.3 BEHAVIOR-1K 数据生成.…. 24

A.3.1. Dataset Replay … 24
A.3.1. 数据集回放… 24

A.3.2 Clip Filtering … 24 A.3.2 视频片段过滤…. 24

A.3.3. 3D Point Flows from Simulation … 24A.3.3. 来自仿真的 3D 点流… 24

A.4 Model Training Details … 26 A.4 模型训练细节 … 26

A.5. DROID Evaluation Protocol … 29 A.5. DROID 评估协议.…. 29

A.6 Real-Robot Experiment Details … 30

A.6 真实机器人实验细节… 30

A.6.1. 基于模型的规划 … 30

A.6.2. Task Specification … 30

A.6.2. 任务规范 … 30

A.6.3. Evaluation Protocol … 30 A.6.3. 评估协议 . 30

A.6.4. Effect of Training Mixture … 30

A.7. Additional 3D Annotation Examples … 31

A.7. 更多 3D 标注示例… 31

A.8 Additional Model Rollouts … 33

A.8 其他模型 Rollouts … 33

Static Initial State. The model takes as input RGB-D point cloud together with a finite-horizon sequence of robot actions and predicts how points will move in response. Because no prior frames or velocities are provided, this formulation assumes that the world is static at the observation instant. Supporting fully dynamic initial conditions would require augmenting the input with externally tracked trajectories or recurrent state, which we leave as future work.

静态初始状态。该模型以RGB-D点云以及有限时域的机器人动作序列作为输入，并预测点将如何随之移动。由于未提供先前帧或速度信息，这种表述假设世界在观测时刻是静态的。要支持完全动态的初始条件，则需要在输入中加入外部跟踪的轨迹或循环状态，这部分我们留作未来工作。

Reward/Cost Specification for Action Inference. In this work, we explore PointWorld’s use case for action inference in manipulation by integrating it with a sampling-based planner, MPPI [12], which requires an explicit specification of reward/cost functions in the same state-action space of 3D point flows. For the scope of this work, we restrict ourselves to manual specification (e.g., moving a subset of points to target locations). Future work may consider automatically specifying (single or multi-stage) reward via vision-language models [145], or inferring reward from demonstrations using inverse reinforcement learning [168], while keeping PointWORLD as the dynamics function. In addition to planning, action inference can also be alternatively performed by learning a parameterized policy by interacting with the model as the environment via reinforcement learning [65]

用于动作 Inference的奖励/成本规范。在这项工作中，我们通过将 PointWorld 与基于采样的规划器 MPPI [12] 集成，探索了其在操作中进行动作 Inference的用例；该规划器需要在与3D 点流相同的状态-动作空间中对奖励/成本函数进行显式定义。在这项工作的范围内，我们将自己限制于手动指定（例如，将一部分点移动到目标位置）。未来的工作可以考虑通过视觉-语言模型[145]自动指定（单阶段或多阶段）奖励，或使用逆强化学习[168]从演示中推断奖励，同时保持PointWORLD作为动力学函数。除了规划之外，还可以通过强化学习[65]将模型作为环境进行交互，学习参数化策略，从而以另一种方式执行动作 Inference。

Fine-Scale Objects and Calibration Noise. Thin or very small objects (e.g., pens or cables) are challenging to annotate accurately in 3D: modest depth or extrinsic errors can be comparable to the object thickness and lead to ambiguous separation between robot and scene points during groundtruth annotation. In such scenarios, mis-registrations in the ground-truth flows propagate into training and can cause the model to confuse overlapping motions between the gripper and nearby scene points. Improved calibration and depth estimation could strengthen supervision for these fine-grained interactions.

精细尺度物体与标定噪声。细长或非常小的物体（例如笔或电缆）在3D中难以被准确标注：适度的深度或外参误差可能与物体厚度相当，并在 groundtruth 标注过程中导致机器人点与场景点之间的分离产生歧义。在这种情况下，ground-truth flow中的配准误差会传播到训练过程中，并可能导致模型混淆 gripper与附近场景点之间相互重叠的运动。改进标定和深度估计可以为这些细粒度交互提供更强的监督。

Correlation vs. Causation. Given an observed context frame and a sequence of robot actions, PointWorld is trained to predict the subsequent sequence of scene states. As such, it primarily captures correlations present in the training distribution between robot actions, robot motion, and observed scene evolution. In settings where exogenous factors (such as other agents or environment changes not controlled by the robot) also influence the future, these influences are entangled with the robot-induced effects in the data and are not disentangled as separate causal mechanisms. Our experiments therefore evaluate predictive fidelity and planning performance under the observed action-conditioned distribution, rather than claiming to recover the underlying causal structure of the environment.

相关性 vs. 因果性。给定一个观测到的上下文帧和一系列机器人动作，PointWorld 被训练用于预测后续的场景状态序列。因此，它主要捕捉训练分布中机器人动作、机器人运动与观测到的场景演化之间存在的相关性。在外生因素（例如其他智能体，或不受机器人控制的环境变化）也会影响未来的设定中，这些影响会在数据中与由机器人引起的效应纠缠在一起，而不会被解耦为独立的因果机制。因此，我们的实验评估的是在观测到的、以动作为条件的分布下的预测保真度和规划性能，而非声称能够恢复环境的底层因果结构。

Lack of Photometric Dynamics. PointWorld only outputs displacements of 3D points captured from RGB-D inputs, which focus on geometry and physical interactions rather than appearance. While often visually plausible when rendered as point clouds, it is insufficient if one desires to reason about photometric changes of the environment caused by the robot actions. such as lights or screens turning on and off. Combining PointWorld with appearance models that predict emitted radiance, such as those from Gaussian Splatting [4o] or Neural Radiance Fields [39], may be necessary for tasks where such photometric dynamics are critical.

缺乏光度动态。PointWorld 仅输出从 RGB-D 输入捕获的 3D点的位移，这些输入关注的是几何形状和物理交互，而非外观。虽然在渲染为点云时通常在视觉上是合理的，但如果希望推理由机器人动作引起的环境光度变化，例如灯光或屏幕的开关，仅此还不够。对于这类光度动态至关重要的任务，可能有必要将PointWorld与用于预测发射辐亮度的外观模型结合起来，例如 Gaussian Splatting [40] 或 Neural Radiance Fields [39] 中的模型。

Rigid-Body Robot Assumption. Robot embodiment is represented as a kinematic tree of rigid links, and we propagate a fixed set of robot surface points by forward kinematics. This ignores deformations of soft, tendon-driven, or compliant structures (e.g., fin-ray grippers) and non-rigid effects of the robot body. As a result, PointWorld reasons only about how the scene moves in response to a forecasted robot geometry, rather than how contact may reshape the robot itself. Extending the representation to include deformable links [169] would enable reasoning about how contact deforms the robot body and how those deformations, in turn, affect contact geometry.

刚体机器人假设。机器人embodiment被表示为由刚性连杆组成的运动学树，我们通过正向运动学传播一组固定的机器人表面点。这忽略了软体、腱驱动或柔顺结构（例如 fin-ray grippers）的形变，以及机器人本体的非刚性效应。因此，PointWorld 只能推理场景如何响应于预测的机器人几何体而运动，而不能推理接触如何重塑机器人本身。将该表示扩展为包含可变形连杆[169]，将能够推理接触如何使机器人本体发生形变，以及这些形变又如何反过来影响接触几何。

Actuation and Tracking Assumptions. Our formulation treats the robot trajectory as a known, fully realized sequence of joint configurations and uses forward kinematics to construct robot flows. As a result, PointWorld effectively models “what the environment does if the robot body follows this path,” rather than “whether and how the robot will actually realize this path” under a particular controller, actuation limits, or contact-induced tracking errors. This quasi-static, kinematic view of the robot action representation can break down for underactuated or compliant joints (e.g., tendondriven or compliant hand fingers), or when strong contacts, payloads, or controller changes induce large tracking errors. Extending the method to jointly reason about robot and scene dynamics is an important avenue for future work.

驱动与跟踪假设。我们的方法将机器人轨迹视为一个已知且完全实现的关节配置序列，并使用正向运动学来构建机器人流。因此，PointWorld 有效建模的是“如果机器人本体沿着这一路径运动，环境会如何变化”，而不是在特定控制器、驱动限制或接触引起的跟踪误差下，“机器人是否以及如何真正实现这一路径”。这种对机器人动作表征的准静态、运动学视角，在欠驱动或柔顺关节（例如，腱驱动或柔顺手指关节）情况下，或当强接触、负载变化或控制器变更引发较大跟踪误差时，可能会失效。将该方法扩展为对机器人与场景动力学进行联合推理，是未来工作中的一个重要方向。

Lack of Explicit Physics Priors. The current formulation is purely data-driven and does not incorporate explicit physics priors such as Newtonian mechanics or conservationlaw constraints, to provide a focused study on scaling 3D world models without priors of objectness and of their material and physical properties. Despite this, we observe that PointWorld captures many aspects of rigid, articulated, and deformable behavior from data alone. Incorporating physics-informed regularization or hybrid simulators [35] could further improve generalization and extrapolation, particularly in regimes that in-domain interaction data can be collected for accurate scene/object reconstruction, not only for their geometries but also for their physics parameters.

缺乏显式物理先验。当前方法完全由数据驱动，未引入显式物理先验，例如牛顿力学或守恒定律约束，其目的是聚焦研究在不依赖对象性先验以及材料和物理属性先验的情况下扩展3D 世界模型。尽管如此，我们观察到 PointWorld 仅从数据中就捕捉到了刚体、关节连接体和可变形体行为的许多方面。引入物理信息正则化或混合模拟器[35]可能会进一步提升泛化和外推能力，尤其是在那些能够收集域内交互数据以实现精确场景/对象重建的情形中，这种重建不仅包括其几何结构，还包括其物理参数。

A.2. DROID 3D Annotation Pipeline

A.2. DROID 3D 标注流程

DROID [7] is a large-scale robot manipulation dataset with human-teleoperated interactions collected with a wristmounted camera and two externally-mounted cameras (randomized over the left and right sides of the workspace). We use all DROID episodes for which raw camera streams are available, irrespective of task success or failure, since 3D world modeling depends only on the observed interactions rather than the task-specific outcomes of the manipulation. Each episode provides stereo RGB streams with groundtruth camera intrinsics for all three cameras, plus robot joint states and a known kinematic model of the robot. In this work, we use the recovered 3D scene flows from the two externally-mounted cameras. All data share a synchronized timestamp. We augment the robot model to include the Robotiq 2F-85 gripper and the custom camera mount used in the standardized DROID setup. For each scene, the pipeline aligns timestamps to a reference stream (binary search to nearest), downsamples by 2 , then runs: (i) dense metric depth; (ii) external-camera extrinsic refinement by aligning rendered robot mesh to observed depth; (iii) 2D tracking under workspace and robot masks; (iv) 3D trajectory reconstruction, slicing, and postprocessing. Note that we do not store robot point flows because those can be reconstructed efficiently at training/inference time given known robot URDF and the given joint actions.

DROID[7]是一个大规模机器人操作数据集，其中包含由人类遥操作采集的交互数据，使用一台腕部安装相机和两台外部安装相机（在工作空间的左侧和右侧随机布置）进行采集。我们使用所有具有原始相机流可用的DROIDepisode，而不考虑任务成功还是失败，因为3D世界建模仅依赖于观测到的交互，而非操作任务的特定结果。每个episode都提供三台相机的立体 RGB 流及其groundtruth 相机内参，以及机器人的关节状态和已知的机器人运动学模型。在这项工作中，我们使用从两台外部安装相机恢复得到的3D scene flow。所有数据共享同步时间戳。我们对机器人模型进行了扩展，将 Robotiq 2F-85 gripper 和标准化 DROID 设置中使用的自定义相机支架纳入其中。对于每个场景，该 pipeline会将时间戳对齐到参考流（使用二分查找寻找最近项)，按2倍下采样，然后依次执行：(i)稠密度量深度；(ii)通过将渲染的机器人 mesh 与观测深度对齐来细化外部相机外参；(iii) 在工作空间和机器人 mask 约束下进行 2D 跟踪；(iv)3D轨迹重建、切片和后处理。请注意，我们不存储机器人点流，因为在训练/Inference时，给定已知的机器人URDF和给定的关节动作，这些都可以被高效重建。

A.2.1. 深度估计

Per-view metric depth is obtained using a high-quality stereo estimator, FoundationStereo [9]. Note that unlike typical sensor depth, the estimated depth from FoundationStereo does not have a minimum depth threshold for valid depth perception. However, it is still observed that its estimated depth can be inaccurate for distant, especially texture-less, regions (such as walls). Therefore, depth values are sanitized by clamping to a trusted range $[0,4{]}_{\mathrm{m}}$ and producing a perpixel validity mask, which is also propagated to 3D points as a per-point depth-valid flag

使用高质量双目估计器FoundationStereo[9] 获取每个视角的度量深度。请注意，不同于典型的传感器深度，FoundationStereo估计得到的深度没有用于有效深度感知的最小深度阈值。然而，仍然观察到，对于较远的区域，尤其是无纹理区域（如墙面），其估计深度可能不准确。因此，会通过将深度值裁剪到可信范围[0,4m并生成逐像素有效性 mask 来对深度值进行净化，该 mask 也会被传播到 3D 点，作为每个点的深度有效标志。

A.2.2. Camera Pose Estimation

A.2.2. 相机位姿估计

We do not use dataset-provided extrinsics. Instead, we compute camera extrinsics using a two-stage procedure that leverages the accurate metric depth obtained from FoundationStereo discussed previously. First, we initialize the camera pose estimates using VGGT [1o]. Second, we refine all camera poses of the two external cameras from all timesteps jointly by aligning rendered robot geometry to observed depth using the recorded robot joint states.

我们不使用数据集提供的外参。相反，我们采用一个两阶段流程来计算相机外参，该流程利用了前文讨论的FoundationStereo所获得的精确度量深度。首先，我们使用 VGGT[10]初始化相机位姿估计。其次，我们利用记录的机器人关节状态，通过将渲染的机器人几何与观测深度对齐，对所有时间步中两个外部相机的全部相机位姿进行联合优化。

Camera Pose Initialization. Our goal is to estimate, for each externally mounted camera $C_{i,}$ a single rigid transform $T_{C_{i}B}\in \mathrm{SE}$ (3) that maps 3D points from the robot base frame Binto the camera frame and is fixed throughout a

相机位姿初始化。我们的目标是为每个外部安装的相机 $C_i$ 估计一个单一的刚性变换 $T_{C_{i}B}\in \mathrm{SE}$ (3)，该变换将 3D点从机器人基座坐标系B映射到相机坐标系中，并在整个过程中保持固定

DROID episode. We denote the robot base frame by $B,$ the wrist camera frame at time t by $W_t,$ and the external cameras by Ci. A multi-view pose estimator (VGGT [1o]) is applied to time-aligned images from the two external cameras and the wrist camera; it treats the first external camera at an initial timestep as the reference frame, and returns rigid transforms $T_{E_0{C}_i}$ and $T_{E_0{W}_i}$ that map points from each camera $C_i$ or $W_t$ into this reference frame $E_0$ . Independently, forward kinematics applied to the robot joint states provide the pose of the gripper in the base frame, $T_{G_t{B}_{\cdot }}$ For each physical robot (given by the recorded robot serial), we assume that the wrist camera is rigidly mounted relative to the end effector and reused across all episodes. We empirically found that this assumption is largely valid for robots used in DROID as the averaged transforms exhibit sub-millimeter alignment error with each other. Under this assumption, we can obtain a known gripperto-wrist transform $T_{WG}$ for each robot. Using this transform and the forward-kinematics model, we obtain a time-varying wrist pose in the base frame,

DROID 片段。我们将机器人基座坐标系记为 $B$ ，将时刻t的腕部相机坐标系记为 $W_t$ ，将外部相机记为 $C_i$ 。对来自两个外部相机和腕部相机的时间对齐图像应用多视角位姿估计器（VGGT[101）；它将初始时间步的第一个外部相机作为参考坐标系，并返回刚体变换 $T_{E_0{C}_i}$ 和 $T_{E_0{W}_t}$ ，将每个相机 $C_i$ 或 $W_t$ 中的点映射到该参考坐标系 $E_0$ 。独立地，对机器人关节状态应用正向运动学可得到夹爪在基座坐标系中的位姿，即 $T_{G_{t}B}$ 。对于每个物理机器人（由记录的机器人序列号给定），我们假设腕部相机相对于末端执行器是刚性安装的，并在所有片段中重复使用。我们通过实验发现，这一假设对于DROID 中使用的机器人基本成立，因为平均变换彼此之间表现出亚毫米级的 Alignment 误差。在这一假设下，我们可以为每个机器人获得一个已知的夹爪到腕部变换 $T_{WG}$ 。利用该变换和正向运动学模型，我们可以获得基座坐标系中随时间变化的腕部位姿，

$$T_{W_{t}B}=T_{WG}{T}_{G_{t}B}.$$

Combining this with the estimator’s transform between the wrist and the reference external camera yields per-frame estimates of the reference camera pose in the base frame,

将其与估计器给出的腕部与参考外部相机之间的变换结合起来，可得到参考相机在基座坐标系中的逐帧位姿估计，

$$T_{E_{0}B}^{(t)}=T_{E_0{W}_t}{T}_{W_{t}B},$$

which we average over all valid wrist frames to obtain a single $T_{E_0{B}_{\cdot }}$ For any other external camera $C_i$ , the estimator provides a relative transform $T_{E_0{C}_i}$ . We convert this to a base-frame extrinsic

我们对所有有效的腕部帧取平均，以获得单个 $T_{E_{0}B}$ 。对于任何其他外部相机 $C_i$ ，估计器会提供一个相对变换$T_{E_0{C}_i}$ 。我们将其转换为基座坐标系下的外参

$$T_{C_{i}B}=T_{C_i{E}_0}{T}_{E_{0}B},T_{C_i{E}_0}=T_{E_0{C}_i}^{-1},$$

so that all external cameras are expressed in the same robot base frame before the refinement stage

从而在细化阶段之前，所有外部相机都在同一个机器人基座坐标系中表示。

Camera Pose Refinement. Starting from the initialized base-frame extrinsics $T_{C_{i}B,}$ we jointly refine the poses of all external cameras using robot-depth reprojection. Let external cameras be indexed by i, timesteps by t, and let k index valid robot pixels after filtering (in front of the camera, within image bounds, deduplicated in the image plane, and with observed depth in the trusted range). For each camera i we optimize a small 6-DoF update on top of the initialization, parameterized by translation and rotation and scaled such that optimization can be done within a good numerical range. Given an

observed depth value $d_{i,t,k}^{\mathrm{obs}}$ at a valid robot pixel and the corresponding predicted depth $d_{i,t,k}^{\mathrm{pred}}$ obtained by projecting robot surface points from the base frame through the current extrinsics $T_{C_i{B}_{,}}$ we define the robot-depth reprojection loss

相机位姿细化。从初始化的基坐标系外参 $T_{C_{i}B}$ 出发，我们使用机器人深度重投影联合细化所有外部相机的位姿。设外部相机由i索引，时间步由t索引，表示过滤后有效的机器人像素索引（位于相机前方、在图像边界内、在图像平面中已去重，并且观测深度位于可信范围内）。对于每个相机i，我们在初始化结果的基础上优化一个小的6-DoF更新，使用平移和旋转进行参数化，并进行缩放，使优化能够在良好的数值范围内进行。给定有效机器人像素处的观测深度值 $d_{i,t,k}^{\mathrm{obs}}$ ，以及通过当前外参 $T_{C_{i}B}$ 将机器人表面点从基坐标系投影得到的对应预测深度 $d_{i,t,k}^{\mathrm{pred}}$ ，我们定义机器人深度重投影损失

$$L_{\mathrm{robot}-\mathrm{depth}}=\frac{1}{K}\sum _{i,t,k}\left|d_{i,t,k}^{\mathrm{cbs}}-d_{i,t,k}^{\mathrm{pred}}\right|$$

where the sum runs over all valid robot pixels across cameras and frames and K is their total count. We optimize the 6DoF updates for all external cameras jointly using a firstorder optimizer (1oo iterations, learning rate $10^{-3}\cdot$ ), and restrict supervision to robot pixels whose observed depth lies in a trusted range of [0.3, 2.0|m. To ensure reliable gradients, we further require that each camera-frame pair contributes at least 2,ooo valid robot points; frames that fail this criterion for any external camera are discarded before refinement. With these procedures, we can accurately label camera poses for around 6o% of all episodes recorded in DROID. Quantitative metrics are reported in the main text.

其中，求和遍历了跨相机和帧的所有有效机器人像素，K 是它们的总数。我们使用一个 firstorder Optimizer（100 次迭代，Learning Rate 为 $10^{-3}$ ）联合优化所有外部相机的6DoF更新，并将监督限制在观测深度位于可信范围[0.3,2.0]m内的机器人像素上。为确保梯度可靠，我们还要求每个相机-帧对至少贡献2,000 个有效机器人点；对于任外部相机不满足此标准的帧，会在细化前被丢弃。通过这些流程，我们能够为DROID中记录的约60%的所有episode 准确标注相机位姿。定量指标见正文。

A.2.3.3D 标注的基准指标

Depth Reprojection. For each scene and calibration variant, we render the robot mesh into each external camera using the candidate extrinsics and recorded joint states, discard depth outside [o.3,2.o] m or out-of-bounds robot pixels, and compute an L1 depth reprojection loss over valid robot pixels. The per-frame losses are averaged over valid pixels and frames to produce a point-weighted value reported per scene and then aggregated into cumulative curves.

深度重投影。对于每个场景和标定变体，我们使用候选外参和记录的关节状态，将机器人mesh 渲染到每个外部相机中，丢弃超出[0.3,2.0] m 范围的深度值或越界的机器人像素，并在有效机器人像素上计算L1 深度重投影损失。每帧损失会在有效像素和帧上取平均，以生成按点加权的数值，并按场景报告，随后聚合为累计曲线。

Two-view F1 @ 5/20 mm. For each external camera, we back-project valid depth into 3D using corresponding intrinsics/extrinsics, remove robot pixels, and crop to workspace. Given the resulting paired point clouds, we compute precision, recall, and F1 via symmetric nearest-neighbor matching: a point in cloud A (resp. B) is a true positive if its nearest neighbor in B (resp. A) lies within the threshold (5 mm or 2o mm ); otherwise it contributes to the falsepositive/falsenegative count. Metrics are accumulated over frames, normalized by the number of valid points per view, and then aggregated per scene into the cumulative counts shown in Figure 5. Scenes with missing depth/extrinsics for a given combo are omitted from evaluation for that combo.

双视图 F1 @5/20 mm• 对于每个外部相机，我们使用对应的内参/外参将有效深度反投影到 3D，去除机器人像素，并裁剪到工作空间。给定得到的成对点云，我们通过对称最近邻匹配来计算 precision、recall 和 F1：如果点云 A（或B)中的一个点在 B（或 A)中的最近邻位于阈值范围内(5 mm 或 20 mm)，则该点被视为 true positive；否则，它将计入false-positive/false-negative。指标在各帧上累积，按每个视图中的有效点数进行归一化，然后按场景聚合为Figure 5 中所示的累计计数。对于某个给定组合，如果某些场景缺少 depth/extrinsics，则这些场景在该组合的评估中被省略。

A.2.4. Occlusion-Aware Tracking in 3D

A.2.4.3D 中的遮挡感知跟踪

Given depth and camera poses, we herein describe how we can obtain 3D point flows. We describe the following in order: (i) filtering clips based on robot motion, (ii) tracking visible points within each retained clip, and (iii) postprocessing the resulting 3D trajectories.

给定深度和相机位姿，我们在此描述如何获得 3D 点流。我们将按以下顺序进行说明：(i)基于机器人运动对片段进行过滤，(ii)在每个保留的片段内跟踪可见点，以及(iii)对得到的3D 轨迹进行后处理。

Clip Filtering. We slice each episode into overlapping clips of length F = 16 frames with stride s = 1. Each clip covers approximately one second of the episode. Clips are retained if either the gripper changes state within the clip or endeffector motion exceeds thresholds. Thresholds depend on whether the gripper is predominantly open or closed within the clip: position/rotation thresholds are (0.005 m, 0.10rad) when open and (0.002 m, 0.05rad) when closed; either exceeding suffices to keep a clip, and any change in gripper state also keeps the clip.

片段过滤。我们将每个 episode 切分为长度为 F = 16 帧、步长为 s = 1 的重叠片段。每个片段大约覆盖该 episode的一秒钟。如果夹爪在片段内发生状态变化，或者末端执行器运动超过阈值，则保留该片段。阈值取决于夹爪在片段内主要处于打开还是闭合状态：打开时的位置/旋转阈值为(0.005 m,0.10rad），闭合时为(0.002 m,0.05rad）；任阈值被超过都足以保留片段，而夹爪状态的任何变化也会保留该片段。

Tracking. After obtaining depth and camera poses and slicing episodes into short clips, we perform dense tracking to obtain 3D scene flows on a per-clip basis. Clips cover roughly one-second windows and, depending on where they fall within a longer episode, may observe quite different regions of the workspace; tracking after clip selection ensures that each clip has its own consistent set of tracked regions and avoids mixing trajectories across widely separated time intervals. To improve efficiency and robustness, we perform tracking restricted to workspace and non-robot regions. To construct the workspace mask, we project a fixed 3D workspace volume to the image. To construct the non-robot mask, we render the robot’s URDF and project the mesh to the image plane. We then track 2D points using 2D point trackers (CoTracker3 [11]) on the masked regions only, producing dense 2D trajectories. The tracker also outputs a per-point visibility mask over time; we store this visibility for each 2D trajectory so that occluded points can later be excluded from supervision after lifting to 3D. Because PointWorld is trained on 3D point flows rather than image-plane trajectories, we lift each tracked 2D point to a 3D worldspace trajectory by back-projecting it with the corresponding depth, intrinsics, and extrinsics at each timestep. We next reconstruct 3D trajectories and apply cliplevel postprocessing. For each frame, valid depth, intrinsics, and extrinsics backproject tracked pixels to world-frame 3D points with RGB. Tracks across time yield temporally consistent per-point trajectories. We store per-camera trajectories to avoid mixing viewpoints prematurely and keep per-point visibility and depth-valid flags that are later used to mask supervision.

跟踪。在获得深度和相机位姿并将 episode 切分为短 clip后，我们执行稠密跟踪，以按 clip 为单位获得 3D sceneflow。clip 大致覆盖约一秒的时间窗口，并且根据它们在更长 episode中所处的位置，可能会观测到工作空间中截然不同的区域；在选择clip之后再进行跟踪，可确保每个clip都拥有自己一致的一组被跟踪区域，并避免在时间上相距较远的区间之间混合轨迹。为提高效率和鲁棒性，我们将跟踪限制在工作空间和非机器人区域内。为构建工作空间mask，我们将一个固定的3D工作空间体积投影到图像中。为构建非机器人mask，我们渲染机器人的 URDF，并将 mesh 投影到图像平面。随后，我们仅在这些 mask 区域内，使用 2D 点跟踪器（CoTracker3[111）对 2D 点进行跟踪，从而生成稠密的2D 轨迹。跟踪器还会输出随时间变化的逐点可见性mask；我们会为每条2D轨迹存储该可见性信息，以便在后续提升到3D后，将被遮挡的点从监督中排除。由于PointWorld 是基于3D点流而非图像平面轨迹进行训练的，因此我们通过在每个时间步使用对应的 depth、intrinsics 和 extrinsics 进行反投影，将每个被跟踪的 2D 点提升为 3Dworld-space 轨迹。接下来，我们重建 3D轨迹并应用clip 级别的后处理。对于每一帧，有效的 depth、intrinsics 和extrinsics 会将被跟踪的像素反投影为带有RGB 的 world-frame 3D点。跨时间的轨迹可生成时间上一致的逐点轨迹。我们按每个 camera 存储轨迹，以避免过早混合不同视角，并保留每个点的 visibility 和depth-valid 标志，这些标志随后将用于对监督进行掩码处理。

Postprocessing. To further improve the quality of the obtained 3D point flows, we apply two postprocessing steps: DBSCAN-based outlier removal and per-frame normal estimation. For each clip we first remove spatial outliers using multi-scale DBSCAN clustering [17o] across all external cameras: at each timestep, we run DBSCAN with radii ε ∈ {0.02, 0.05}m and minimum core size 5 , and mark point flows that are classified as outliers in more than 20% of frames as outliers to be discarded. From the remaining trajectories, we estimate normals per frame using local neighborhoods (up to k = 30 neighbors within radius o.1 m ) and orient them toward the camera, followed by a temporal consistency step that flips back-facing normals so that normal directions remain coherent over time.

后处理。为了进一步提升所获得的 3D 点流的质量，我们应用了两个后处理步骤：基于 DBSCAN的离群点移除和逐帧法线估计。对于每个片段，我们首先使用跨所有外部相机的多尺度DBSCAN聚类[170]来移除空间离群点：在每个时间步，我们以半径ε ∈ {0.02,0.05}m 和最小核心大小 5 运行 DBSCAN，并将那些在超过 20% 帧中被判定为离群点的点流标记为应丢弃的离群点。对于剩余的轨迹，我们使用局部邻域（半径 0.1 m 内最多 k = 30 个邻居）逐帧估计法线，并将其朝向相机定向，随后执行时间一致性步骤，将背向的法线翻转回来，以使法线方向在时间上保持一致。

A.3. BEHAVIOR-1K Data Generation

A.3. BEHAVIOR-1K 数据生成

BEHAVIOR-1K (BiK) [8] is a large-scale benchmark of everyday household activities in photorealistic simulation built on NVIDIA Isaac Sim. As part of the 2025 BEHAVIOR Challenge, it provides approximately 10,000 humanteleoperated episodes (average length ≈ 6.6 minutes) across 50 tasks executed by a bimanual mobile robot (Galexea R₁ Pro). We replay these episodes in the original simulator and attach three virtual cameras (left shoulder, right shoulder, and head) to extract short clips with meaningful interactions and dense 3D point flows, as detailed below.

BEHAVIOR-1K (B1K)[8] 是一个基于 NVIDIA Isaac Sim 构建的大规模日常家庭活动照片级真实感仿真基准。作为2025 BEHAVIOR Challenge 的一部分，它提供了约 10,000 段由人类遥操作的 episode（平均时长 ≈ 6.6 分钟），涵盖由双臂移动机器人（Galexea R1 Pro）执行的 50 项任务。我们在原始模拟器中回放这些 episode，并附加三个虚拟相机（左肩、右肩和头部），以提取包含有意义交互的短片段和稠密的3D点流，详见下文。

A.3.1 数据集回放

For each BEHAVIOR-1K episode, we replay it in the simulator using the recorded sequence of environment states and actions. To prevent physics leakage and adhere to the original demonstrations, we iterate over the stored trajectory and, at every recorded step, load the corresponding simulator state and advance the simulator once with the recorded action. At every such step, we also render three external RGB-D cameras mounted on the robot: a left and right shoulder camera attached near the base, and a head-mounted camera. All three cameras have ground-truth intrinsics and extrinsics, and produce per-pixel depth, surface normals, and per-link segmentation in addition to RGB. All extrinsics are recorded in the robot base frame in the first timestep of each clip.

对于每个 BEHAVIOR-1K episode，我们使用记录下来的环境状态和动作序列在模拟器中对其进行重放。为了防止physicsleakage并遵循原始演示，我们遍历存储的轨迹，并在每个记录的步骤加载对应的模拟器状态，然后使用记录的动作将模拟器推进一次。在每个这样的步骤中，我们还会渲染安装在机器人上的三个外部RGB-D相机：一个安装在底座附近的左肩和右肩相机，以及一个头戴式相机。这三个相机都具有 ground-truth intrinsics 和 extrinsics，除 RGB外，还会生成逐像素深度、表面法线以及按 link 划分的分割结果。所有 extrinsics 都记录在每个 clip 第一个 timestep的机器人 base frame 中。

A.3.2. Clip Filtering

A.3.2. 片段过滤

We aim to extract short clips of fixed length $F=11$ frames that contain meaningful interaction between the robot and the scene while discarding static or uninteresting intervals. To generate candidates, we slide an overlapping window of length F over each replayed episode at a fixed temporal stride; any window for which all external cameras have no visible, inworkspace scene objects-that is, no non-robot, non-ground meshes with valid depth inside the workspace bounds-is immediately discarded.

我们的目标是提取长度固定为F=11帧的短片段，这些片段包含机器人与场景之间有意义的交互，同时丢弃静态或无趣的时间区间。为生成候选片段，我们以固定的时间步长在每个回放episode上滑动一个长度为F的重叠窗口；如果某个窗口中所有外部相机都没有可见的工作空间内场景对象一即在工作空间边界内不存在具有有效深度的非机器人、非地面 mesh——则立即将其丢弃。

For each remaining candidate window, we maintain a set of motion indicators and contact indicators that are updated over the clip. Let $M_g$ denote the event that at least one arm’s end-effector exhibits sufficient translational or rotational motion over the clip or undergoes a change in gripper open/closed state, with thresholds that depend on whether the gripper is predominantly open or closed. Let $M_j$ denote the event that any non-base robot joint moves more than a fixed threshold over the clip. Using the object trajectories described in Section A.3.3, we define ${\hat{M}}_o$ as the event that at least one object moves more than an object-movement threshold in position or orientation relative to its pose at the first frame of the clip.

对于每个剩余的候选窗口，我们维护一组在整个clip 期间更新的运动指示器和接触指示器。 $\underset{⌣}{♠}{M}_g$ 表示如下事件：至少一个机械臂的末端执行器在该clip期间表现出足够的平移或旋转运动，或夹爪的开/闭状态发生变化；其中阈值取决于夹爪在大多数时间里是打开还是闭合。令 $M_j$ 表示如下事件：任意非基座机器人关节在该clip 期间的运动超过固定阈值。利用第 A.3.3 节中描述的物体轨迹，我们将 $\overline{M_o}$ 定义为如下事件：至少一个物体相对于该clip 第一帧中的位姿，其位置或朝向变化超过物体运动阈值。

From the ground-truth simulation state, we further construct contact indicators. Let $C_t$ denote the event that any trunk or arm link experiences a nonzero contact impulse

根据 ground-truth simulation state，我们进一步构建接触指示器。 $\underset{⌣}{⟺}{C}_t$ 表示如下事件：任意躯干或机械臂连杆经历了非零接触冲量

during the clip, and let $C_f$ denote the event that any gripper finger link experiences contact. Clips that contain large simulator-induced discontinuities (such as scene resets) are filtered internally before applying the following logical

criterion.

在该 clip 期间，并令 $C_f$ 表示如下事件：任意夹爪手指连杆发生接触。在应用以下逻辑判据之前，包含由 simulator引起的大幅不连续性（如场景重置）的clip会先在内部被过滤掉。

At the end of the horizon, a remaining candidate is accepted as a valid clip if and only if

在时间范围的末尾，当且仅当以下条件成立时，剩余候选片段才被接受为有效片段

$$\neg C_t\wedge ((M_o\wedge M_j)\vee (M_o\wedge C_f)\vee (\neg M_o\wedge M_g\wedge M_j)).$$

The term $\neg C_t$ discards clips that contain trunk or arm collisions. The first disjunct in equation 3 retains clips where object motion is causally associated with non-base joint motion. The second disjunct retains clips where object motion primarily arises through gripper-finger contacts, which covers behaviors such as pushing an object with only base motion rather than arm motion (e.g., pushing a door by moving the base). The third disjunct retains “negative” clips in which the robot moves but no objects move, providing supervision on background dynamics and free-space motions.

项 $\neg C_t$ 会丢弃包含躯干或手臂碰撞的片段。公式3中的第一个析取项保留对象运动与非底座关节运动存在因果关联的片段。第二个析取项保留对象运动主要由 gripper-finger接触产生的片段，这涵盖了诸如仅通过底座运动而非手臂运动来推动对象的行为（例如，通过移动底座来推门）。第三个析取项保留“负”片段，即机器人发生运动但没有任何对象发生运动，从而为背景动力学和自由空间运动提供监督。

A.3.3. 3D Point Flows from Simulation

A.3.3. 来自仿真的 3D 点流

For each accepted clip and each external camera, we construct a compact representation of 3D point flows that exploits three properties of the simulation setting: (i) the environment is composed of rigid objects decomposed into rigid links; (ii) we have access to ground-truth link-level instance segmentation in rendered images; and (iii) we can query the exact rigid trajectory of every link throughout the clip. At the first frame of the clip, we back-project depth for each visible link to obtain a set of surface points in that link’s local frame, together with associated colors and normals, after filtering out background and robot meshes and enforcing workspace bounds in the robot base frame at the clip start. We then record the time-varying poses of all visible links and cameras in this same clip-start robot base frame. This factorized representation, local link points plus per-link trajectories, allows us to reconstruct exact per-point 3D trajectories for any clip while remaining far more storage-efficient than storing dense point clouds at every frame. Note that while we access ground-truth simulator state for obtaining ground-truth 3D point flows, the simulator state is never exposed to the model.

对于每个被接受的 clip 和每个外部相机，我们构建一种紧凑的 3D 点流表示，利用了仿真环境的三个特性：(i)环境由刚体对象组成，并可分解为刚性连杆；(ii)我们可以访问渲染图像中的真实link级实例分割；以及(iii)我们可以查询整个clip 中每个 link的精确刚体轨迹。在 clip 的第一帧，我们对每个可见 link的深度进行反投影，以获得该link 局部坐标系中的一组表面点，以及关联的颜色和法线；在此之前，会滤除背景和机器人mesh，并在clip 开始时的机器人基座坐标系中施加工作空间边界约束。随后，我们在同一个clip起始机器人基座坐标系中记录所有可见link和相机的随时间变化的位姿。这种因子化表示——局部 link 点加上每个 link 的轨迹——使我们能够为任意 clip 重建精确的逐点 3D 轨迹，同时在存储效率上远优于为每一帧存储稠密点云。请注意，尽管我们会访问真实的模拟器状态来获取真实的3D点流，模拟器状态绝不会暴露给模型。

Operation 操作	Description
Camera subsampling	Sample randomized calibrated RGB-D views per timestep and concatenate their 3D points into a single scene cloud.
相机下采样 Bounds filtering	每个时间步采样随机且经过标定的 RGB-D 视图，并将其 3D 点拼接为单个场景 点云。 Retain only scene points that stay within a workspace cube (approx. $[ - 3 , 3 ] ^ { 3 } \mathrm { m } _ { \mathrm { ) f o r } }$ the entire clip, dropping particles that ever exit the bounds.
边界过滤	仅保留在整个片段期间始终位于工作空间立方体内（约为 $[ - 3 , 3 ] ^ { 3 } \mathrm { m }$ ）的场景 点，删除任何曾超出边界的粒子。
居中	先对首帧场景和机器人点云进行中心化处理。
Image resize 图像缩放	Downscale RGB-D images to 320 × 180. 将 RGB-D 图像下采样到 $3 2 0 \times 1 8 0$
Voxel downsampling	Voxel-grid sampling at 1.5 cm ; select one point per occupied voxel at t = 0, and apply the same indices to all timesteps.
体素下采样	以 1.5 cm 进行体素网格采样；在t = 0 处为每个已占用体素选择一个点，并将 相同的索引应用于所有时间步。
Multi-sphere cropping	Iteratively remove spheres of points far from the robot (up to three spheres, radii in [0.10,0.80]m with buffer 0.25 m ) until the scene fal elow the target budget.
多球裁剪	迭代移除远离机器人的点球体（最多三个球体，半径位于[0.10,0.80]m，缓 冲区为 0.25 m），直到场景低于目标预算。
Max scene / robot points	Randomly subsample scene points if more than 12ooo remain after cropping; robot
场景/机器人最大点数	points are capped at 50o by construction. 如果裁剪后场景点仍超过12000个，则随机下采样；机器人点在构造时被限制
Random yaw	为最多500个。
随机偏航	Uniform rotation about the vertical axis over [ —π, π ]. 绕垂直轴在[一π，π]范围内进行均匀旋转。
Uniform scaling	Isotropic scaling with factor sampled uniformly from [o.9, 1.1]
均匀缩放	各向同性缩放，缩放因子从 [0.9， 1.1] 中均匀采样。
Random reflection	With probability o.5 , reflect the scene and robot across either the x - or -axis.
随机反射	以0.5 的概率，使场景和机器人关于 x 轴或y轴进行反射。
Chromatic auto-contrast	Apply chromatic auto-contrast to RGB channels with probability o.2 and blend factor up to 0.2 .
色彩自动对比度	以0.2 的概率对 RGB 通道应用色彩自动对比度，混合因子最高为 0.2。
Chromatic translation	Add a global RGB offset with magnitude 2% with probability 0.95.
色彩平移	以 0.95 的概率添加幅度为 2%的全局 RGB 偏移。
Chromatic jitter	Add per-point RGB noise with standard deviation 2% with probability 0.95.
色彩抖动	以0.95 的概率为每个点添加标准差为 2%的 RGB 噪声。

Table 3: Table 3. Data Preprocessing and Augmentations.

表 3：表 3. 数据预处理和增强。

Point set 点集	Feature	Definition 定义
Robot	Position ptobot	3D coordinates of robot points over time.
机器人	$\{ \stackrel { \cdot } { \underline { { \breve { \mathbf { \delta } } } } } \equiv \stackrel { p _ { t , j } ^ { \mathrm { r o b c t } } } { \equiv } $	机器人点随时间变化的 3D 坐标。
Robot	Color crobt	Constant magenta color (1, 0, 1) indicating robot identity, shared across timesteps.
机器人	$\check { \Xi } \bar { \mathcal { Q } } \bar { \Xi } \stackrel { c _ { j } ^ { \mathrm { r o b o t } } } { \Xi }$	恒定的洋红色（1,0,1）表示机器人身份，在各 个时间步之间共享。
Robot	Normal ntobot	Surface normals of robot points from the known robot
机器人		URDF. 来自已知机器人 URDF的机器人点表面法向量。
		Scalar gripper open value per timestep, broadcast to all
Robot 机器人	Gripper openness ğt 夹爪开合度ğt	robot points.
		每个时间步的标量夹爪打开值，广播到所有机器人 点。 Per-poiņt velocity from mid-point finite differences
Robot 机器人	Velocity bot	across time.
	$\mathrm { i } \overline { { \mathbb { R } } } \vec { \mathbb { E } } ^ { \mathrm { ~ r c b o t ~ } }$	上通过中点有限差分计算的每点 速度。
Robot 机器人	Acceleration aobot	Per-point acceleration from mid-point finite differences over vtt across time.
	$\hbar \underset { \mathrm { \# } } { \mathrm { \tt m i } } \frac { 1 } { 2 } \underset { \mathrm { } } { \mathrm { \# } } a _ { t , j } ^ { \mathrm { r o b o t } }$	的中点有限差分计算每个点的 加速度。
Scene 场景	Position x0,i	3D coordinates of scene points at the first frame after preprocessing.
	位置x0.i	预处理后第一帧中场景点的 3D 坐标。
Scene	Color coc n e	RGB color of scene points at the first frame.
场景	$\gnapproxeq c _ { 0 , i } ^ { \mathrm { s c e n e } }$	第一帧中场景点的 RGB 颜色。
Scene	Normal nscene	(Estimated) surface normals of scene points at the first frame.
场景	$\therefore \pm \underline { { { 4 } } } \geq n _ { 0 , i } ^ { \mathrm { s c e n e } }$	第一帧中场景点的（估计）表面法线。
Scene	Gripper openness sequence 90:T-1	Sequence of gripper openness values over the context and prediction horizon, broadcast to every scene point.
场景	夹爪开合序列 .g0:T-1	在上下文与预测范围内的夹爪开合值序列，广播到 每个场景点。
Scene	Distance-to-robot d0:T-1,i	For each timestep, distance from the first-frame position of scene point i to the closest robot point, stacked across time.
场景	到机器人距离 do:T-1,i	对于每个时间步，场景点i相对于第一帧位置到最 近机器人点的距离，并按时间堆叠。

表 4：表 4. 每点输入特征。

A.4. Model Training Details

A.4. 模型训练细节

Data Preprocessing and Augmentations. Here we describe the data preprocessing and augmentations used in our experiments. Each training sample fuses calibrated RGB-D views before passing through workspace filtering, centering, and deterministic voxel sampling with multi-sphere cropping so that the fused cloud respects fixed budgets for scene and robot points. Geometric augmentations consist of random yaw rotations, isotropic scaling, and reflections; photometric augmentations apply auto-contrast, global color shifts, and per-point jitter to the RGB channels. For evaluation, we ensure the pipeline to be fully deterministic and disable all augmentations.

数据预处理与增强。这里我们介绍实验中使用的数据预处理与增强方法。每个训练样本都会先融合经过标定的RGB-D视图，然后依次经过工作空间过滤、居中处理以及带有多球裁剪的确定性体素采样，从而使融合后的点云满足场景点和机器人点的固定预算。几何增强包括随机偏航旋转、各向同性缩放和反射；光度增强则对RGB通道应用自动对比度、全局颜色偏移以及逐点抖动。对于评估，我们确保整个流水线完全确定性，并禁用所有增强。

Per-Point Input Features. Here we describe the per-point features produced as part of the data pipeline, prior to their consumption by the model. Details are listed in Table 4. Robot features stack positions, surface normals, a gripper scalar, and velocity and acceleration terms:

逐点输入特征。这里我们介绍数据流水线中生成的逐点特征，这些特征会在被模型使用之前产生。详细信息见表4。机器人特征将位置、表面法线、夹爪标量以及速度和加速度项堆叠在一起：

$${\varphi }_{t,j}^{\mathrm{robot}}=\left[p_{t,j}^{\mathrm{robot}},c_j^{\mathrm{robot}},n_{t,j}^{\mathrm{robot}},{\tilde{g}}_t,v_{t,j}^{\mathrm{robot}},a_{t,j}^{\mathrm{robot}}\right]$$

where $p_{t,j}^{\mathrm{robot}}$ and $n_{t,j}^{\mathrm{robot}}$ are position and normal, $c_j^{\mathrm{robot}}$ is a fixed color tag, ${\tilde{g}}_t$ is the normalized gripper openness, and ( $v_{t,j}^{\mathrm{robot}},a_{t,j}^{\mathrm{robot}})$ come from mid-point finite differences across the horizon with zero-velocity boundary conditions at the first and last timestep, i.e., we assume the robot is stationary at the boundaries of each model window. Scene features are computed for only the first frame $t=0$ and combine positions, colors, estimated normals, gripper openness sequence, and distances to the nearest robot point:

其中， $p_{t,j}^{\mathrm{robot}}$ 和 $n_{t,j}^{\mathrm{robct}}$ 分别是位置和法向量， $c_j^{\mathrm{robot}}$ 是固定颜色标签， ${\tilde{g}}_t$ 是归一化的夹爪张开度，而（$v_{t,j}^{\mathrm{robot}},a_{t,j}^{\mathrm{robot}}{}_{\}}$ 来自在时间范围内采用零速度边界条件的一阶中点有限差分，即我们假设机器人在每个模型窗口的边界处处于静止状态。场景特征仅针对第一帧 $t=0$ 计算，并结合了位置、颜色、估计法向量、夹爪张开度序列，以及到最近机器人点的距离：

$${\varphi }_i^{\mathrm{scene}}=[x_{0,i},c_{0,i}^{\mathrm{scene}},n_{0,i}^{\mathrm{scene}},g_{0:T-1},d_{0:T-1,i}],$$

where $c_{0,i}^{\mathrm{scene}}$ and $n_{0,i}^{\mathrm{scene}}$ are the RGB color and normal at the first frame, $g_{0:T-1}\in {\mathbb{R}}^T$ is the sequence of gripper openness values over the context-plus-prediction horizon broadcast to every scene point, and $d_{t,i}$ is the distance from scene point i to the closest robot point at timestep t,

其中， $c_{0,i}^{\mathrm{scene}}$ 和 $n_{0,i}^{\mathrm{scene}}$ 是第一帧中的 RGB 颜色和法向量， $g_{0:T-1}\in {\mathbb{R}}^T$ 是在上下文加预测时间范围内的夹爪张开度值序列，并广播到每个场景点， $d_{t,i}$ 是场景点i到时间步t时最近机器人点的距离，

$$d_{t,i}=\underset{j}{\min }{\Vert x_{0,i}-r_{t,j}\Vert }_2,d_{0:T-1,i}\in {\mathbb{R}}^T.$$

The distance field $d_{0:T-1,i}$ is obtained from nearest-neighbor queries between first-frame scene points and robot points at every timestep.

距离场 $d_{0:T-1,i}$ 通过对第一帧场景点与每个时间步的机器人点之间执行最近邻查询获得。

3D Scene Featurization with DINOv3. Prior to the point cloud backbone, PointWorld uses a 2D scene encoder based on DINOv3 ViT-L/16 by aggregating its multi-layer features. To featurize the 3D scene points with the imagebased encoder, the first-frame scene coordinates $x_{0,i}\in {\mathbb{R}}^3$ are projected into each chosen camera. For camera c

使用 DINOv3 进行 3D 场景特征化。在点云骨于网络之前，PointWorld 通过聚合 DINOv3 ViT-L/16 的多层特征，使用一个基于2D场景编码器。为了使用基于图像的编码器对3D场景点进行特征化，首先将第一帧场景坐标 $x_{0,i}\in {\mathbb{R}}^3$ 投影到每个选定的相机中。对于相机c

$$withintrinsics.K_{c}indextrinsics(R_c,t_c)weform{\tilde{u}}_{c,i}=K_c(R_c{x}_{0,i}+t_c)\mathrm{and}\mathrm{obtain}\mathrm{the}\mathrm{pixel}\mathrm{corrdinate}\mathrm{as}$$

其内参为 $K_c$ 、外参为 $(R_c,t_c)$ ，我们构造 ${\tilde{u}}_{c,i}=K_c\left(R_c{x}_{0,i}+t_c\right)$ 并得到像素坐标为

$$\begin{array}{r}{\mathbf{u}}_{\mathbf{c},\mathbf{i}}={\left[{\tilde{u}}_{c,i}^{(1)}/{\tilde{u}}_{c,i}^{(3)},{\tilde{u}}_{c,i}^{(2)}/{\tilde{u}}_{c,i}^{(3)}\right]}^{\top }.\end{array}$$

The same intrinsics and extrinsics support a depthconsistency mask that compares the projected depth of each point with the given depth image, so only views whose discrepancy is below a few millimeters contribute features.

相同的内参和外参还支持一个深度一致性掩码，用于将每个点的投影深度与给定的深度图像进行比较，因此只有偏差低于几毫米的视图才会贡献特征。

For each visible point-camera pair $(i,c)$ , DINOv3 patch tokens are sampled at $u_{c,i}$ by bilinear interpolation on the patchtoken grid, using a coordinate mapping that aligns token centers with pixel centers. Let $f_{c,i}\in {\mathbb{R}}^{D_{\mathrm{patch}}}$ denote the concatenated multi-layer patch feature for point ¿ in camera ${}^{c,}$ and let m^c,i∈{0,1}\mathbf { \hat { \it m } } _ { c , i } \in \{ 0 , 1 \}m^c,i​∈{0,1} indicate visibility and depth consistency. Features are aggregated across cameras by averaging over the contributing views,

对于每个可见的点-相机对 $(i,c)$ ，在 $u_{c,i}$ 处通过在 patch-Token 网格上进行双线性插值来采样 DINOv3 patchTokens，并使用一种将 Token中心与像素中心对齐的坐标映射。 $\underset{⌣}{⟺}{f}_{c,i}\in {\mathbb{R}}^{D_{\mathrm{patch}}}$ 表示相机 c 中点i 的拼接多层patch 特征，   ⟺  ⌣mc,i∈{0,1}\underset { \smile } { \iff } m _ { c , i } \in \{ 0 , 1 \}⌣⟺​mc,i​∈{0,1} 表示可见性和深度一致性。通过对所有有贡献的视图取平均，在相机之间聚合特征，

$$f_i=\frac{1}{\max \left(1,{\sum }_c{m}_{c,i}\right)}\sum _c{m}_{c,i}{f}_{c,i}$$

The averaged token is mapped to the backbone width (256 channels) by a learned projection and fused with a separately projected version of the raw scene features from Table 4; layer normalization is applied to each stream before concatenation, and a final linear layer produces the per-point embedding supplied to the dynamics backbone. The 2D encoder is kept frozen during training and evaluation

平均Token 通过一个学习得到的投影映射到主于网络宽度（256 个通道），并与表4 中原始场景特征经单独投影后的版本进行融合；在拼接之前，对每个流都应用layer normalization，最后通过一个线性层生成提供给动态主干网络的逐点Embedding。2D 编码器在训练和评估期间保持冻结。

Visibility-Aware Supervision. For real-world domains, we restrict training on 3D point trajectories to correspondences that are both geometrically and photometrically reliable. The annotation pipeline supplies per-point visibility (from 2D trackers [11] on real data and from ground-truth simulator state on synthetic data) together with per-pixel depth-validity; both signals are propagated to the lifted 3D trajectories and stored as binary flags per scene point and timestep. During training, we construct a per-timestep mask that selects scene points that are visible in the camera view and have valid depth support. The weighted dynamics objective from Section 3 is then evaluated only over this subset of correspondences (points filtered out receive zero loss weight), so that gradients are driven by non-occluded, depthvalid 3D flows. For simulation domains, 可见性感知监督。对于真实世界域，我们将3D点轨迹上的训练限制在同时具备几何可靠性和光度可靠性的对应关系 上。标注流水线为每个点提供可见性信息（真实数据来自 2D trackers [11]，合成数据来自 ground-truth simulaton state），并同时提供每像素的深度有效性；这两种信号都会传播到提升后的3D轨迹中，并以每个场景点、每个时间步 的二进制标志存储。在训练期间，我们构建一个逐时间步的mask，用于选择在相机视图中可见且具有有效深度支持的 场景点。随后，仅在这一对应关系子集上评估第3节中的加权动力学目标（被过滤掉的点将获得零损失权重），从而使 梯度由未被遮挡且深度有效的3Dflow驱动。对于仿真域，由于轨迹和深度是无噪声的，且遮挡被显式建模，因此所有 场景点都会对损失作出贡献。

Training Configuration. We train the 1 B -parameter version of PoINTWORLD on both BEHAVIOR-1K and DROID, with configuration and PointTransformerV3 (PTv3) design summarized in Table 5 and Table 6, respectively. For the main experiments in the paper, training configuration and PTv3 design are summarized in Table 7 and Table 8.

训练配置。我们在 BEHAVIOR-1K 和 DROID 上训练了 PoINTWORLD 的 1 B -parameter 版本，其配置和PointTransformerV3(PTv3)设计分别总结于表 5 和表 6。对于论文中的主要实验，训练配置和 PTv3 设计总结于表 7和表8。

Aleatoric Uncertainty on Simulation Data. When training on mixtures of real and simulated domains, directly learning perpoint uncertainty everywhere can collapse the model because simulated trajectories are noise-free. In the objective from Section 3 the residual term for point ¿ at step k is weighted as $w_{k,i}{\rho }_{\delta }\left({\hat{\mathbf{P}}}_{t+k,i}-{\mathbf{P}}_{t+k,i}\right){\mathrm{e}}^{-s_{k,i}}+w_{k,i}{s}_{k,i}$ . For vanishing residuals (typical in simulation), minimizing the loss drives $s_{k,i}$ toward log ${\rho }_{\delta }(\cdot )$ . Since ${\rho }_{\delta }(\cdot )$ approaches zero, the optimal $s_{k,i}$ becomes a large negative number, $i.e.$ , the predicted variance ${\sigma }_{k,i}^2=\exp (s_{k,i})$ collapses toward zero. As a consequence, ${\mathrm{e}}^{-s_{k,i}}$ explodes, so any small numerical discrepancy in simulated residuals produces excessively large gradients that overwhelm the real-data contributions and destabilize joint training. To stabilize training, we treat aleatoric variance on simulation domains as a constant: the uncertainty head is trained normally on real data, but for simulated domains its logvariance is replaced by a batchwise constant that matches the average variance observed on real samples (or a small fixed value when only simulation is present). This preserves heteroscedastic weighting where it is most useful (real, noisy supervision) while preventing the model from exploiting the uncertainty head to down-weight clean simulated gradients.

模拟数据上的偶然不确定性。在真实域与模拟域的混合数据上训练时，若在所有位置直接学习逐点不确定性，模型可能会发生塌缩，因为模拟轨迹是无噪声的。在第3节的目标函数中，步骤处点i的残差项的权重为

$w_{k,i}{\rho }_{\delta }\left({\hat{\mathbf{P}}}_{t+k,i}-{\mathbf{P}}_{t+k,i}\right){\mathrm{e}}^{-s_{k,i}}+w_{k,i}{s}_{k,i}$ 。当残差趋于消失时（这在模拟中很常见），最小化损失会驱动 $s_{k,i}$ 向$\log {\rho }_{\delta }(\cdot )$ 靠近。由于 ${\rho }_{\delta }(\cdot )$ 趋近于零，最优的 $s_{k,i}$ 会变成一个很大的负数，也就是说，预测方差 ${\sigma }_{k,i}^2=\exp {\left(s_{k,i}\right)}_{\hat{\mathbf{\Gamma }}}{\mathbf{\Gamma }}_{\hat{\mathbf{\Gamma }}}$ 塌缩至接近零。结果， ${\mathrm{e}}^{-s_{k,i}}$ 会爆炸，因此模拟残差中的任何微小数值偏差都会产生过大的梯度，淹没真实数据的贡献并破坏联合训练的稳定性。为稳定训练，我们将模拟域上的偶然方差视为常数：不确定性头在真实数据上正常训练，但对于模拟域，其对数方差会被替换为一个按batch计算的常数，该常数与真实样本上观测到的平均方差相匹配（或者在仅存在模拟数据时使用一个较小的固定值）。这保留了异方差加权在其最有用之处（真实、含噪监督）上的作用，同时防止模型利用 uncertainty head 来降低干净的模拟梯度的权重。

Setting	Value
设置	值
Optimizer Learning rate	AdamW $\overline { { 1 \times 1 0 ^ { - 4 } } }$
Epochs
Weight decay	$\textstyle { \frac { 3 0 0 } { 1 0 ^ { - 2 } } }$
Global batch size	1920 sequences
全局批量大小 Gradient clipping	1920 个序列 Global $\ell _ { 2 }$ norm capped at 5
梯度裁剪	全局 $\ell _ { 2 }$ 范数上限为 5
Loss	Huber loss with δ = 5.0 with movement weighting and aleatoric uncertainty
损失	带有δ = 5.0、运动加权和 aleatoric uncertainty 的 Huber loss
Prediction horizon	10 steps
预测时域	10步
Training GPUs	128 NVIDIA H100 GPUs
训练 GPU	128 个 NVIDIA H100 GPU
Training time	20 days
训练时间	20天
Component组件	Values取值
Grid size网格大小	1.5 cm1.5 厘米
Encoder depth编码器深度	(4, 4, 8, 8, 12, 12, 4)
Encoder channels编码器通道数	(256, 384, 384, 512, 512, 768, 1024)
Encoder heads编码器头数	(8, 12, 12, 16, 16, 24, 32)
Encoder stride编码器步幅	(1, 2, 2, 2, 2, 2, 2)
Encoder patch size编码器 patch 大小	(256, 256, 256, 256, 256, 256, 256)
Decoder depth解码器深度	(4, 4, 4, 4, 4, 4)
Decoder channels解码器通道数	(256, 384, 384, 512, 512, 768)
Decoder heads解码器头数	(8, 12, 12, 16, 16, 24)
Decoder patch size解码器 patch 大小	(256, 256, 256, 256, 256, 256)
Setting	Value
设置	值
Optimizer	AdamW
Learning rate	$\overline { { 1 \times 1 0 ^ { - 4 } } }$
Epochs	200
Weight decay
权重衰减 Global batch size	$1 0 ^ { - 2 }$ 176 sequences
全局 batch size	176 个序列
Gradient clipping	Global $\overline { { \ell _ { 2 } } }$ norm capped at 5
梯度裁剪 Loss	全局 $\ell _ { 2 }$ 范数上限为 5 Huber loss with δ = 5.0 with movement weighting and aleatoric uncertainty
损失	带有 $\delta = 5 . 0$ 、运动加权和偶然不确定性的 Huber 损失
Prediction horizon	10 steps
预测视界	10步
Training GPUs	8 NVIDIA H100 GPUs
训练 GPU	8 个 NVIDIA H100 GPU
Training time	7 days
训练时间	7天

Table 5: Table 5. Training Configuration for PointWorld-1B.
Table 7: Table 7. Training Configuration for PointWorld-411M.
表 5：PointWorld-1B 的训练配置。

表6：用于 PointWorld-1B 的 PointTransformerV3 (PTv3） 架构。编码器和解码器配置按阶段排序。

Table 6: Table 6. PointTransformerV3 (PTv3) Architecture for PointWorld-1B. Encoder and decoder configurations are ordered by stage.

表7：PointWorld-411M 的训练配置。

Component组件	Values值
Grid size网格大小	1.5 cm1.5 厘米
Encoder depth编码器深度	(4, 4, 4, 8, 8, 12, 4)
Encoder channels编码器通道数	(256, 256, 256, 384, 384, 512, 768)
Encoder heads编码器头数	(4, 4, 4, 8, 8, 16, 24)
Encoder stride编码器步幅	(1, 2, 2, 2, 2, 2, 2)
Encoder patch size编码器 patch size	(256, 256, 256, 256, 256, 256, 256)
Decoder depth解码器深度	(2, 2, 2, 2, 2, 2)
Decoder channels解码器通道数	(256, 256, 256, 384, 384, 512)
Decoder heads解码器头数	(4,4,4,8,8,16)
Decoder patch size解码器 patch size	(256, 256, 256, 256, 256, 256)

Table 8. PointTransformerV3 (PTv3) Architecture for PointWorld-411M. Encoder and decoder configurations are ordered by stage.

A.5. DROID Evaluation Protocol

A.5. DROID 评估协议

Following the protocol in Section 5, we measure persequence losses and aggregate them into dataset-level summaries. Alongside the overall per-point, per-timestep l2 distance, we report the same metric separately on moved and static points, since movers form a minority of the points but dominate perceived quality. We use these metrics directly for simulation data (BEHAVIOR-1K) since they are noiseless. However, for real-world data (DROID), we further apply expert confidence filtering to obtain filtered metrics. Details are described below.

遵循第5节中的协议，我们测量每个序列的损失，并将其汇总为数据集级别的摘要。除了总体的每点、每时间步 ${\ell }_2$ 距离之外，我们还分别在移动点和静态点上报告相同指标，因为移动物体只占点中的少数，但却主导了感知质量。对于模拟数据（BEHAVIOR-1K），由于其无噪声，我们直接使用这些指标。然而，对于真实世界数据（DROID），我们进一步应用专家置信度过滤以获得过滤后的指标。具体细节如下。

Expert Confidence Filtering. Although the mover- ${\ell }_2$ score highlights the behavior we care about most, imperfect real-world annotations mean that a noticeable fraction of mover points correspond to outliers or background clutter, because those points tend to have large movement magnitudes due to unstable depth estimation. During training, the aleatoric uncertainty regularization down-weights those points, but at evaluation time different models produce their own uncertainty predictions, making comparisons challenging. To obtain a model-agnostic notion of trustworthy ground-truth, we train an expert model only on the evaluation split with uncertainty predictions, convert the predicted variance into a per-point confidence in $[0,1]$ , and threshold this per-timestep per-point confidence at the o.8 quantile over all points. Points below this confidence are treated as lowconfidence outliers. We voxelize these low-confidence sets in world coordinates using the training grid size g and cache the resulting voxel grids for each evaluation sample so that the same filtering masks can be reused across subsequent evaluation runs and model variants. Note that the expert model is only used to compute the low-confidence voxel grids and does not share any training data or parameters with any evaluated models.

专家置信度过滤。尽管 $\mathsf{mover}-{\ell }_2$ 分数突出了我们最关心的行为，但由于现实世界中的标注并不完美，相当一部分mover点实际上对应于离群点或背景杂波，因为这些点往往会由于深度估计不稳定而具有较大的运动幅度。在训练期间，aleatoric uncertainty regularization会降低这些点的权重；但在评估时，不同模型会产生各自的不确定性预测这使得比较变得具有挑战性。为了获得一种与模型无关的、可信的ground-truth 概念，我们仅在带有不确定性预测的评估划分上训练一个专家模型，将预测的方差转换为[0,1中的逐点置信度，并将这种按时间步、按点的置信度阈值设为所有点上的0.8分位数。低于该置信度的点会被视为低置信度离群点。我们使用训练网格大小9在世界坐标中对这些低置信度集合进行体素化，并为每个评估样本缓存生成的体素网格，以便在后续的评估运行和不同模型变体之间复用相同的过滤掩码。请注意，expert model仅用于计算低置信度体素网格，并且不与任何被评估模型共享任何训练数据或参数。

Filtered Evaluation. To evaluate a model, for each sample, we first reconstruct world-coordinate voxel indices of scene points and then determine whether each point lies inside a precomputed low-confidence voxel. This yields a binary filter mask so that only high-confidence points at highconfidence timesteps (deemed by the shared expert model) contribute to the filtered metrics.

过滤评估。为了评估一个模型，对于每个样本，我们首先重建场景点的世界坐标体素索引，然后判断每个点是否位于预先计算的低置信度体素内部。这样会生成一个二值过滤 mask，使得只有处于高置信度 timestep的高置信度点（由共享的 expert model 判定）会对过滤后的指标有贡献。

Mover/Static Splits. Let ${\hat{P}}_{t,i}$ and $P_{t,i}$ denote predicted and ground-truth 3D positions. We compute per-point error $e_{t,i}={{\bar{P}}_{t,i}-P_{t,i}}_2$ 2 and report

Mover/Static 划分。令 ${\hat{P}}_{t,i}$ 和 $P_{t,i}$ 分别表示预测的和真实的3D位置。我们计算每点误差 $e_{t,i}={{\hat{P}}_{t,i}-P_{t,i}}_2$ 并报告

$${\ell }_2=\frac{1}{T}\sum _t\frac{1}{\left|V_t\right|}\sum _{i\in V_t}{e}_{t,i},$$

where $V_t$ denotes valid points at timestep t. Mover- ${\ell }_2$ and static- ${\ell }_2$ use the same definition but restrict $V_t$ to moved or其中， $V_t$ 表示 timestep t 的有效点。 $\mathsf{Mover}-{\ell }_2$ 和 $\mathsf{static}-{\ell }_2$ 使用相同的定义，但将 $V_t$ 限制为已移动或static points identified from the ground-truth trajectories via a small displacement threshold.

通过小位移阈值从真实轨迹中识别出的静态点。

A.6. 真实机器人实验细节

Real-robot experiments use a 7-DoF Franka arm equipped with a 3D-printed fin ray gripper [171, 172]. The robot is mounted on a wheeled, non-motorized base for in-thewild deployments. Since PointWorld is trained on data containing Robotiq 2F-85 and Galexea R₁ Pro grippers, the fin ray gripper geometry remains fully unseen by the model, illustrating cross-gripper geometry generalization. Since the pipeline predicts 6-DoF end-effector poses, we run position control at 20 Hz : each predicted target pose is clipped to a predefined workspace, then linearly interpolated from the current pose with steps of 5 mm in translation and 1° in rotation. For every interpolated pose, inverse kinematics (PyBullet solver) produces target joint positions that are tracked with the Deoxys joint-impedance controller [173]. We use one RealSense D435 mounted on the left shoulder of the robot to capture RGB and the stereo IR images. The stereo IR images are used to estimate the metric depth using FoundationStereo [9], given known baseline and camera intrinsics.

真实机器人实验使用一台 7-DoF 的 Franka 机械臂，配备一个 3D 打印的 fin ray gripper [171， 172]。该机器人安装在个带轮、无动力的底座上，以便进行野外部署。由于 PointWorld 是在包含 Robotiq 2F-85 和 Galexea R1 Progripper 的数据上训练的，因此 fin ray gripper 的几何形状对模型来说是完全未见过的，这说明了其跨 gripper 几何形状的泛化能力。由于该流程预测的是6-DoF末端执行器位姿，我们以20 Hz运行位置控制：将每个预测的目标位姿裁剪到预定义工作空间内，然后从当前位姿进行线性插值，平移步长为5 mm，旋转步长为1°。对于每个插值位姿，逆运动学（PyBullet solver）会生成目标关节位置，并由 Deoxys关节阻抗控制器[173]进行跟踪。我们使用一台安装在机器人左肩上的 ReaISense D435 来采集 RGB 和双目 IR 图像。双目 IR 图像在已知基线和相机内参的条件下，使用FoundationStereo [9] 来估计度量深度。

A.6.1. Model-Based Planning

A.6.1. 基于模型的规划

In this work, we use a single pre-trained PointWorld as the dynamics model. The model is pre-trained jointly on both realworld and simulated data. We use a samplingbased model-predictive path integral (MPPI) controller that samples action sequences around a zero-initialized nominal using cubic splines with ${\hat{\mathbf{n}}}_{\mathrm{knots}}=\ddot{4}$ and degree 3 . Noise scales are scheduled between ${\sigma }_{\min }=0.05$ and ${\sigma }_{\max }=0.50$ (in normalized action units). Each refinement iteration draws 256 samples; importance weights use temperature $\beta \stackrel{\cdot }{=}0.05$ , and the nominal is updated with an exponential moving average $(EMA=0.9)$ . We perform planning for 3o steps into the future, and the horizon is chunked to match the prediction window of the dynamics model. We perform 2o refinement iterations. The planning time is typically around a few seconds depending on task complexity and specific model size variant used. While we do not perform replanning in this work, replanning can be done at a real-time frequency by warm-starting from the previous nominal trajectory.

在这项工作中，我们使用单个预训练的PointWorld 作为动力学模型。该模型在真实世界和模拟数据上联合进行预训练。我们使用一种基于采样的模型预测路径积分（MPPI）控制器，该控制器使用带有 $n_{\mathrm{knots}}=4$ 和 degree 3 的cubic splines，在以零初始化的 nominal周围对动作序列进行采样。噪声尺度在 ${\sigma }_{\min }=0.05$ 和 ${\sigma }_{\max }\bar{\mathrm{\Gamma }}=0.50$ 之间调度（以归一化动作单位表示）。每次细化迭代抽取 256 个样本；重要性权重使用 Temperature $\beta =0.05$ ，并且使用指数移动平均 $(EMA=0.9)$ 更新nominal。我们对未来30 个时间步进行规划，并将规划时域分块，以匹配动力学模型的预测窗口。我们执行20次细化迭代。规划时间通常在几秒左右，具体取决于任务复杂度以及所使用的特定模型尺寸变体。虽然我们在这项工作中不执行重新规划，但可以通过以上一次的 nominal trajectory 作为热启动，以实时频率进行重新规划。

A.6.2. Task Specification

A.6.2. 任务规格说明

We specify tasks through a GUI tool that allows users to select object masks using SAM2 [174] and specify target positions in the world frame. We find this simple objective as a unified interface for specifying diverse real-world tasks including rigid pushing, deformable manipulation, articulatedobject interaction, and tool use. Following common practices in reward design [175], we add a mild end-effector proximity term to encourage exploration in the object’s neighborhood without prescribing a particular contact pattern. For deformable and tool-use tasks we begin from a pre-grasped configuration so that subsequent motion primarily probes deformable dynamics and object-object contacts. All tasks

我们通过一个 GUI工具来指定任务，该工具允许用户使用 SAM2[174] 选择对象 mask，并在世界坐标系中指定目标位置。我们发现，这种简单的目标可作为一个统一接口，用于指定多种现实世界任务，包括刚体推动、可变形物体操作、关节物体交互以及工具使用。遵循奖励设计中的常见做法[175]，我们加入了一个温和的末端执行器接近项，以鼓励在物体邻域内进行探索，而不预设特定的接触模式。对于可变形和工具使用任务，我们从预抓取配置开始，以便后续运动主要探测可变形动力学和物体-物体接触。所有任务

share same control regularization comprising SE(3) pathlength penalties and IK-based reachability residual.

共享相同的控制正则化，包括 SE(3)路径长度惩罚和基于IK的可达性残差。

A.6.3. Evaluation Protocol

A.6.3. 评估协议

We conduct evaluations on the following tasks: rigid pushing (tissue box, book), deformable manipulation (scarf fold, pillow place), articulated manipulation (microwave open, drawer close), and tool use (duster sweep, broom sweep). Each task is evaluated with ten randomly sampled initial configurations. The configurations are sampled prior to evaluation and verified to be kinematically feasible for the robot. For each trial, a human operator restores the scene to the designated configuration and triggers execution. We consider the trial successful if the task objective is met. Otherwise we declare failure. If the optimization produces a solution that is considered unsafe for execution, the trial is considered failure too. The success rates are reported in the main paper.

我们对以下任务进行了评估：刚体推动（纸巾盒、书本）、可变形物体操作（围巾折叠、枕头放置）、关节物体操作(打开微波炉、关闭抽屉）以及工具使用（掸子清扫、扫帚清扫）。每项任务均使用十个随机采样的初始配置进行评估。这些配置在评估前完成采样，并验证为对机器人而言在运动学上可行。对于每次试验，由一名人类操作员将场景恢复到指定配置并触发执行。如果任务目标达成，则认为该次试验成功；否则判定为失败。如果优化过程产生了被认为不适合执行的危险解，该次试验也视为失败。成功率见论文正文。

A.6.4. Effect of Training Mixture

A.6.4. 训练混合的影响

Beyond the quantitative success rates for real-world deployment, we observe interesting qualitative traits when using different variants of PointWorld pre-trained on different data mixtures. We empirically observe that models trained only on real data tend to be conservative: a common failure mode is for scene points to remain static even when the robot establishes contact, which we attribute to heavy regularization coping with annotation noise. On the other hand, models trained only on simulation data excel on rigid objects but frequently mis-segment cluttered real scenes implicitly, causing background points to move together with the target. Models trained on both real and simulated domains yield the most balanced behavior in practice, combining realistic contact handling with the ability to generalize to novel realworld scenes. A systematic study of how training-mixture design shapes deployment-time behavior, e.g., by varying real/ sim proportions or task/domain coverage under controlled conditions, remains an important direction for future work.

除了面向真实世界部署的定量成功率之外，我们还观察到，在使用基于不同数据混合预训练的不同PointWorld 变体时，会呈现出一些有趣的定性特征。我们通过实验发现，仅使用真实数据训练的模型往往较为保守：一种常见的失败模式是，即使机器人已经建立接触，场景点仍然保持静止，我们将其归因于用于应对标注噪声的强正则化。另一方面，仅使用模拟数据训练的模型在刚体物体上表现出色，但经常会对杂乱的真实场景产生隐式错误分割，导致背景点与目标一起移动。而同时在真实域和模拟域上训练的模型，在实践中表现出最为均衡的行为，既能处理真实接触，又具备泛化到新的真实世界场景的能力。系统性研究训练数据混合设计如何塑造部署时行为——例如，在受控条件下改变真实al/模拟比例或任务/领域覆盖范围 仍然是未来工作的重要方向。

A.7. Additional 3D Annotation Examples

Interactive visualizations available at project website.

FS Depth + OptimizedExtrinsics (ours)
Sensor Depth+ DatasetExtrinsics V2 (original)

Figure 14: Figure 14. DROID 3D annotations, including robot-overlaid RGBs, depths, point clouds, and comparisons to original dataset.
图 14：图 14。DROID 3D 标注，包括叠加机器人后的 RGB、深度图、点云，以及与原始数据集的对比。

Figure 15: Figure 15. DROID 3D annotations, including robot-overlaid RGBs, depths, point clouds, and comparisons to original dataset.
图 15：图 15。DROID 3D 标注，包括叠加机器人后的 RGB、深度图、点云，以及与原始数据集的对比。

Interactive visualizations available at project website.

可在项目网站查看交互式可视化内容。

Figure 16: Figure 16. DROID Unseen Rollouts, including deformable manipulation, robot-object interactions, and objectobject interactions.
图 16：图 16。DROID 未见过场景的 Rollout，包括可变形物体操作、机器人-物体交互以及物体-物体交互。

Figure 17: Figure 17. DROID Unseen Rollouts, including deformable manipulation, and grasping behaviors.
图 17：图 17。DROID 未见过的 Rollouts，包括可变形物体操作和抓取行为。

Figure 18: Figure 18. DROID Unseen Rollouts, including grasping behaviors, gravity effects, and glass objects.
图 18：图 18。DROID 未见过的 Rollouts，包括抓取行为、重力效应和玻璃物体。

Figure 19: Figure 19. BEHAVIOR-1K Unseen Rollouts, including constrained bimanual lifting, gravity effects (dropped laptop), object-object interactions (laptop v.s. table), and articulated manipulation (fridge).
图 19：图 19。BEHAVIOR-1K 未见过的 Rollouts，包括受约束的双手抬举、重力效应（掉落的laptop）、物体与物体交互（laptop v.s. table）以及关节物体操作（fridge）。

Figure 20: Figure 20. BEHAVIOR-1K Unseen Rollouts, including object-object interactions (within basket), gravity effects (within basket), and whole-body behaviors.
图 20：图 20。BEHAVIOR-1K 未见过的 Rollouts，包括物体与物体交互（basket 内）、重力效应（basket 内）以及全身行为。

Figure 21: Figure 21. BEHAVIOR-1K Unseen Rollouts, including bimanual manipulation, whole-body behaviors, and implicit shape completion.
图 21：图 21. BEHAVIOR-1K 未见过的 rollout，包括双手操作、全身行为和隐式形状补全。