Three Stages of Returning Robots Trained in Virtual Space to Real Machines: Differences Between Omniverse and Isaac Sim, and Why LAS Point Clouds Cannot Be Used Directly

I often receive requests involving relatively new technologies. I frequently work with GIS, associated AI applications, and point cloud data, and as an extension of that, I was consulted on whether it would be possible to create a digital twin of a real-world site and train robots within it.

For example, the concept is to recreate site data acquired through drone surveying or LiDAR measurement in a virtual space, and have physical AI mobile robots run there. The idea is to have them learn how to handle uneven terrain, slopes, steps, and obstacles, and then reflect those results back to the actual machines.

However, when I actually tried it, it wasn't just a matter of preparing 3D data.

In particular, the following three points need to be considered separately.

Furthermore, even if you try to import LAS format point clouds acquired by drones or LiDAR into Isaac Sim, they may not work as expected.

In this article, I will organize these relationships from the perspective of actual verification.

*The official notations are "NVIDIA Omniverse" and "NVIDIA Isaac Sim".

What is the difference between NVIDIA Omniverse and Isaac Sim?

Because Omniverse and Isaac Sim handle 3D spaces with similar screens, it can be difficult to understand the differences at first.

However, they are not the same type of software.

Omniverse is a platform that connects 3D data and applications

NVIDIA Omniverse is a collection of libraries and microservices for building 3D applications such as industrial digital twins and robot simulations.

Centered on OpenUSD, it plays the role of connecting different 3D software, CAD, simulators, rendering, and data processing. In other words, rather than being a single complete simulator, Omniverse is a foundation for integrating 3D data and building the necessary applications on top of it.

Isaac Sim is a simulator for testing robots

Isaac Sim is an open-source reference framework for robot simulation, testing, and synthetic data generation, built on top of the Omniverse libraries.

You can place robot models, set up physics calculations, reproduce sensors such as cameras, LiDAR, and IMU, and verify operations in conjunction with ROS 2 and the like.

Isaac Lab is responsible for the learning aspect

Furthermore, when performing full-scale reinforcement learning or imitation learning for robots, Isaac Lab, built on Isaac Sim, is used.

Isaac Lab is a framework for handling robot learning, such as reinforcement learning, learning from demonstrations, and motion planning.

To summarize, the relationship is as follows.

Name Main Role Easy Analogy NVIDIA Omniverse A foundation for integrating 3D data and apps centered on OpenUSD Infrastructure such as cities and roads NVIDIA Isaac Sim A simulator that reproduces robots, physics, and sensors A robot testing ground NVIDIA Isaac Lab An environment for performing reinforcement learning and imitation learning A robot training school ROS 2/Manufacturer SDK A bridge connecting virtual space and real machines that transmits learning results and commands to the real machine

Therefore, Omniverse and Isaac Sim are not competing, separate products.

is the relationship.

What does it mean to "return learning content to a real machine"?

When you hear "returning content learned in simulation to a real machine," it sounds like copying the movements learned in virtual space directly to the real machine.

However, what is actually returned is not just a record of actions.

A trained model or policy that outputs actions in response to inputs such as robot sensor values and posture is run on the real machine.

For example, the relationship between input and output is as follows.

入力
・ロボットの姿勢
・関節角度
・関節速度
・IMU
・カメラ
・LiDAR
・接地状態
・目標速度
・周辺の地形

　　　↓

学習済みモデル

　　　↓

出力
・進行方向
・走行速度
・関節の目標位置
・関節速度
・姿勢の変更
・停止や回避の判断

In Isaac Lab as well, learning tasks are constructed by defining the observations the robot receives and the actions given to the robot.

However, the method of reflection on the real machine differs significantly depending on how much of the task is assigned to the AI.

I think of this in the following three stages.

Stage 1: Learning path, speed, and action judgment

In Stage 1, the standard walking control or posture control provided by the robot manufacturer is used as-is.

What the AI judges is the action at a higher level than that.

For example, it includes the following content.

• Which route to take

• Where to decelerate

• Whether to pass to the left or right of an obstacle

• Whether to stop when sensing danger

• Whether to return based on remaining battery level

• When to switch between wheel-based travel and walking

• From which location to photograph the inspection target

• Whether to turn back if communication becomes weak

At this stage, the AI does not directly move the robot's joints, but rather issues relatively high-level commands such as "move forward," "turn right," or "slow down."

Since the manufacturer's standard safety features and posture control can be retained, this is the most realistic approach for initial commercialization.

Furthermore, it is easier to accommodate differences between robots, making it easier to deploy not only limited physical AI but also to other manufacturers' quadruped robots, wheeled robots, and AMRs.

Stage 2: Learning Walking and Posture Control Itself

In the second stage, the learning model directly or semi-directly controls the robot's joint positions, joint velocities, and torque.

Learning targets include the following:

• Maintaining posture on uneven terrain

• Ascending and descending slopes stably

• Overcoming steps

• Recovering posture when slipping

• Adapting to changes in the center of gravity due to payloads

• Not falling over even when subjected to external forces

• Changing how legs are extended according to the road surface

• Balancing power consumption and stability

• Righting its posture if it is about to fall

At this stage, it is necessary to reproduce the robot's mass, center of gravity, joint characteristics, motor response, communication latency, and ground friction quite accurately in the simulation.

Also, if the real machine does not have an SDK or development mode that accepts low-level control, the learned model cannot be deployed.

Even if the same 3D model can be loaded into Isaac Sim, if you cannot send commands to the real machine's joints, you cannot proceed to Sim-to-Real.

Stage 3: Learning movement and tasks as an integrated whole

In the third stage, you learn the entire task, including not just movement, but also the robot arm, gripper, camera, tools, and so on.

For example, tasks such as the following:

• Moving to the target facility

• Pointing the camera at the inspection point

• Opening a door

• Operating levers or valves

• Lifting parts

• Transporting cargo

• Changing posture to fit into narrow spaces

• Returning to a safe location after work

At this stage, movement control, posture control, object recognition, grasping, contact, and work procedures must be integrated.

There are also methods such as collecting data operated by humans to use for imitation learning, or generating a large number of work patterns in the simulation. Isaac Lab also provides a mechanism to generate additional demonstrations from a small number of manual demonstrations for use in learning.

While the market is very large, aiming for this stage from the beginning makes the development scope too broad.

Therefore, I believe it is more realistic to first build a track record from the first and second stages, and then expand to the third stage in the future.

The path from simulation to real machines is not one-way

The ideal workflow is not to learn once in simulation and then be done with it.

現場データを取得
　　　↓
デジタルツインを作成
　　　↓
ロボットモデルを配置
　　　↓
シミュレーションで学習
　　　↓
別のシミュレーターや条件で再検証
　　　↓
安全条件を付けて実機へ配置
　　　↓
実機の走行ログを回収
　　　↓
現実とシミュレーションの差を分析
　　　↓
物理条件を補正して再学習
　　　↓
再び実機へ配置

Creating this cycle is important.

Official Isaac Lab documentation also organizes the process from simulation-based learning to deployment on real robots as a continuous flow. Additionally, methods such as domain randomization—which varies friction, mass, position, and sensor conditions—are used to minimize the gap between simulation and reality.

However, I believe that online learning, where AI continues to learn autonomously on the real machine, should be avoided in the initial stages.

In practice,

is a workflow that is easier to manage and easier to explain to customers.

LAS data acquired by drones or LiDAR does not work well in Isaac Sim

I am attempting to load LAS-format point clouds acquired through drone surveying or LiDAR measurement into Isaac Sim.

However, simply loading the LAS file does not always display it as expected, and even if it does display, the robot may not be able to navigate correctly.

The cause of this is not necessarily low precision in the point cloud data.

In the first place, LAS and Isaac Sim have different purposes for using data.

LAS is a 'collection of points,' not a ground surface for physics calculations

LAS and LAZ are formats for storing large numbers of points acquired from real-world space.

Each point can hold information such as coordinates, height, reflection intensity, color, and classification.

However, it does not necessarily mean that a surface exists between the points.

While humans can recognize ground, walls, buildings, and pipes when looking at the screen, from the perspective of a physics simulator, it is simply a collection of points arranged in space.

Therefore, even if the point cloud is visible on the screen,

• The robot's feet pass through the ground

• The wheels do not make contact

• It is not recognized as a step

• Collision detection is not possible

• It does not become an obstacle for pathfinding

• Friction cannot be set

These problems occur.

Regarding the point clouds displayed by Isaac Sim's Debug Draw, official documentation also explains that they do not interact with the physics scene. Furthermore, the function to generate occupancy grid maps assumes that collision shapes are set in the USD stage.

In other words,

.

LAS is not included in general 3D model conversion targets

Omniverse's general Asset Converter can convert FBX, OBJ, GLTF/GLB, STL, etc., to USD.

On the other hand, LAS and LAZ are not included in the list of supported formats for general Asset Converters.

Therefore, the workflow of dragging LAS into Isaac Sim with the same feeling as a 3D model and instantly converting it into a space for simulation does not exist.

Note that in Omniverse Kit 110.2, released on July 24, 2026, a sample script for converting E57, PTS, and LAS/LAZ to the Potree 2.0 format has been added for the Point Cloud Manager.

This is a major step forward for point cloud utilization.

However, conversion to Potree 2.0 is primarily a mechanism for streaming large-scale point clouds. Omniverse's point cloud extension is also configured to stream Potree-format data to the RTX renderer.

Therefore,

are still separate processes.

Too many points

Point clouds acquired by drones or terrestrial laser scanners can range from tens of millions to hundreds of millions of points.

Point clouds tend to become very large, and the high computational load for display and manipulation is also explained in NVIDIA's point cloud extension documentation.

If you bring this directly into a simulation space, problems occur such as:

• Long loading times

• High GPU memory usage

• Viewport sluggishness

• Reduced physics update speed

• Inability to run reinforcement learning in parallel

• Slow sensor simulation

occur.

In reinforcement learning in particular, the same environment is sometimes replicated hundreds or thousands of times for trials.

Therefore, it is not efficient to use high-density point clouds that reproduce reality in detail as learning environments as-is.

Point clouds contain noise, missing data, and unnecessary objects

LAS files from the field also contain points that are unnecessary for simulation.

For example, these include:

• Vegetation

• Moving vehicles and people

• Erroneous measurements of water surfaces

• Anomalous points due to reflection

• Isolated points remaining in mid-air

• Missing data inside or behind buildings

• Parts not visible from the drone

• Thin pipes and electrical wires

• Temporarily placed materials

Even if the point cloud accurately represents the current state, converting it into a shape for physics calculations can result in holes, protrusions, steps, and missing walls.

NVIDIA's documentation for scene reconstruction also explains that if the reconstructed shape is incomplete or contains noise, the accuracy of depth information cannot be guaranteed, and the quality of the occupancy map depends on the collision shapes within the USD stage.

Do not discard LAS; use it by separating its roles

Just because point clouds are difficult to use as-is in Isaac Sim does not mean that LAS is unnecessary.

Rather, LAS is extremely important as a record of real-world space.

What is needed is to separate the roles of the point cloud rather than trying to use it for only one purpose.

1. Point cloud layer for current status verification and evidence

The original LAS/LAZ files are retained for survey results and current status verification.

• Confirming actual shapes

• Measuring cross-sections

• Comparing with past data

• Extracting changed areas

• Returning simulation results to the field

This layer is displayed using point cloud viewers or formats like Potree.

2. Lightweight 3D layer for visualization

Create a mesh for visualization from the point cloud.

Reduce the polygon count and texture size while maintaining the appearance to a certain extent.

This layer is used by users to view the digital twin.

3. Collision layer for physics simulation

Create the ground, walls, stairs, steps, and obstacles that the robot will come into contact with as simplified meshes or height models.

In this layer, you set

• colliders

• friction

• restitution

• contact detection

• traversable/non-traversable attributes

and so on.

In physics simulation, it is more important to be able to perform stable contact detection than to have visual detail.

Official Isaac Sim documentation also explains that simulation performance improves as collision shapes are simplified, and that reducing contact points has a significant effect.

In other words, you should separate the high-definition visualization mesh from the simplified physics mesh.

Realistic conversion flow from LAS to Isaac Sim

The realistic workflow I envision is as follows.

ドローン・LiDARで現場を取得
　　　↓
LAS／LAZを保存
　　　↓
対象範囲を切り出す
　　　↓
ノイズ除去・点群分類
　　　↓
地面・構造物・植生・障害物を分離
　　　↓
ローカル座標へ変換
　　　↓
表示用メッシュを作成
　　　↓
物理演算用の簡略メッシュを作成
　　　↓
USDへ変換
　　　↓
コライダー・摩擦・属性を設定
　　　↓
Isaac Simでロボットを走行
　　　↓
学習・検証
　　　↓
結果を実座標へ戻してGISに表示

For the ground, it can be converted into DEM, DSM, height maps, or triangular meshes depending on the application.

For buildings and equipment, rather than reproducing everything in high definition, prioritize shapes that affect the robot's movement and sensors.

For example, the following shapes are important for robot verification.

• Ground slope

• Step height

• Stair tread and riser

• Passageway width

• Ceiling height

• Walls and pillars

• Piping

• Holes and grooves

• Obstacles the robot will contact

• Space required for turning

On the other hand, there is no need to include fine branches of distant trees or small irregularities that do not affect travel in the physical model.

A 'service to convert LAS to USD' alone is not enough

What is important here is that this is not just simple file conversion.

Simply converting LAS to USD and displaying it on the screen does not make it a digital twin for robot simulation.

What is needed is

.

This process requires knowledge of surveying, point clouds, GIS, 3D modeling, physics simulation, and robot control.

Conversely, this can be a major point of differentiation.

Many companies are knowledgeable about robot hardware and AI, but they lack expertise in how to acquire on-site point clouds, how to reduce their size, and to what extent they should be converted into physical models.

On the other hand, surveying companies and point cloud processing companies are strong at on-site acquisition, but they may not handle Isaac Sim or Sim-to-Real.

If we can bridge this gap,

• on-site measurement

• point cloud processing

• digital twin creation

• robot model placement

• traversability verification

• learning environment creation

• trained model creation

• real machine deployment support

• re-reflecting real machine logs

I am experimenting daily with the idea that we could offer these as a series of services. The video below shows the simulation I initially conducted, now made accessible externally.

The scope to aim for first

I do not think it is necessary to leave all robot joint control to AI from the start.

It is realistic to start by focusing on the first stage: behavioral decision-making.

Specifically,

• creating traversable areas from point clouds

• setting start and destination points

• Determine steps, slopes, and path widths

• Select a safe route

• Decelerate at dangerous locations

• Avoid impassable areas

• Determine limitations based on payload

• Compile simulation results into a report

These are the functions.

After that, once the target robot's SDK and safety conditions are confirmed, we proceed to the second stage: gait and posture learning.

Furthermore, when dealing with robot arms or humanoids, I believe the third stage involves combining movement and tasks.

Summary

To briefly summarize the content covered here, it is as follows:

And there are three stages for returning what was learned in simulation to a real machine:

1. Returning path, speed, and behavioral judgment

2. Returning gait, joint, and posture control

3. Returning movement and tasks as an integrated whole

Also, LAS obtained from drones or LiDAR are important point clouds that record real space, but they cannot be used as ground for physics calculations as they are.

is.

Therefore, it is necessary to create separate models for visualization, physics simulation, and GIS/attribute information while retaining the original point cloud.

Moving forward, I want to move beyond digital twins that are merely for display, and instead aim for

.