Genesis Introduction (9) - Learning Locomotion Policies with Reinforcement Learning
I have summarized the learning of locomotion policies using "Reinforcement Learning" in "Genesis".
Previous
1. Learning Locomotion Policies with Reinforcement Learning
"Genesis" supports parallel simulation and is ideal for efficiently learning locomotion policies with "Reinforcement Learning" (RL). In this section, I will explain a complete learning example for obtaining a basic locomotion policy that enables the "Unitree Go2" to walk. Using "Genesis", you can learn a locomotion policy that can be deployed in the real world in less than 26 seconds (benchmarked on an RTX 4090).
This tutorial is inspired by and built upon several basic concepts from "Legged Gym".
2. Environment
Create a Gym-style environment.
2-1. Initialization
init() sets up the simulation environment.
・Control Frequency
The simulation runs at 50Hz, the same as the actual robot's control frequency. To further bridge the sim2real gap, we also manually simulate the action latency (approx. 20ms, 1dt) that occurs on the actual robot.
・Scene Creation
Create a scene containing the robot and a static plane.
・PD Controller Setup
First, motors are identified based on their names. Next, the stiffness and damping for each motor are set.
・Reward Registration
Register the reward functions defined in the config to guide the policy.
・Buffer Initialization
Buffers are initialized to store environment states, observations, and rewards.
2-2. Reset
reset_idx() resets the initial pose and state buffer of the specified environment. This ensures that the robot starts from a predefined configuration, which is essential for consistent learning.
2-3. Step
step() receives actions and returns new observations and rewards.
・Action Execution
The input action is clipped, rescaled, and added on top of the default motor positions. The transformed action, indicating the target joint positions, is sent to the robot controller for one step of execution.
・State Update
Robot states such as joint positions and velocities are retrieved and saved to the buffer.
・Termination Check
The environment terminates if "the episode length exceeds the maximum allowed value" or "the robot's body orientation deviates significantly." Terminated environments are automatically reset.
・Reward Calculation
・Observation Calculation
Observations used for learning include base angular velocity, projected gravity, commands, degrees of freedom positions, degrees of freedom velocities, and previous actions.
2-4. Reward
The reward function is critical for policy guidance.
In this instance, we use the following.
・tracking_lin_vel : Tracking of linear velocity commands (xy-axis)
・tracking_ang_vel : Tracking of angular velocity commands (yaw)
・lin_vel_z : Penalization of linear velocity in the z-axis
・action_rate : Penalization of the rate of change in actions
・base_height : Penalization when the base height deviates from the target value
・similar_to_default : Penalization when the posture deviates significantly from the default
3. Training
The training procedure in "rsl-rl" is as follows.
(1) Install the package.
git clone https://github.com/leggedrobotics/rsl_rl
cd rsl_rl && git checkout v1.0.2 && pip install -e .
cd ..
pip install tensorboard(2) Execute training.
python examples/locomotion/go2_train.py(3) Monitor the training process using "TensorBoard".
tensorboard --logdir logs
4. Evaluation
The procedure for executing evaluation is as follows.
(1) Execute evaluation.
python examples/locomotion/go2_eval.pyYou can observe the quadruped robot walking.
If you own an actual robot, you can also deploy the policy.
