xr_teleoperate

English | 中文 | 日本語

📺 Video Demo

G1 (29DoF) + Dex3-1

H1_2 (Arm 7DoF)

🔖Release Note

🏷️ v1.5 (2025.12.29)

support simulation
add CycloneDDS interface name parameter
add caching to speed-up urdf loading
...

0. 📖 Introduction

This repository implements teleoperation control of a Unitree humanoid robot using XR (Extended Reality) devices (such as Apple Vision Pro, PICO 4 Ultra Enterprise, or Meta Quest 3).

If you have never worked with a Unitree robot before, please at least read up to the “Application Development” chapter in the official documentation first. Additionally, the Wiki of this repo contains a wealth of background knowledge that you can reference at any time.

Here are the required devices and wiring diagram,

The currently supported devices in this repository:

🤖 Robot	⚪ Status
G1 (29 DoF)	✅ Complete
G1 (23 DoF)	✅ Complete
H1 (4‑DoF arm)	✅ Complete
H1_2 (7‑DoF arm)	✅ Complete
Dex1‑1 gripper	✅ Complete
Dex3‑1 dexterous hand	✅ Complete
Inspire dexterous hand	✅ Complete
BrainCo dexterous hand	✅ Complete
···	···

1. 📦 Installation

We tested our code on Ubuntu 20.04 and Ubuntu 22.04, other operating systems may be configured differently. This document primarily describes the default mode.

For more information, you can refer to Official Documentation and OpenTeleVision.

1.1 📥 basic

# Create a conda environment
(base) unitree@Host:~$ conda create -n tv python=3.10 pinocchio=3.1.0 numpy=1.26.4 -c conda-forge
(base) unitree@Host:~$ conda activate tv
# Clone this repo
(tv) unitree@Host:~$ git clone https://github.com/unitreerobotics/xr_teleoperate.git
(tv) unitree@Host:~$ cd xr_teleoperate
# Shallow clone submodule
(tv) unitree@Host:~/xr_teleoperate$ git submodule update --init --depth 1

# Install teleimager submodule
(tv) unitree@Host:~/xr_teleoperate$ cd teleop/teleimager
(tv) unitree@Host:~/xr_teleoperate/teleop/teleimager$ pip install -e . --no-deps

# Install televuer submodule
(tv) unitree@Host:~/xr_teleoperate$ cd teleop/televuer
(tv) unitree@Host:~/xr_teleoperate/teleop/televuer$ pip install -e .

# Configure SSL certificates for the televuer module so that XR devices (e.g., Pico / Quest / Apple Vision Pro) can securely connect via HTTPS / WebRTC
# 1. Generate certificate files
# 1.1 For Pico / Quest XR devices
(tv) unitree@Host:~/xr_teleoperate/teleop/televuer$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout key.pem -out cert.pem
# 1.2 For Apple Vision Pro
(tv) unitree@Host:~/xr_teleoperate/teleop/televuer$ openssl genrsa -out rootCA.key 2048
(tv) unitree@Host:~/xr_teleoperate/teleop/televuer$ openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 365 -out rootCA.pem -subj "/CN=xr-teleoperate"
(tv) unitree@Host:~/xr_teleoperate/teleop/televuer$ openssl genrsa -out key.pem 2048
(tv) unitree@Host:~/xr_teleoperate/teleop/televuer$ openssl req -new -key key.pem -out server.csr -subj "/CN=localhost"
# Create server_ext.cnf file with the following content (IP.2 should match your host IP, e.g., 192.168.123.2. Use ifconfig or similar to check)
(tv) unitree@Host:~/xr_teleoperate/teleop/televuer$ vim server_ext.cnf
subjectAltName = @alt_names
[alt_names]
DNS.1 = localhost
IP.1 = 192.168.123.164
IP.2 = 192.168.123.2
(tv) unitree@Host:~/xr_teleoperate/teleop/televuer$ openssl x509 -req -in server.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out cert.pem -days 365 -sha256 -extfile server_ext.cnf
(tv) unitree@Host:~/xr_teleoperate/teleop/televuer$ ls
build  cert.pem  key.pem  LICENSE  pyproject.toml  README.md  rootCA.key  rootCA.pem  rootCA.srl  server.csr  server_ext.cnf  src  test
# Copy rootCA.pem to Apple Vision Pro via AirDrop and install it

# Enable firewall
(tv) unitree@Host:~/xr_teleoperate/teleop/televuer$ sudo ufw allow 8012

# 2. Configure certificate paths, choose one method
# 2.1 User config directory (optional)
(tv) unitree@Host:~/xr_teleoperate/teleop/televuer$ mkdir -p ~/.config/xr_teleoperate/
(tv) unitree@Host:~/xr_teleoperate/teleop/televuer$ cp cert.pem key.pem ~/.config/xr_teleoperate/
# 2.2 Environment variables (optional)
(tv) unitree@Host:~/xr_teleoperate/teleop/televuer$ echo 'export XR_TELEOP_CERT="$HOME/xr_teleoperate/teleop/televuer/cert.pem"' >> ~/.bashrc
(tv) unitree@Host:~/xr_teleoperate/teleop/televuer$ echo 'export XR_TELEOP_KEY="$HOME/xr_teleoperate/teleop/televuer/key.pem"' >> ~/.bashrc
(tv) unitree@Host:~/xr_teleoperate/teleop/televuer$ source ~/.bashrc

1.2 🕹️ unitree_sdk2_python

# Install unitree_sdk2_python library which handles communication with the robot
(tv) unitree@Host:~$ git clone https://github.com/unitreerobotics/unitree_sdk2_python.git
(tv) unitree@Host:~$ cd unitree_sdk2_python
(tv) unitree@Host:~/unitree_sdk2_python$ pip install -e .

Note 1: For xr_teleoperate versions v1.1 and above, please ensure that the unitree_sdk2_python repository is checked out to a commit equal to or newer than 404fe44d76f705c002c97e773276f2a8fefb57e4.

Note 2: The unitree_dds_wrapper in the original h1_2 branch was a temporary version. It has now been fully migrated to the official Python-based control and communication library: unitree_sdk2_python.

Note 3: All identifiers in front of the command are meant for prompting: Which device and directory the command should be executed on.

In the Ubuntu system's ~/.bashrc file, the default configuration is: PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '

Taking the command (tv) unitree@Host:~$ pip install meshcat as an example:

(tv) Indicates the shell is in the conda environment named tv.

unitree@Host:~ Shows the user \u unitree is logged into the device \h Host, with the current working directory \w as $HOME.

$ shows the current shell is Bash (for non-root users).

pip install meshcat is the command unitree wants to execute on Host.

You can refer to Harley Hahn's Guide to Unix and Linux and Conda User Guide to learn more.

1.3 🚀 Launch Parameter Description

Basic control parameters

⚙️ Parameter	📜 Description	🔘 Available Options	📌 Default
`--frequency`	Set the FPS for recording and control	Any reasonable float value	30.0
`--input-mode`	Choose XR input mode (how to control the robot)	`hand` (hand tracking)`controller` (controller tracking)	`hand`
`--display-mode`	Choose XR display mode (how to view the robot perspective)	`immersive` (immersive)`ego` (pass-through + small first-person window)`pass-through` (pass-through only)	`immersive`
`--arm`	Select the robot arm type (see 0. 📖 Introduction)	`G1_29` `G1_23` `H1_2` `H1`	`G1_29`
`--ee`	Select the end-effector type of the arm (see 0. 📖 Introduction)	`dex1` `dex3` `inspire_ftp` `inspire_dfx` `brainco`	None
`--img-server-ip`	Set the image server IP address for receiving image streams and configuring WebRTC signaling	`IPv4` address	`192.168.123.164`
`--network-interface`	Set the network interface for CycloneDDS communication	Network Interface Name	`None`

Mode switch parameters

⚙️ Parameter	📜 Description
`--motion`	Enable motion control mode When enabled, the teleoperation program can run alongside the robot’s motion control program.In hand tracking mode, the R3 controller can be used to control normal robot walking; in controller tracking mode, joysticks can also control the robot’s movement.
`--headless`	Enable headless mode For running the program on devices without a display, e.g., the Development Computing Unit (PC2).
`--sim`	Enable simulation mode
`--ipc`	Inter-process communication mode Allows controlling the xr_teleoperate program’s state via IPC. Suitable for interaction with agent programs.
`--affinity`	CPU affinity mode Set CPU core affinity. If you are unsure what this is, do not set it.
`--record`	Enable data recording mode Press r to start teleoperation, then s to start recording; press s again to stop and save the episode. Press s repeatedly to repeat the process.
`--task-*`	Configure the save path, target, description, and steps of the recorded task.

1.4 🔄 State Transition Diagram

2. 💻 Simulation Deployment

2.1 📥 Environment Setup

First, install unitree_sim_isaaclab. Follow that repo’s README.

Then launch the simulation with a G1(29 DoF) and Dex3 hand configuration:

(base) unitree@Host:~$ conda activate unitree_sim_env
(unitree_sim_env) unitree@Host:~$ cd ~/unitree_sim_isaaclab
(unitree_sim_env) unitree@Host:~/unitree_sim_isaaclab$ python sim_main.py --device cpu --enable_cameras --task Isaac-PickPlace-Cylinder-G129-Dex3-Joint --enable_dex3_dds --robot_type g129

💥💥💥 NOTICE❗

After simulation starts, click once in the window to activate it.

The terminal will show: controller started, start main loop...

Here is the simulation GUI:

2.2 🚀 Launch

This program supports XR control of a physical robot or in simulation. Choose modes with command-line arguments:

Assuming hand tracking with G1(29 DoF) + Dex3 in simulation with recording:

(tv) unitree@Host:~$ cd ~/xr_teleoperate/teleop/
(tv) unitree@Host:~/xr_teleoperate/teleop/$ python teleop_hand_and_arm.py --xr-mode=hand --arm=G1_29 --ee=dex3 --sim --record
# Simplified (defaults apply):
(tv) unitree@Host:~/xr_teleoperate/teleop/$ python teleop_hand_and_arm.py --ee=dex3 --sim --record

After the program starts, the terminal shows:

Next steps:

Wear your XR headset (e.g. Apple Vision Pro, PICO4, etc.)
Connect to the corresponding Wi‑Fi
Only proceed if your head camera has WebRTC enabled (cam_config_server.yaml → head_camera → enable_webrtc: true); otherwise jump to Step 4. Open a browser (e.g. Safari or PICO Browser) and go to:
https://192.168.123.164:60001
Note 1: This IP is the address of PC2—the machine running teleimager service.
Note 2: You may see a warning page like step 4. Click Advanced, then Proceed to IP (unsafe). Once the page loads, press the start button in the top-left corner; if you see the head-camera preview, the check is successful.

Note 3: This step serves two purposes:
1. Verify that the teleimager service is running correctly.
2. Manually trust the WebRTC self-signed certificate.
Once this has been done on the same device with the same certificate, you can skip it on subsequent launches.
Open a browser (e.g. Safari or PICO Browser) and go to: https://192.168.123.2:8012/?ws=wss://192.168.123.2:8012

Note 1: This IP must match your Host IP (check with ifconfig).

Note 2: You may see a warning page. Click Advanced, then Proceed to IP (unsafe).
In the Vuer web, click Virtual Reality. Allow all prompts to start the VR session.

You’ll see the robot’s first-person view in the headset. The terminal prints connection info:

websocket is connected. id:dbb8537d-a58c-4c57-b49d-cbb91bd25b90
default socket worker is up, adding clientEvents
Uplink task running. id:dbb8537d-a58c-4c57-b49d-cbb91bd25b90

Align your arm to the robot’s initial pose to avoid sudden movements at start:
Press r in the terminal to begin teleoperation. You can now control the robot arm and dexterous hand.
During teleoperation, press s to start recording; press s again to stop and save. Repeatable process.

Note 1: Recorded data is stored in xr_teleoperate/teleop/utils/data by default, with usage instructions at this repo: unitree_IL_lerobot.

Note 2: Please pay attention to your disk space size during data recording.

Note 3: In v1.4 and above, the “record image” window has been removed.

2.3 🔚 Exit

Press q in the terminal (or “record image” window) to quit.

3. 🤖 Physical Deployment

Physical deployment steps are similar to simulation, with these key differences:

3.1 🖼️ Image Service

In the simulation environment, the image service is automatically enabled. For physical deployment, you need to manually start the image service based on your specific camera hardware. The steps are as follows:

Install the image service program on the Development Computing Unit PC2 of the Unitree robot (G1/H1/H1_2, etc.)

# SSH into PC2 and download the image service repository

(base) unitree@PC2:~$ cd ~
(base) unitree@PC2:~$ git clone https://github.com/silencht/teleimager

# Configure the environment according to the instructions in the teleimager repository README: https://github.com/silencht/teleimager/blob/main/README.md

On the local host, execute the following commands:

# Copy the `key.pem` and `cert.pem` files configured in Section 1.1 from the **local host** `xr_teleoperate/teleop/televuer` directory to the corresponding path on PC2

# These two files are required by teleimager to start the WebRTC service
(tv) unitree@Host:~$ scp ~/xr_teleoperate/teleop/televuer/key.pem ~/xr_teleoperate/teleop/televuer/cert.pem [email protected]:~/teleimager

# On PC2, configure the certificate path according to the teleimager repository README, for example:
(teleimager) unitree@PC2:~$ cd teleimager
(teleimager) unitree@PC2:~$ mkdir -p ~/.config/xr_teleoperate/
(teleimager) unitree@PC2:~/teleimager$ cp cert.pem key.pem ~/.config/xr_teleoperate/

On the development computing unit PC2, configure cam_config_server.yaml according to the teleimager documentation and start the image service.

(teleimager) unitree@PC2:~/image_server$ python -m teleimager.image_server

# The following command works the same way
(teleimager) unitree@PC2:~/image_server$ teleimager-server

On the local host, execute the following command to subscribe to the images

(tv) unitree@Host:~$ cd ~/xr_teleoperate/teleop/teleimager/src
(tv) unitree@Host:~/xr_teleoperate/teleop/teleimager/src$ python -m teleimager.image_client --host 192.168.123.164

# If the WebRTC image stream is set up, you can also open the URL [https://192.168.123.164:60001](https://192.168.123.164:60001) in a browser and click the Start button to test.

3.2 ✋ Inspire Hand Service (optional)

Note 1: Skip this if your config does not use the Inspire hand.

Note 2: For G1 robot with Inspire DFX hand, related issue #46.

Note 3: For Inspire FTP hand, related issue #48. FTP dexterous hand is now supported. Please refer to the --ee parameter for configuration.

First, use this URL: DFX_inspire_service to clone the dexterous hand control interface program. And Copy it to PC2 of Unitree robots.

On Unitree robot's PC2, execute command:

unitree@PC2:~$ sudo apt install libboost-all-dev libspdlog-dev
# Build project
unitree@PC2:~$ cd DFX_inspire_service && mkdir build && cd build
unitree@PC2:~/DFX_inspire_service/build$ cmake ..
unitree@PC2:~/DFX_inspire_service/build$ make -j6

# (For unitree g1) Terminal 1.
unitree@PC2:~/DFX_inspire_service/build$ sudo ./inspire_g1
# or (For unitree h1) Terminal 1.
unitree@PC2:~/DFX_inspire_service/build$ sudo ./inspire_h1 -s /dev/ttyUSB0

# Terminal 2. Run example
unitree@PC2:~/DFX_inspire_service/build$ ./hand_example

If two hands open and close continuously, it indicates success. Once successful, close the ./hand_example program in Terminal 2.

3.3 ✋ BrainCo Hand Service (Optional)

Please refer to the Repo README for setup instructions.

3.4 ✋ Unitree Dex1_1 Service (Optional)

Please refer to the Repo README for setup instructions.

3.5 🚀 Launch

Everyone must keep a safe distance from the robot to prevent any potential danger!

Please make sure to read the Official Documentation at least once before running this program.

To use motion mode (with --motion), ensure the robot is in control mode (via R3 remote).

In motion mode:

Right controller A = Exit teleop

Both joysticks pressed = soft emergency stop (switch to damping mode)

Left joystick = drive directions;

right joystick = turning;

max speed is limited in the code.

Same as simulation but follow the safety warnings above.

3.6 🔚 Exit

To avoid damaging the robot, it is recommended to position the robot's arms close to the initial pose before pressing q to exit.

In Debug Mode: After pressing the exit key, both arms will return to the robot's initial pose within 5 seconds, and then the control will end.

In Motion Mode: After pressing the exit key, both arms will return to the robot's motion control pose within 5 seconds, and then the control will end.

Same as simulation but follow the safety warnings above.

4. 🗺️ Codebase Overview

xr_teleoperate/
│
├── assets                    [Stores robot URDF-related files]
│
├── teleop
│   ├── teleimager            [New image service library, supporting multiple features]
│   │
│   ├── televuer
│   │      ├── src/televuer
│   │         ├── television.py       [Captures head, wrist, and hand/controller data from XR devices using Vuer]
│   │         ├── tv_wrapper.py       [Post-processing of captured data]
│   │      ├── test
│   │         ├── _test_television.py [Test program for television.py]
│   │         ├── _test_tv_wrapper.py [Test program for tv_wrapper.py]
│   │
│   ├── robot_control
│   │      ├── src/dex-retargeting [Dexterous hand retargeting algorithm library]
│   │      ├── robot_arm_ik.py     [Inverse kinematics for the arm]
│   │      ├── robot_arm.py        [Controls dual-arm joints and locks other parts]
│   │      ├── hand_retargeting.py [Wrapper for the dexterous hand retargeting library]
│   │      ├── robot_hand_inspire.py  [Controls Inspire dexterous hand]
│   │      ├── robot_hand_unitree.py  [Controls Unitree dexterous hand]
│   │
│   ├── utils
│   │      ├── episode_writer.py          [Used to record data for imitation learning]
│   │      ├── weighted_moving_filter.py  [Filter for joint data]
│   │      ├── rerun_visualizer.py        [Visualizes recorded data]
│   │      ├── ipc.py                     [Handles inter-process communication with proxy programs]
│   │      ├── motion_switcher.py         [Switches motion control states]
│   │      ├── sim_state_topic.py         [For simulation deployment]
│   │
│   └── teleop_hand_and_arm.py    [Startup script for teleoperation]

5. 🛠️ Hardware

please see Device document.

6. 🙏 Acknowledgement

This code builds upon following open-source code-bases. Please visit the URLs to see the respective LICENSES:

Name		Name	Last commit message	Last commit date
Latest commit History 159 Commits
assets		assets
img		img
teleop		teleop
.gitignore		.gitignore
.gitmodules		.gitmodules
CHANGELOG.md		CHANGELOG.md
CHANGELOG_zh-CN.md		CHANGELOG_zh-CN.md
Device.md		Device.md
Device_zh-CN.md		Device_zh-CN.md
LICENSE		LICENSE
README.md		README.md
README_ja-JP.md		README_ja-JP.md
README_zh-CN.md		README_zh-CN.md
requirements.txt		requirements.txt

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

xr_teleoperate

📺 Video Demo

🔖Release Note

🏷️ v1.5 (2025.12.29)

0. 📖 Introduction

1. 📦 Installation

1.1 📥 basic

1.2 🕹️ unitree_sdk2_python

1.3 🚀 Launch Parameter Description

1.4 🔄 State Transition Diagram

2. 💻 Simulation Deployment

2.1 📥 Environment Setup

2.2 🚀 Launch

2.3 🔚 Exit

3. 🤖 Physical Deployment

3.1 🖼️ Image Service

3.2 ✋ Inspire Hand Service (optional)

3.3 ✋ BrainCo Hand Service (Optional)

3.4 ✋ Unitree Dex1_1 Service (Optional)

3.5 🚀 Launch

3.6 🔚 Exit

4. 🗺️ Codebase Overview

5. 🛠️ Hardware

6. 🙏 Acknowledgement

About

Uh oh!

Releases

Packages

Uh oh!

Contributors 8

Uh oh!

Languages

License

unitreerobotics/xr_teleoperate

Folders and files

Latest commit

History

Repository files navigation

xr_teleoperate

📺 Video Demo

🔖Release Note

🏷️ v1.5 (2025.12.29)

0. 📖 Introduction

1. 📦 Installation

1.1 📥 basic

1.2 🕹️ unitree_sdk2_python

1.3 🚀 Launch Parameter Description

1.4 🔄 State Transition Diagram

2. 💻 Simulation Deployment

2.1 📥 Environment Setup

2.2 🚀 Launch

2.3 🔚 Exit

3. 🤖 Physical Deployment

3.1 🖼️ Image Service

3.2 ✋ Inspire Hand Service (optional)

3.3 ✋ BrainCo Hand Service (Optional)

3.4 ✋ Unitree Dex1_1 Service (Optional)

3.5 🚀 Launch

3.6 🔚 Exit

4. 🗺️ Codebase Overview

5. 🛠️ Hardware

6. 🙏 Acknowledgement

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors 8

Uh oh!

Languages

Packages