Adds Raycaster with tracking for dynamic meshes #3298

renezurbruegg · 2025-08-28T08:35:58Z

Description

This PR introduces MultiMeshRayCaster and MultiMeshRayCasterCamera, an extension of the default RayCaster with the following enhancements:

Raycasting against multiple target types : Supports primitive shapes (spheres, cubes, …) as well as arbitrary meshes.
Dynamic mesh tracking : Keeps track of specified meshes, enabling raycasting against moving parts (e.g., robot links, articulated bodies, or dynamic obstacles).
Memory-efficient caching : Avoids redundant memory usage by caching and reusing duplicate meshes.

This is joint work with @pascal-roth and @Mayankm96.

The default RayCaster was limited to static environments and required manual handling of moving meshes, which restricted its use for robotics scenarios where robots or obstacles move dynamically.

MultiMeshRayCaster addresses these limitations by and now supports raycasting against robot parts and other moving entities.

Usage

For a quick demo, run:

python scripts/demos/sensors/multi_mesh_raycaster.py --num_envs 16 --asset_type <allegro_hand|anymal_d|multi>

Drop-in replacement

Example change to migrate from RayCasterCfg to MultiMeshRayCasterCfg:

- ray_caster_cfg = RayCasterCfg(
+ ray_caster_cfg = MultiMeshRayCasterCfg(
      prim_path="{ENV_REGEX_NS}/Robot",
      mesh_prim_paths=[
         "/World/Ground",
+         MultiMeshRayCasterCfg.RaycastTargetCfg(target_prim_expr="{ENV_REGEX_NS}/Robot/LF_.*/visuals"),
+         MultiMeshRayCasterCfg.RaycastTargetCfg(target_prim_expr="{ENV_REGEX_NS}/Robot/RF_.*/visuals"),
+         MultiMeshRayCasterCfg.RaycastTargetCfg(target_prim_expr="{ENV_REGEX_NS}/Robot/LH_.*/visuals"),
+         MultiMeshRayCasterCfg.RaycastTargetCfg(target_prim_expr="{ENV_REGEX_NS}/Robot/RH_.*/visuals"),
+         MultiMeshRayCasterCfg.RaycastTargetCfg(target_prim_expr="{ENV_REGEX_NS}/Robot/base/visuals"),
      ],
      pattern_cfg=patterns.GridPatternCfg(resolution=resolution, size=(5.0, 5.0)),
 )

Benchmarking & Validation

To benchmark the new raycaster, run:

python scripts/benchmarks/benchmark_ray_caster.py

Then plot the results with:

python scripts/benchmarks/plot_raycast_results.py

This will generate outputs under:
outputs/benchmarks/raycast_benchmark...

Example plots

Type of Change

New feature (non-breaking change which adds functionality)
This change requires a documentation update

Checklist

I have run the pre-commit checks with ./isaaclab.sh --format
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
I have updated the changelog and the corresponding version in the extension's config/extension.toml file
I have added my name to the CONTRIBUTORS.md or my name already exists there

Add the efficient multi-mesh raycasting function implemented in Orbit. Moreover, this PR fixes the test of it. The new raycaster allows to raycast against multiple objects, which can be located at different positions in each environment. The positions can be tracked over time if enabled in the config. - New feature (non-breaking change which adds functionality) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Co-authored-by: zrene <[email protected]>

…ssion (isaac-sim#48) Fixes number of meshes in the `RayCaster` when raycasting dynamically against a regex expression of multiple objects in the scene. - Bug fix (non-breaking change which fixes an issue) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there

Change Mulit-mesh raycaster and raycaster camera to own files, restore the ones of main to simplify the merge. NOTE: test of the camera is currently failing, similar as on public main at that time, should be fixed after update to latest main - Breaking change (fix or feature that would cause existing functionality to not work as expected) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there

…d fixes tests (isaac-sim#65) ClassVar not correctly destroyed, thus removing it (follow changes of original RayCaster). Fixing tests to comply with changes of our internal code with 1.4.1. - Bug fix (non-breaking change which fixes an issue) - Breaking change (fix or feature that would cause existing functionality to not work as expected) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Co-authored-by: zrene <[email protected]>

…_meshes for now

…nstances to benchmark. Fix callback issues for mulit mesh

greptile-apps

Greptile Overview

Greptile Summary

Adds MultiMeshRayCaster and MultiMeshRayCasterCamera classes that extend the default raycaster with support for dynamic mesh tracking, multiple target types (primitives and arbitrary meshes), and memory-efficient caching of duplicate meshes.

Key Changes:

New raycasting implementation supporting dynamic meshes with transform tracking via physics views
Mesh deduplication system to avoid redundant GPU memory usage
Warp kernel operations for efficient multi-mesh raycasting with both static and dynamic meshes
Comprehensive test coverage and demo scripts showcasing the new capabilities
New utility module for converting USD primitives to trimesh objects

Issues Found:

Critical syntax error in raycaster_sensor.py (line 128-136) causing indentation mismatch that will prevent the script from running
Previous comments identified various issues in tests and configuration files that appear to have been addressed

Confidence Score: 4/5

This PR is mostly safe to merge after fixing the syntax error in raycaster_sensor.py
The implementation is well-structured with proper mesh caching, deduplication logic, and comprehensive testing. The core multi-mesh raycasting functionality appears sound with proper handling of dynamic transforms. However, there is one critical syntax error in a demo script that must be fixed before merging. The new functionality is appropriately isolated and shouldn't break existing code.
scripts/demos/sensors/raycaster_sensor.py requires immediate attention for syntax error fix

Important Files Changed

File Analysis

Filename	Score	Overview
scripts/demos/sensors/raycaster_sensor.py	3/5	Demo script with syntax error in conditional block indentation that will cause parse failure
source/isaaclab/isaaclab/sensors/ray_caster/multi_mesh_ray_caster.py	4/5	Core multi-mesh raycaster implementation - solid architecture with proper mesh caching and deduplication
source/isaaclab/isaaclab/utils/warp/ops.py	4/5	Warp raycasting operations - well-structured with proper handling of static and dynamic meshes
source/isaaclab/isaaclab/utils/mesh.py	5/5	Mesh utility functions for converting USD primitives to trimesh - clean implementation with proper primitive support
source/isaaclab/isaaclab/sensors/ray_caster/multi_mesh_ray_caster_camera.py	4/5	Camera variant of multi-mesh raycaster - properly extends base functionality for camera-based raycasting

Sequence Diagram

sequenceDiagram
    participant User
    participant MultiMeshRayCaster
    participant MeshCache
    participant PhysicsView
    participant WarpOps
    participant WarpKernel

    User->>MultiMeshRayCaster: Initialize with config
    MultiMeshRayCaster->>MultiMeshRayCaster: Parse mesh_prim_paths
    
    loop For each target mesh
        MultiMeshRayCaster->>MeshCache: Check if mesh exists
        alt Mesh cached
            MeshCache-->>MultiMeshRayCaster: Return mesh ID
        else New mesh
            MultiMeshRayCaster->>MultiMeshRayCaster: Load USD mesh/primitive
            MultiMeshRayCaster->>MultiMeshRayCaster: Convert to trimesh
            MultiMeshRayCaster->>MultiMeshRayCaster: Check for duplicates
            MultiMeshRayCaster->>WarpOps: convert_to_warp_mesh()
            WarpOps-->>MeshCache: Store warp mesh
            MeshCache-->>MultiMeshRayCaster: Return mesh ID
        end
        
        alt track_mesh_transforms enabled
            MultiMeshRayCaster->>PhysicsView: Create ArticulationView/RigidBodyView
            PhysicsView-->>MultiMeshRayCaster: Store view + offsets
        end
    end

    User->>MultiMeshRayCaster: update(dt)
    MultiMeshRayCaster->>MultiMeshRayCaster: Check if update needed
    
    alt Dynamic meshes
        loop For each tracked mesh
            MultiMeshRayCaster->>PhysicsView: obtain_world_pose_from_view()
            PhysicsView-->>MultiMeshRayCaster: Return pos, orientation
            MultiMeshRayCaster->>MultiMeshRayCaster: Apply offsets
        end
    end
    
    MultiMeshRayCaster->>WarpOps: raycast_dynamic_meshes()
    WarpOps->>WarpKernel: Launch kernel with mesh transforms
    
    loop For each mesh, env, ray
        WarpKernel->>WarpKernel: Transform ray to mesh local frame
        WarpKernel->>WarpKernel: Perform raycast
        WarpKernel->>WarpKernel: Keep closest hit
    end
    
    WarpKernel-->>WarpOps: Return hits, distances, normals, mesh_ids
    WarpOps-->>MultiMeshRayCaster: Return ray hit results
    MultiMeshRayCaster-->>User: data.ray_hits_w available

Additional Comments (1)

scripts/demos/sensors/raycaster_sensor.py, line 128-136 (link)

syntax: indentation incorrect - the else on line 135 should align with the if on line 128

_{22 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

kellyguo11 · 2025-11-07T18:50:46Z

@renezurbruegg @pascal-roth @Mayankm96 - could we try to get another approval for this PR?

also looks like the doc building job is failing - https://github.com/isaac-sim/IsaacLab/actions/runs/19120760997/job/54640579518?pr=3298

greptile-apps · 2025-11-11T01:17:11Z

Greptile Overview

Greptile Summary

This PR adds MultiMeshRayCaster and MultiMeshRayCasterCamera, extending the existing raycaster to support dynamic mesh tracking, multiple target types, and memory-efficient mesh caching.

Key Changes:

New MultiMeshRayCaster class enabling raycasting against moving meshes (robot links, articulated bodies) with per-environment tracking
Warp kernel infrastructure (raycast_static_meshes_kernel, raycast_dynamic_meshes_kernel) for GPU-accelerated raycasting with atomic operations to handle multiple meshes per ray
Mesh deduplication system comparing vertex arrays to avoid redundant GPU memory usage across environments
Camera variant (MultiMeshRayCasterCamera) generating depth images with proper coordinate frame transformations
New isaaclab.utils.mesh module converting USD primitives (spheres, cubes, cylinders, etc.) to trimesh objects
Comprehensive test coverage (1114 new test lines) validating multi-mesh scenarios, dynamic transforms, and mesh ID tracking

Technical Implementation:

Class-level mesh dictionaries (meshes, mesh_offsets, mesh_views) shared across instances for memory efficiency
Per-environment mesh ID arrays (mesh_ids_wp) enabling batched raycasting across multiple environments
Optional transform tracking via track_mesh_transforms flag - static meshes skip pose updates for performance
Mesh merging capability combining multiple child meshes into single raycasting target

Issues Found:

Minor: Empty mesh prim handling at line 206 uses continue which could leave _num_meshes_per_env incomplete, potentially causing KeyError in later processing
Most previously reported issues are either false positives or acknowledged design choices (e.g., Warp atomic_min race condition is documented upstream issue)

Confidence Score: 4/5

Safe to merge with one minor logic issue noted that should be reviewed but likely doesn't affect typical usage scenarios
The implementation is well-architected with comprehensive test coverage (1114 new test lines), proper GPU memory management through mesh deduplication, and clean separation of concerns. The warp kernel race condition is a known upstream issue documented in the code. The one issue identified (incomplete _num_meshes_per_env handling when mesh prims are empty) is edge-case behavior that likely won't affect normal operation but should be validated. The PR successfully achieves its goal of enabling dynamic mesh raycasting while maintaining backward compatibility.
source/isaaclab/isaaclab/sensors/ray_caster/multi_mesh_ray_caster.py - validate empty mesh prim handling around line 206-215 doesn't cause downstream KeyErrors

Important Files Changed

File Analysis

Filename	Score	Overview
source/isaaclab/isaaclab/sensors/ray_caster/multi_mesh_ray_caster.py	4/5	Core implementation of multi-mesh raycasting with dynamic tracking, mesh deduplication, and per-environment mesh management. Minor issues with early-return logic and mesh counting.
source/isaaclab/isaaclab/utils/warp/kernels.py	4/5	New warp kernels for static and dynamic mesh raycasting. Race condition is documented as known issue with Warp library. Docstring shape mismatches noted in previous reviews.
source/isaaclab/isaaclab/utils/warp/ops.py	4/5	Added raycast_single_mesh and raycast_dynamic_meshes functions with proper tensor management. Quaternion conversion and initialization patterns are correct.
source/isaaclab/isaaclab/sensors/ray_caster/multi_mesh_ray_caster_camera.py	4/5	Camera implementation with image-based raycasting output. Properly handles depth clipping and coordinate frame transformations. Multiple inheritance from RayCasterCamera and MultiMeshRayCaster works as expected.
source/isaaclab/isaaclab/utils/mesh.py	5/5	New utility module for converting USD geometric primitives to trimesh objects. Clean implementation with proper axis handling for cylinders and capsules.
source/isaaclab/isaaclab/sensors/ray_caster/ray_caster.py	4/5	Modified base RayCaster class to support mesh tracking infrastructure. Added class-level mesh dictionaries and tracking logic.

Sequence Diagram

sequenceDiagram
    participant User
    participant MultiMeshRayCaster
    participant MultiMeshRayCasterCamera
    participant WarpOps
    participant WarpKernels
    participant MeshUtils
    
    Note over User,MeshUtils: Initialization Phase
    User->>MultiMeshRayCaster: __init__(cfg)
    MultiMeshRayCaster->>MultiMeshRayCaster: _initialize_warp_meshes()
    MultiMeshRayCaster->>MeshUtils: find_matching_prims(target_prim_expr)
    MeshUtils-->>MultiMeshRayCaster: target_prims[]
    
    loop For each target prim
        MultiMeshRayCaster->>MeshUtils: get_all_matching_child_prims()
        MeshUtils-->>MultiMeshRayCaster: mesh_prims[]
        MultiMeshRayCaster->>MeshUtils: create_trimesh_from_geom_*()
        MeshUtils-->>MultiMeshRayCaster: trimesh_mesh
        MultiMeshRayCaster->>MultiMeshRayCaster: Check for duplicate vertices
        alt Duplicate found and reference_meshes=True
            MultiMeshRayCaster->>MultiMeshRayCaster: Reuse existing wp_mesh.id
        else New mesh
            MultiMeshRayCaster->>WarpOps: convert_to_warp_mesh()
            WarpOps-->>MultiMeshRayCaster: wp_mesh
        end
        MultiMeshRayCaster->>MultiMeshRayCaster: Store in class-level meshes dict
    end
    
    MultiMeshRayCaster->>MultiMeshRayCaster: Create mesh_ids_wp array (num_envs x num_meshes)
    alt track_mesh_transforms=True
        MultiMeshRayCaster->>MultiMeshRayCaster: _get_trackable_prim_view()
        MultiMeshRayCaster->>MultiMeshRayCaster: Store in mesh_views dict
    end
    
    Note over User,WarpKernels: Update/Raycast Phase
    User->>MultiMeshRayCaster: data (property access)
    MultiMeshRayCaster->>MultiMeshRayCaster: _update_outdated_buffers()
    MultiMeshRayCaster->>MultiMeshRayCaster: _update_buffers_impl(env_ids)
    
    alt track_mesh_transforms=True
        loop For each tracked mesh view
            MultiMeshRayCaster->>MultiMeshRayCaster: obtain_world_pose_from_view()
            MultiMeshRayCaster->>MultiMeshRayCaster: Apply mesh_offsets
            MultiMeshRayCaster->>MultiMeshRayCaster: Update _mesh_positions_w, _mesh_orientations_w
        end
    end
    
    MultiMeshRayCaster->>WarpOps: raycast_dynamic_meshes()
    WarpOps->>WarpOps: Initialize output tensors
    WarpOps->>WarpOps: Convert quaternions wxyz->xyzw
    
    alt Static meshes (no transforms)
        WarpOps->>WarpKernels: raycast_static_meshes_kernel()
    else Dynamic meshes (with transforms)
        WarpOps->>WarpKernels: raycast_dynamic_meshes_kernel()
    end
    
    WarpKernels->>WarpKernels: Thread per (mesh_id, env, ray)
    loop For each mesh
        WarpKernels->>WarpKernels: Transform ray to mesh local space
        WarpKernels->>WarpKernels: wp.mesh_query_ray()
        alt Hit found and closer than current
            WarpKernels->>WarpKernels: atomic_min(ray_distance)
            WarpKernels->>WarpKernels: Update ray_hits, ray_normal, ray_face_id, ray_mesh_id
        end
    end
    
    WarpKernels-->>WarpOps: Hit results
    WarpOps-->>MultiMeshRayCaster: ray_hits_w, mesh_ids
    MultiMeshRayCaster->>MultiMeshRayCaster: Store in _data container
    MultiMeshRayCaster-->>User: MultiMeshRayCasterData
    
    Note over User,WarpKernels: Camera-Specific Flow
    User->>MultiMeshRayCasterCamera: data (property access)
    MultiMeshRayCasterCamera->>MultiMeshRayCasterCamera: _update_buffers_impl()
    MultiMeshRayCasterCamera->>MultiMeshRayCasterCamera: Update mesh transforms (inherited)
    MultiMeshRayCasterCamera->>WarpOps: raycast_dynamic_meshes()
    WarpOps-->>MultiMeshRayCasterCamera: ray_hits, ray_depth, ray_normal, mesh_ids
    
    alt distance_to_image_plane requested
        MultiMeshRayCasterCamera->>MultiMeshRayCasterCamera: Transform to camera frame
        MultiMeshRayCasterCamera->>MultiMeshRayCasterCamera: Extract z-component
        MultiMeshRayCasterCamera->>MultiMeshRayCasterCamera: Apply depth clipping
    end
    
    MultiMeshRayCasterCamera->>MultiMeshRayCasterCamera: Reshape to image shape (H, W)
    MultiMeshRayCasterCamera-->>User: MultiMeshRayCasterCameraData

greptile-apps

_{22 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps · 2025-11-11T01:17:10Z

source/isaaclab/isaaclab/sensors/ray_caster/multi_mesh_ray_caster.py

+                if len(mesh_prims) == 0:
+                    warn_msg = (
+                        f"No mesh prims found at path: {target_prim.GetPath()} with supported types:"
+                        f" {PRIMITIVE_MESH_TYPES + ['Mesh']}"
+                        " Skipping this target."
+                    )
+                    for prim in sim_utils.get_all_matching_child_prims(target_prim.GetPath(), lambda prim: True):
+                        warn_msg += f"\n - Available prim '{prim.GetPath()}' of type '{prim.GetTypeName()}'"
+                    carb.log_warn(warn_msg)
+                    continue


logic: continue on empty mesh prims skips the target but allows processing to continue with potentially incomplete mesh setup. If no mesh prims are found for any target in this loop, the code continues but _num_meshes_per_env won't be updated correctly for this target. Later code at line 310 expects all target_cfgs to have entries in _num_meshes_per_env, which could cause KeyError. Should this be tracked or validated at the end of mesh initialization?

XiaoYi-Wei · 2025-11-12T10:10:06Z

Simulation startup becomes extremely slow when using the RayCaster sensor with large numbers of environments

Description

When running the RayCaster sensor branch in IsaacSim, the simulation initialization time increases drastically as the number of environments grows.

The program does not crash, but the simulation hangs for a long time at the message:
[INFO]: Starting the simulation. This may take a few seconds. Please wait...

Steps to Reproduce

Launch the simulation using the RayCaster sensor branch.
Set the number of environments to a large value (e.g., 1000, 2000, or more).
Observe the time it takes to start the simulation.

Observed Behavior

The console shows normal terrain and scene creation messages, then remains at picture
The program stays in this state for a long time without proceeding.
Startup time increases roughly linearly with the number of environments.

Number of Environments	Approx. Startup Time
1	Immediate
1000	~1 minute
4000	Several minutes

Expected Behavior

The simulation should start within a reasonable time regardless of the number of environments, consistent with other sensors or configurations.

Environment

IsaacSim version: [your version]
RayCaster branch: [branch name or commit hash]
GPU / OS: [your specs]
Number of environments tested: up to 4000

XiaoYi-Wei · 2025-11-13T07:29:21Z

### Issue Description

I found a potential bug in the file:

`source/isaaclab/isaaclab/sensors/ray_caster/multi_mesh_ray_caster_camera.py`  
at **line 178**.

The current code is:

```python
self.ray_hits_w, ray_depth, ray_normal, _, ray_mesh_ids = raycast_dynamic_meshes()

However, the first returned value of raycast_dynamic_meshes() has the shape:

[len(env_ids), num_rays, 3]

This causes a CUDA kernel out-of-bounds access, because self.ray_hits_w expects indexing over all environments, while the returned tensor only corresponds to len(env_ids).
The issue doesn't throw an error immediately because self.ray_hits_w is not used later in this path, but the kernel still performs invalid memory access.

Suggested Fix

It seems the assignment should correctly write only to the selected environments:

self.ray_hits_w[env_ids], ray_depth, ray_normal, _, ray_mesh_ids = raycast_dynamic_meshes()

This aligns the shapes and prevents hidden OOB kernel access.

source/isaaclab/isaaclab/sensors/ray_caster/multi_mesh_ray_caster.py

…ter.py Signed-off-by: Pascal Roth <[email protected]>

pascal-roth · 2025-11-18T23:09:20Z

@XiaoYi-Wei thanks for the catch, should be fixed now

…egg/IsaacLab into feature/multi-mesh-ray-caster

pascal-roth

Should have addressed everything that was still open.

TODO: rerun the examples!

pascal-roth · 2025-11-19T22:12:29Z

need to merge #3924 first

Signed-off-by: Kelly Guo <[email protected]>

Removed unused imports for cleaner code. Signed-off-by: Kelly Guo <[email protected]>

kellyguo11 · 2025-12-10T19:57:05Z

@pascal-roth @renezurbruegg could someone help take a look at the doc build failure?

Mayankm96 · 2025-12-16T12:06:23Z

@kellyguo11 I made an MR now to address the comments remaining here. I think Pascal and Rene are OoO now.

# Description This MR adds the remaining comments in #3298 ## Type of change - Bug fix (non-breaking change which fixes an issue) ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Signed-off-by: renezurbruegg <[email protected]> Signed-off-by: Mayank Mittal <[email protected]> Signed-off-by: James Tigue <[email protected]> Signed-off-by: Pascal Roth <[email protected]> Signed-off-by: Kelly Guo <[email protected]> Co-authored-by: Pascal Roth <[email protected]> Co-authored-by: zrene <[email protected]> Co-authored-by: Pascal Roth <[email protected]> Co-authored-by: James Tigue <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Kelly Guo <[email protected]> Co-authored-by: zrene <[email protected]> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

@pascal-roth

# Description This PR introduces `MultiMeshRayCaster` and `MultiMeshRayCasterCamera`, an extension of the default `RayCaster` with the following enhancements: 1. **Raycasting against multiple target types** : Supports primitive shapes (spheres, cubes, …) as well as arbitrary meshes. 2. **Dynamic mesh tracking** : Keeps track of specified meshes, enabling raycasting against moving parts (e.g., robot links, articulated bodies, or dynamic obstacles). 3. **Memory-efficient caching** : Avoids redundant memory usage by caching and reusing duplicate meshes. This is joint work with @pascal-roth and @Mayankm96. The default `RayCaster` was limited to static environments and required manual handling of moving meshes, which restricted its use for robotics scenarios where robots or obstacles move dynamically. `MultiMeshRayCaster` addresses these limitations by and now supports **raycasting against robot parts and other moving entities**. --- ## Usage For a quick demo, run: ```bash python scripts/demos/sensors/multi_mesh_raycaster.py --num_envs 16 --asset_type <allegro_hand|anymal_d|multi> ``` <img width="2882" height="804" alt="demo image" src="https://github.com/user-attachments/assets/a019e9d4-e991-4ca2-a94e-8cdba790f26d" /> ### Drop-in replacement Example change to migrate from `RayCasterCfg` to `MultiMeshRayCasterCfg`: ```diff - ray_caster_cfg = RayCasterCfg( + ray_caster_cfg = MultiMeshRayCasterCfg( prim_path="{ENV_REGEX_NS}/Robot", mesh_prim_paths=[ "/World/Ground", + MultiMeshRayCasterCfg.RaycastTargetCfg(target_prim_expr="{ENV_REGEX_NS}/Robot/LF_.*/visuals"), + MultiMeshRayCasterCfg.RaycastTargetCfg(target_prim_expr="{ENV_REGEX_NS}/Robot/RF_.*/visuals"), + MultiMeshRayCasterCfg.RaycastTargetCfg(target_prim_expr="{ENV_REGEX_NS}/Robot/LH_.*/visuals"), + MultiMeshRayCasterCfg.RaycastTargetCfg(target_prim_expr="{ENV_REGEX_NS}/Robot/RH_.*/visuals"), + MultiMeshRayCasterCfg.RaycastTargetCfg(target_prim_expr="{ENV_REGEX_NS}/Robot/base/visuals"), ], pattern_cfg=patterns.GridPatternCfg(resolution=resolution, size=(5.0, 5.0)), ) ``` --- ## Benchmarking & Validation To benchmark the new raycaster, run: ```bash python scripts/benchmarks/benchmark_ray_caster.py ``` Then plot the results with: ```bash python scripts/benchmarks/plot_raycast_results.py ``` This will generate outputs under: `outputs/benchmarks/raycast_benchmark...` ### Example plots <table style="border-collapse:collapse; width:100%;">  <tr> <td colspan="2" style="text-align:center; padding-bottom:8px;"> <img width="1000" height="500" alt="big image" src="https://github.com/user-attachments/assets/ff69253c-0329-4ab6-a944-7fcaac233923" /> </td> </tr>  <tr> <td style="text-align:center; padding-right:8px;"> <img width="500" height="500" alt="left image" src="https://github.com/user-attachments/assets/5375ed6c-f419-448d-ba25-759c1b16bcdd" /> </td> <td style="text-align:center;"> <img width="500" height="500" alt="right image" src="https://github.com/user-attachments/assets/bc36a0a6-aedd-4cdb-9909-9a05b1f2be0e" /> </td> </tr>  <tr> <td colspan="2" style="text-align:center; padding-top:8px;"> <img width="500" height="500" alt="bottom image" src="https://github.com/user-attachments/assets/67b75944-f64e-4c0b-b51f-aa20da3cf9b1" /> </td> </tr> </table> --- ## Type of Change * [x] New feature (non-breaking change which adds functionality) * [ ] This change requires a documentation update ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there  --------- Signed-off-by: renezurbruegg <[email protected]> Signed-off-by: Mayank Mittal <[email protected]> Signed-off-by: James Tigue <[email protected]> Signed-off-by: Pascal Roth <[email protected]> Signed-off-by: Kelly Guo <[email protected]> Co-authored-by: Pascal Roth <[email protected]> Co-authored-by: Pascal Roth <[email protected]> Co-authored-by: James Tigue <[email protected]> Co-authored-by: Mayank Mittal <[email protected]> Co-authored-by: Mayank Mittal <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Kelly Guo <[email protected]> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

# Description This MR adds the remaining comments in isaac-sim#3298 ## Type of change - Bug fix (non-breaking change which fixes an issue) ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Signed-off-by: renezurbruegg <[email protected]> Signed-off-by: Mayank Mittal <[email protected]> Signed-off-by: James Tigue <[email protected]> Signed-off-by: Pascal Roth <[email protected]> Signed-off-by: Kelly Guo <[email protected]> Co-authored-by: Pascal Roth <[email protected]> Co-authored-by: zrene <[email protected]> Co-authored-by: Pascal Roth <[email protected]> Co-authored-by: James Tigue <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Kelly Guo <[email protected]> Co-authored-by: zrene <[email protected]> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

…c-sim#3371) # Description This MR adds two functions to obtain the pose and scale of a prim respectively. This is needed for isaac-sim#3298. ## Type of change - New feature (non-breaking change which adds functionality) - This change requires a documentation update ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Signed-off-by: Mayank Mittal <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Kelly Guo <[email protected]>

@pascal-roth

# Description This PR introduces `MultiMeshRayCaster` and `MultiMeshRayCasterCamera`, an extension of the default `RayCaster` with the following enhancements: 1. **Raycasting against multiple target types** : Supports primitive shapes (spheres, cubes, …) as well as arbitrary meshes. 2. **Dynamic mesh tracking** : Keeps track of specified meshes, enabling raycasting against moving parts (e.g., robot links, articulated bodies, or dynamic obstacles). 3. **Memory-efficient caching** : Avoids redundant memory usage by caching and reusing duplicate meshes. This is joint work with @pascal-roth and @Mayankm96. The default `RayCaster` was limited to static environments and required manual handling of moving meshes, which restricted its use for robotics scenarios where robots or obstacles move dynamically. `MultiMeshRayCaster` addresses these limitations by and now supports **raycasting against robot parts and other moving entities**. --- ## Usage For a quick demo, run: ```bash python scripts/demos/sensors/multi_mesh_raycaster.py --num_envs 16 --asset_type <allegro_hand|anymal_d|multi> ``` <img width="2882" height="804" alt="demo image" src="https://github.com/user-attachments/assets/a019e9d4-e991-4ca2-a94e-8cdba790f26d" /> ### Drop-in replacement Example change to migrate from `RayCasterCfg` to `MultiMeshRayCasterCfg`: ```diff - ray_caster_cfg = RayCasterCfg( + ray_caster_cfg = MultiMeshRayCasterCfg( prim_path="{ENV_REGEX_NS}/Robot", mesh_prim_paths=[ "/World/Ground", + MultiMeshRayCasterCfg.RaycastTargetCfg(target_prim_expr="{ENV_REGEX_NS}/Robot/LF_.*/visuals"), + MultiMeshRayCasterCfg.RaycastTargetCfg(target_prim_expr="{ENV_REGEX_NS}/Robot/RF_.*/visuals"), + MultiMeshRayCasterCfg.RaycastTargetCfg(target_prim_expr="{ENV_REGEX_NS}/Robot/LH_.*/visuals"), + MultiMeshRayCasterCfg.RaycastTargetCfg(target_prim_expr="{ENV_REGEX_NS}/Robot/RH_.*/visuals"), + MultiMeshRayCasterCfg.RaycastTargetCfg(target_prim_expr="{ENV_REGEX_NS}/Robot/base/visuals"), ], pattern_cfg=patterns.GridPatternCfg(resolution=resolution, size=(5.0, 5.0)), ) ``` --- ## Benchmarking & Validation To benchmark the new raycaster, run: ```bash python scripts/benchmarks/benchmark_ray_caster.py ``` Then plot the results with: ```bash python scripts/benchmarks/plot_raycast_results.py ``` This will generate outputs under: `outputs/benchmarks/raycast_benchmark...` ### Example plots <table style="border-collapse:collapse; width:100%;">  <tr> <td colspan="2" style="text-align:center; padding-bottom:8px;"> <img width="1000" height="500" alt="big image" src="https://github.com/user-attachments/assets/ff69253c-0329-4ab6-a944-7fcaac233923" /> </td> </tr>  <tr> <td style="text-align:center; padding-right:8px;"> <img width="500" height="500" alt="left image" src="https://github.com/user-attachments/assets/5375ed6c-f419-448d-ba25-759c1b16bcdd" /> </td> <td style="text-align:center;"> <img width="500" height="500" alt="right image" src="https://github.com/user-attachments/assets/bc36a0a6-aedd-4cdb-9909-9a05b1f2be0e" /> </td> </tr>  <tr> <td colspan="2" style="text-align:center; padding-top:8px;"> <img width="500" height="500" alt="bottom image" src="https://github.com/user-attachments/assets/67b75944-f64e-4c0b-b51f-aa20da3cf9b1" /> </td> </tr> </table> --- ## Type of Change * [x] New feature (non-breaking change which adds functionality) * [ ] This change requires a documentation update ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there  --------- Signed-off-by: renezurbruegg <[email protected]> Signed-off-by: Mayank Mittal <[email protected]> Signed-off-by: James Tigue <[email protected]> Signed-off-by: Pascal Roth <[email protected]> Signed-off-by: Kelly Guo <[email protected]> Co-authored-by: Pascal Roth <[email protected]> Co-authored-by: Pascal Roth <[email protected]> Co-authored-by: James Tigue <[email protected]> Co-authored-by: Mayank Mittal <[email protected]> Co-authored-by: Mayank Mittal <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Kelly Guo <[email protected]> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

# Description This MR adds the remaining comments in isaac-sim#3298 ## Type of change - Bug fix (non-breaking change which fixes an issue) ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Signed-off-by: renezurbruegg <[email protected]> Signed-off-by: Mayank Mittal <[email protected]> Signed-off-by: James Tigue <[email protected]> Signed-off-by: Pascal Roth <[email protected]> Signed-off-by: Kelly Guo <[email protected]> Co-authored-by: Pascal Roth <[email protected]> Co-authored-by: zrene <[email protected]> Co-authored-by: Pascal Roth <[email protected]> Co-authored-by: James Tigue <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Kelly Guo <[email protected]> Co-authored-by: zrene <[email protected]> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

pascal-roth and others added 30 commits August 9, 2025 08:27

forgotten merge

d13331c

revert classvar changes and apply formatter

3f5d008

revert device change in ray caster

5decd09

update tests from unittest to pytest

f585a31

add initial benchmark scripts and formatter

7242ea2

Fix check script

2c6c607

Manually merging pascal-roth changes from static vars branch

86418d3

remove wrongfully pushed wheels

fa6b498

Merge changes from isaac-sim#90 - dropping support for cache_combined…

8beb0d2

…_meshes for now

add memory benchmark

5ca6e6e

fix garbage collection issue

eefcb3a

Add initial offset if mesh transform track is set to false

6eda9c8

avoid visualization when ray hits is not instantiated yet

0712ee5

Add resolve world pose

004184e

Change benchmark to sphere. Add different amount of faces and multi i…

9287efe

…nstances to benchmark. Fix callback issues for mulit mesh

fix local offsets

f80e384

add example allegro hand

bcd747a

Fix local offsets

4f6c772

Added regex lookups

f98150e

Add num assets flag.

5940b98

Add anymal demo

500931c

Add scale randomizations. Drop support for instancer meshes

586858c

add shapes

4d167ec

update benchmark scripts for different use cases - WIP

5b8563d

cleanup tests

17b5819

Add is shared functionality back

b3747c4

greptile-apps bot reviewed Nov 6, 2025

View reviewed changes

Merge branch 'main' into feature/multi-mesh-ray-caster

e786cfe

greptile-apps bot reviewed Nov 11, 2025

View reviewed changes

pascal-roth reviewed Nov 18, 2025

View reviewed changes

source/isaaclab/isaaclab/sensors/ray_caster/multi_mesh_ray_caster.py Outdated Show resolved Hide resolved

pascal-roth and others added 3 commits November 18, 2025 23:28

Merge branch 'main' into feature/multi-mesh-ray-caster

9de7ddf

remove omni.log, fix env_ids bug, formatter

d8be9e2

Update source/isaaclab/isaaclab/sensors/ray_caster/multi_mesh_ray_cas…

1c546f0

…ter.py Signed-off-by: Pascal Roth <[email protected]>

pascal-roth added 2 commits November 19, 2025 00:15

rename target_prim_expr to prim_expr

5cacc70

Merge branch 'feature/multi-mesh-ray-caster' of github.com:renezurbru…

76d2353

…egg/IsaacLab into feature/multi-mesh-ray-caster

pascal-roth approved these changes Nov 18, 2025

View reviewed changes

kellyguo11 added 2 commits December 10, 2025 07:57

Merge branch 'main' into feature/multi-mesh-ray-caster

919b39e

Signed-off-by: Kelly Guo <[email protected]>

Clean up imports in ray_caster_camera.py

fcd9110

Removed unused imports for cleaner code. Signed-off-by: Kelly Guo <[email protected]>

Mayankm96 changed the title ~~Adds Raycaster with tracking for Dynamic Meshes~~ Adds Raycaster with tracking for dynamic meshes Dec 16, 2025

Mayankm96 mentioned this pull request Dec 16, 2025

Addresses docs related issues from MultiMeshRayCaster #4222

Merged

7 tasks

Mayankm96 merged commit d4d8f70 into isaac-sim:main Dec 16, 2025
10 of 13 checks passed

github-project-automation bot moved this from In review to Done in Isaac Lab Dec 16, 2025

Adds Raycaster with tracking for dynamic meshes #3298

Adds Raycaster with tracking for dynamic meshes #3298

Uh oh!

Conversation

renezurbruegg commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Usage

Drop-in replacement

Benchmarking & Validation

Example plots

Type of Change

Checklist

Uh oh!

greptile-apps bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Greptile Overview

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Sequence Diagram

Additional Comments (1)

Uh oh!

kellyguo11 commented Nov 7, 2025

Uh oh!

greptile-apps bot commented Nov 11, 2025

Greptile Overview

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Sequence Diagram

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

XiaoYi-Wei commented Nov 12, 2025

Simulation startup becomes extremely slow when using the RayCaster sensor with large numbers of environments

Description

Steps to Reproduce

Observed Behavior

Expected Behavior

Environment

Uh oh!

XiaoYi-Wei commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Suggested Fix

Uh oh!

Uh oh!

pascal-roth commented Nov 18, 2025

Uh oh!

pascal-roth left a comment

Choose a reason for hiding this comment

Uh oh!

pascal-roth commented Nov 19, 2025

Uh oh!

kellyguo11 commented Dec 10, 2025

Uh oh!

Mayankm96 commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

renezurbruegg commented Aug 28, 2025 •

edited

Loading

greptile-apps bot left a comment •

edited

Loading

XiaoYi-Wei commented Nov 13, 2025 •

edited

Loading

Mayankm96 commented Dec 16, 2025 •

edited

Loading