Skip to content

Conversation

@Mayankm96
Copy link
Contributor

@Mayankm96 Mayankm96 commented Apr 14, 2023

Description

This PR updates the IsaacEnv class to support different rendering modes (as suggested by OpenAI Gym documentation). This allows users to also record videos using the environment by using the gym.wrapper.RecordVideo wrapper.

To enable rendering of the viewport, the user needs to set viewport=True while initializing the environment. This flag will enable extensions required to create the renderer. It is to note that enabling the viewport will lead to a certain performance drop.

As an example of creating the environment while running headless but enabling the rendering:

# name of the task
task_name = "Isaac-Reach-Franka-v0"
# parse configuration
env_cfg = parse_env_cfg(task_name, use_gpu=True, num_envs=1024)
# create environment
env = gym.make(task_name, cfg=env_cfg, headless=True, viewport=True)

# wrap environment to record videos
env = gym.wrappers.RecordVideo(
    env, "videos", step_trigger=lambda step: step % 1000 == 0, video_length=250
)

Additionally, all workflows train.py have been updated to take the command line argument video that performs the above wrapping and saves videos into the experiment logging directory.

python source/standalone/workflows/rsl_rl/train.py --task Isaac-Reach-Franka-v0 --headless --video

Fixes OIGE #17

Type of change

  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

Screenshots

Running the above command will generate a video similar to the follows. The resolution of the image can be changed through the IsaacEnvCfg.ViewerCfg.resolution. By default it is 1280 x 720.

rl-video-step-1500.mp4

Checklist

  • I have run the pre-commit checks with ./orbit.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the changelog and the corresponding version in the extension's config/extension.toml file

@Mayankm96 Mayankm96 added the enhancement New feature or request label Apr 14, 2023
@Mayankm96 Mayankm96 force-pushed the feature/render-modes branch from a4c346c to 0f15911 Compare April 14, 2023 14:15
@Mayankm96 Mayankm96 force-pushed the feature/render-modes branch from f1ca2f9 to 5117a3c Compare April 14, 2023 14:23
@Mayankm96
Copy link
Contributor Author

Mayankm96 commented Apr 14, 2023

Currently this PR is generating the following error which needs to be fixed before merging it.

2023-04-14 14:26:03 [12,409ms] [Error] [carb.events.python] AttributeError: 'PlayButtonGroup' object has no attribute '_play_button'

At:
  /home/mayank/packman-repo/chk/isaac-sim-standalone/2022.2.1-rc.14+2022.2.494.70497c06.tc.linux-x86_64.release/kit/exts/omni.kit.widget.toolbar/omni/kit/widget/toolbar/builtin_tools/play_button_group.py(129): _on_timeline_event
  /home/mayank/packman-repo/chk/isaac-sim-standalone/2022.2.1-rc.14+2022.2.494.70497c06.tc.linux-x86_64.release/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py(549): play
  /home/mayank/packman-repo/chk/isaac-sim-standalone/2022.2.1-rc.14+2022.2.494.70497c06.tc.linux-x86_64.release/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py(387): initialize_physics
  /home/mayank/packman-repo/chk/isaac-sim-standalone/2022.2.1-rc.14+2022.2.494.70497c06.tc.linux-x86_64.release/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py(408): reset
  source/standalone/demo/play_empty.py(69): main
  source/standalone/demo/play_empty.py(88): <module>

The issue is something to do with the enabled extensions. Here's the minimal repro:

import os

from omni.isaac.kit import SimulationApp

# launch omniverse app
app_experience = f"{os.environ['EXP_PATH']}/omni.isaac.sim.python.gym.headless.render.kit"
config = {"headless": True}
simulation_app = SimulationApp(config, experience=app_experience)

import omni.isaac.core.utils.prims as prim_utils
from omni.isaac.core.simulation_context import SimulationContext
from omni.isaac.core.utils.viewports import set_camera_view
from omni.isaac.core.world import World


def main():
    """Spawns lights in the stage and sets the camera view."""

    # Load kit helper
    sim = World(physics_dt=0.01, rendering_dt=0.01, backend="torch")
    # Set main camera
    set_camera_view([2.5, 2.5, 2.5], [0.0, 0.0, 0.0])

    # Spawn things into stage
    # Lights-1
    prim_utils.create_prim(
        "/World/Light/GreySphere",
        "SphereLight",
        translation=(4.5, 3.5, 10.0),
        attributes={"radius": 2.5, "intensity": 600.0, "color": (0.75, 0.75, 0.75)},
    )
    # Lights-2
    prim_utils.create_prim(
        "/World/Light/WhiteSphere",
        "SphereLight",
        translation=(-4.5, 3.5, 10.0),
        attributes={"radius": 2.5, "intensity": 600.0, "color": (1.0, 1.0, 1.0)},
    )

    # Play the simulator
    sim.reset()
    # Now we are ready!
    print("[INFO]: Setup complete...")

    # Simulate physics
    while simulation_app.is_running():
        # If simulation is stopped, then exit.
        if sim.is_stopped():
            break
        # If simulation is paused, then skip.
        if not sim.is_playing():
            sim.step(render=not simulation_app.config["headless"])
            continue
        # perform step
        sim.step()


if __name__ == "__main__":
    # Run empty stage
    main()
    # Close the simulator
    simulation_app.close()

@Mayankm96 Mayankm96 self-assigned this Apr 14, 2023
@Mayankm96 Mayankm96 force-pushed the feature/render-modes branch from 166f83f to b6e571d Compare April 27, 2023 18:52
@Mayankm96 Mayankm96 merged commit 8e42f05 into main Apr 27, 2023
@Mayankm96 Mayankm96 deleted the feature/render-modes branch April 27, 2023 19:34
pascal-roth added a commit that referenced this pull request Aug 18, 2025
Change Mulit-mesh raycaster and raycaster camera to own files, restore
the ones of main to simplify the merge.

NOTE: test of the camera is currently failing, similar as on public main
at that time, should be fixed after update to latest main

- Breaking change (fix or feature that would cause existing
functionality to not work as expected)

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [ ] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Video recording during train

2 participants