Faulty semantic segmentation data?

Omniverse Isaac Sim
1

Isaac Sim Version

4.2.0

Topic Description

Detailed Description

I’m using isaac sim to save semantic segmentation data:

def save_segmentation(self, data, file_name):
        _segmented_data = data["semantic_segmentation"]["data"]
        _semantics_in_frame = data["semantic_segmentation"]["info"]["idToLabels"]
        file_path = FileHelper.get_path(self.directory_path, file_name + ".png")
        self._write_semantic_segmentation(_segmented_data, _semantics_in_frame ,file_path)

I’m seeing something strange at this point which i’m not sure how can be possible. If I get the uniqe ids from the segmented data, i get an array of ids which contain values that are NOT IN semantics_in_frame:

def save_segmentation(self, data, file_name):

    _segmented_data = data["semantic_segmentation"]["data"]

    _unique_ids = np.unique(_segmented_data)

    _semantics_in_frame = data["semantic_segmentation"]["info"]["idToLabels"]

    file_path = FileHelper.get_path(self.directory_path, file_name + ".png")

    self._write_semantic_segmentation(_segmented_data, _semantics_in_frame ,file_path)

image
image464×111 5.39 KB

where as:

image
image418×265 6.78 KB

But how is this possible? how can data[“semantic_segmentation”][“data”] have semantics ids which are not in data[“semantic_segmentation”][“info”][“idToLabels”]??? these are both values of semantic ids in the current frame, or am i confused?

Thank you,

Danielle

2

Thank you for sharing the details of your issue. I’ve tested this on Isaac Sim 5.0.0 but was unable to reproduce the problem. To help us investigate further, could you please try updating to the latest General Availability (GA) version of Isaac Sim and see if the issue persists?

If you’re still encountering the faulty semantic segmentation data after upgrading, it would be very helpful if you could provide:

  • The detailed steps to reproduce the problem
  • Any relevant scripts or code snippets

This information will allow us to better understand and address the issue. Thank you!

3

Hi There @VickNV Can you please confirm that my understanding of the variables is correct and this is indeed an issue with in Isaac Sim?

That is, the two variables:

data["semantic_segmentation"]["data"]

and

data["semantic_segmentation"]["info"]["idToLabels"]

should always contain exactly the same IDs?

Thank you

4

Hi Danielle,

that should not be the case, I created a repro script with prims in view, outside of view, labeled and unlabeled, and it seems I am getting correct results:

import asyncio
import numpy as np
import omni.usd
import omni.replicator.core as rep
from pxr import Sdf


async def run_example_async():
    omni.usd.get_context().new_stage()
    stage = omni.usd.get_context().get_stage()
    dome_light = stage.DefinePrim("/World/DomeLight", "DomeLight")
    dome_light.CreateAttribute("inputs:intensity", Sdf.ValueTypeNames.Float).Set(1000.0)
    
    # Objects in view
    cube_front = rep.create.cube(
        position=(0, 0, 0), 
        scale=0.5, 
        semantics=[("class", "cube"), ("location", "in_view")],
        name="cube_front"
    )
    sphere_front = rep.create.sphere(
        position=(0, 1, 0), 
        scale=0.5, 
        semantics=[("class", "sphere"), ("location", "in_view")],
        name="sphere_front"
    )
    # Unlabeled object in view
    cone_unlabeled = rep.create.cone(
        position=(0, -1, 0), 
        scale=0.5,
        name="cone_unlabeled"
    )
    
    # Outside view
    cube_behind = rep.create.cube(
        position=(0, 0, -10), 
        scale=1.0, 
        semantics=[("class", "cube"), ("location", "behind_camera")],
        name="cube_behind"
    )    
    sphere_side = rep.create.sphere(
        position=(10, 0, 0), 
        scale=1.0, 
        semantics=[("class", "sphere"), ("location", "outside_view")],
        name="sphere_side"
    )

    cam = rep.create.camera(position=(0, 0, 5), look_at=(0, 0, 0))
    rp = rep.create.render_product(cam, resolution=(1024, 1024))    
    semantic_segmentation_annot = rep.annotators.get("semantic_segmentation")
    semantic_segmentation_annot.attach(rp)    
    await rep.orchestrator.step_async()
    semantic_data = semantic_segmentation_annot.get_data()
   
    print(f'semantic_data["data"]: shape={semantic_data["data"].shape}, dtype={semantic_data["data"].dtype}')
    print(f'np.unique(semantic_data["data"])= {np.unique(semantic_data["data"])}')
    print(f'semantic_data["info"]={semantic_data["info"]}')
    print("idToLabels mapping:")
    for sem_id, labels in semantic_data['info']['idToLabels'].items():
        print(f'  ID {sem_id}: {labels}')    
    
    semantic_segmentation_annot.detach()
    rp.destroy()

asyncio.ensure_future(run_example_async())

with output:

semantic_data["data"]: shape=(1024, 1024), dtype=uint32
np.unique(semantic_data["data"])= [0 1 2 3]
semantic_data["info"]={'idToLabels': {'0': {'class': 'BACKGROUND'}, '1': {'class': 'UNLABELLED'}, '2': {'location': 'in_view', 'class': 'sphere'}, '3': {'location': 'in_view', 'class': 'cube'}}}
idToLabels mapping:
  ID 0: {'class': 'BACKGROUND'}
  ID 1: {'class': 'UNLABELLED'}
  ID 2: {'location': 'in_view', 'class': 'sphere'}
  ID 3: {'location': 'in_view', 'class': 'cube'}

Could you check your output? Or against your scene?

If you still see the issue, would it be possible to provide a repro scene? Thanks!

5

Hello!

We noticed that this topic hasn’t received any recent responses, so we are closing it for now to help keep the forum organized.

If you’re still experiencing this issue or have additional questions, please feel free to create a new topic with updated details. When doing so, we recommend mentioning or linking to this original topic in your new post—this helps provide context and makes it easier for others to assist you.

Thank you for being part of the NVIDIA Isaac Sim community.

Best regards,
The NVIDIA Isaac Sim Forum Team