Semantic segmentation skipping some objects

Omniverse Synthetic Data Generation (SDG)
1

Hi, I’m trying to create a semantic segmentation dataset of an arena made of tiles.
I generate an arena like this:

def generate_arena():
    stage = stage_utils.get_current_stage()
    #groundPlane = stage.GetPrimAtPath("/World/GroundPlane")
    #if not groundPlane.IsValid():
        #plane = GroundPlane(prim_path="/World/GroundPlane", size=100, color=np.array([0.5, 0.5, 0.5]), z_position=0)
    
    # delete the arena if it exists
    arena_prim = stage.GetPrimAtPath("/World/Arena")
    if arena_prim.IsValid():
        print("Deleting existing arena")
        stage.RemovePrim("/World/Arena")

    usd_files = [
        #"file:/home/priba/Documents/AI/Robots/Tiles/EndPBR.usd",
        #"file:/home/priba/Documents/AI/Robots/Tiles/StartPBR.usd",
        "file:/home/priba/Documents/AI/Robots/Tiles/TCrossingPBR.usd",
        "file:/home/priba/Documents/AI/Robots/Tiles/TCrossingOneSidePBR.usd",
        "file:/home/priba/Documents/AI/Robots/Tiles/TCrossingNoMarkerPBR.usd",
        "file:/home/priba/Documents/AI/Robots/Tiles/Turn1PBR.usd",
        "file:/home/priba/Documents/AI/Robots/Tiles/Turn2PBR.usd",
        "file:/home/priba/Documents/AI/Robots/Tiles/XCrossingPBR.usd",
        "file:/home/priba/Documents/AI/Robots/Tiles/XCrossingOneSidePBR.usd",
        "file:/home/priba/Documents/AI/Robots/Tiles/XCrossingOneMarkerPBR.usd",
        "file:/home/priba/Documents/AI/Robots/Tiles/XCrossingNoMarkerPBR.usd",
        "file:/home/priba/Documents/AI/Robots/Tiles/ZigZag1PBR.usd",
        "file:/home/priba/Documents/AI/Robots/Tiles/ZigZag2PBR.usd",
    ]

    asset_prims = []
    local_scale = 0.1

    gridsize = 9

    for i, (x, y) in enumerate(np.ndindex(gridsize, gridsize)):
        pos_x = (x - gridsize//2) * 30
        pos_z = (y - gridsize//2) * 30
        pos_y = 1
        angle = random.choice([0, 90, 180, 270])

        prim_path = f"/World/Arena/Asset_{i}"

        usd_path = random.choice(usd_files)

        print(f"Adding asset at {prim_path} with USD path {usd_path} on {pos_x}, {pos_y}, {pos_z}")

        quaternion = get_quaternion_from_euler(0, np.deg2rad(angle), np.pi/2)
        
        stage_utils.add_reference_to_stage(usd_path, prim_path)
        XFormPrim(prim_path).set_world_pose([pos_x, pos_y, pos_z], quaternion)
        XFormPrim(prim_path).set_local_scale([local_scale, local_scale, local_scale])
        # get child of prim path LinePBR material. Take its shader and randomize the diffuse color
        xform_prim = stage.GetPrimAtPath(prim_path)
        child = xform_prim.GetChildren()[0]  # Assuming the first child is the one we want
        
        mat_prim = stage.GetPrimAtPath(child.GetPath().AppendPath("LinePBR"))
        print(prim_path + "/LinePBR")
        print(mat_prim)
        col = random.uniform(0.05, 0.4)
        rand_color = Gf.Vec3f(col, col, col)
        omni.usd.create_material_input(mat_prim, "diffuse_color_constant", rand_color, Sdf.ValueTypeNames.Color3f)


        asset_prims.append(prim_path)
    print("arena generated!")

The usd files have the semantic ids set before hand and when i check the semantic view everything seems to be labeled correctly.
I then add a bunch of random objects, lights, textures etc. and randomize their positions on the arena. They often clip through the tiles but don’t seem to cause issues.
I then noticed that in my generated dataset some of the tiles get completely omitted. The object is there in the rgb scene view and the neighbor objects appear in the segmented view but that object doesn’t. I managed to track down which one it is to move it around and still nothing, even though other copies of that tile show up fine.
I then copied that exact object and moved it over and that copy gets segmented properly.
This is strange behavior and seems like a bug. Am I doing something wrong? What could cause this?
I’m on Isaac Sim 4.2.0

2

@petar.slavicek i am just another user but was curious whether the same behavior happens in 4.5 or 5.0?

3

Just tried it with 4.5.0 and have the same problem.
Haven’t tried 5.0 yet

4

isaacreplicator
isaacreplicator1119×358 439 KB

Here’s a screenshot.
One of the curved tiles is an exact copy of the other, meaning that i took a tile that didn’t segment properly , selected ‘duplicate’ in the hierarchy and moved it over. It is segmented properly. The other square tile is also not detected but if i duplicate it the duplicate is detected properly.

I can’t make a dataset like this if i have false negatives all over the place.

5

the case that i have seen and is a known issue in 4.5 is the opposite of what you are seeing. specifically, it’s the duplicates that cannot be recognized by semantic segmentation:

is there any error in the console at all?

6

I’m getting this error when loading other assets in the scene

And this warning when loading the tiles in:

2025-06-25 21:08:11  [Warning] [omni.usd] Coding Error: in _Get at line 3003 of /builds/omniverse/usd-ci/USD/pxr/usdImaging/usdImaging/delegate.cpp -- Failed verification: ' prim '

The issue doesn’t appear immediately, it starts happening after about 30 steps of replicator.
Here’s another image that shows the issue. All the tiles here are loaded from the same usd file

isaacsimbug
isaacsimbug1905×935 116 KB

The weird thing is when i reload the tiles after this starts happening, the tiles on the same spots are still invisible. Meaning i delete the arena and then regenerate it (in this case with the same tile the whole time) and there are missing lines or green markers in the same spot. I thought that maybe some of the objects have bounding boxes that cover the whole object so the renderer skips it but even if i take that tile and move it somewhere it’s still not detected. That doesn’t make any sense to me…

The linked issue doesn’t seem to be the same thing. My issue is present in both 4.2 and 4.5