Camera data in numpy format in Isaac sim

[syoungk20]

Hi,

How can I get camera image (color, depth) to numpy format?

I tried a code below,

sd = omni.syntheticdata._syntheticdata
sdi = sd.acquire_syntheticdata_interface()

def _get_sensor_data(sensor, dtype):
    width = sdi.get_sensor_width(sensor)
    height = sdi.get_sensor_height(sensor)
    row_size = sdi.get_sensor_ros_size(sensor)
    get_sensor = {
            "uint32": sdi.get_sensor_host_uint32_texture_array,
            "float": sdi.get_sensor_host_float_texture_array,
    }
    return get_sensor[dtype](sensor, width, height, row_size)

depth = _get_sensor_data(sd.SensorType.Depth, "float")

which I found that has error with isaac sim 2022.1
AttributeError: ‘omni.syntheticdata._syntheticdata.SyntheticData’ object has no attribute ‘get_sensor_width’

How can I deal with this?

[ahaidu]

See if the snippets and the code from this tutorial would help you: 4. Online Generation — Omniverse Robotics documentation

the important code snippets would be:

EDIT using new API (Replicator):

RESOLUTION = (1024, 1024)
[..]
self.camera = self.rep.create.camera()
self.render_product = self.rep.create.render_product(self.camera, RESOLUTION)

# Setup annotators that will report groundtruth
self.rgb = self.rep.AnnotatorRegistry.get_annotator("rgb")
self.bbox_2d_tight = self.rep.AnnotatorRegistry.get_annotator("bounding_box_2d_tight")
self.instance_seg = self.rep.AnnotatorRegistry.get_annotator("instance_segmentation")
self.rgb.attach(self.render_product)
self.bbox_2d_tight.attach(self.render_product)
self.instance_seg.attach(self.render_product)
[..]
# Step - Randomize and render
self.rep.orchestrator.step()

# Collect Groundtruth
gt = {
    "rgb": self.rgb.get_data(device="gpu"),
    "boundingBox2DTight": self.bbox_2d_tight.get_data(device="gpu"),
    "instanceSegmentation": self.instance_seg.get_data(device="gpu"),
}

# RGB
# Drop alpha channel
image = self.wp.to_torch(gt["rgb"])[..., :3]
[..]

Old API:

RESOLUTION = (1024, 1024)
[..]
self.camera = self.rep.create.camera()
self.render_product = self.rep.create.render_product(self.camera, RESOLUTION)
self.viewport = omni.kit.viewport_legacy.get_default_viewport_window()
[..]
# Collect Groundtruth
gt = self.sd_helper.get_groundtruth(["rgb", "boundingBox2DTight", "instanceSegmentation"], self.viewport)

# RGB
# Drop alpha channel
image = gt["rgb"][..., :3]