Multiple simultaneous AV1 streams

[etay]

Hi

i am trying to encode two different AV1 video streams simultaneously from two different processes using the jetson multimedia api.

however, i am running into an issue in which due to the precense of the other stream, the capture plane dequeue callback of one stream might only be called over a second after the frame is sent to the hardware encoder

how can i prevent this? i want each frame of that specific stream out of the encoder before the next frame arrives

sincerely

etay markman

[carolyuu]

*** Please note that this reply is generated by LLM automatically ***

To address your issue, I’ll provide a comprehensive answer, comparing the relevant information from the context.

Understanding the Issue

You’re experiencing a delay in the capture plane dequeue callback of one AV1 video stream when another stream is present. This delay is causing the frame to be processed by the hardware encoder only after a second, which is not desirable.

Jetson AGX Orin Series Capabilities

The Jetson AGX Orin series supports a maximum of 16 virtual channels with ISP or 24 virtual channels without ISP. This means that multiple camera streams can be processed simultaneously using virtual channels.

GMSL Camera Framework

The GMSL Camera Framework is designed to work with the Jetson AGX Orin series and supports the Gigabit Multimedia Serial Link (GMSL) protocol. The framework allows for the configuration of virtual channels, which can be used to multiplex multiple camera streams over a single CSI connection.

Serializer-Deserializer (SerDes) Links

The reference GMSL module uses the CSI interface, and the software framework can be used as a reference for developing SerDes links other than GMSL.

Possible Solution

To prevent the delay in the capture plane dequeue callback, you can try the following:

1. Use Virtual Channels: Configure virtual channels for each AV1 video stream using the GMSL Camera Framework. This will allow each stream to be processed independently, reducing the likelihood of delays.
2. Increase the Number of Virtual Channels: If you’re already using virtual channels, try increasing the number of virtual channels allocated to each stream. This might help reduce the contention between streams and minimize delays.
3. Optimize the Encoder Configuration: Review the encoder configuration for each stream and optimize it for better performance. This might involve adjusting parameters such as bitrate, resolution, or frame rate.
4. Use a Separate Encoder Instance: If possible, use a separate encoder instance for each stream. This will ensure that each stream is processed independently, without any contention or delays.
5. Monitor System Resources: Keep an eye on system resources such as CPU, memory, and bandwidth. If any of these resources are constrained, it might be causing the delay. Consider optimizing your system configuration or adding more resources if necessary.

Comparison with Other Jetson Modules

The Jetson AGX Orin series has a more powerful ISP and can support more virtual channels compared to other Jetson modules like the Jetson Xavier NX or Jetson Nano. However, the issue you’re facing might not be specific to the Jetson AGX Orin series, and the solutions mentioned above can be applied to other Jetson modules as well.

In summary, by using virtual channels, optimizing encoder configuration, and monitoring system resources, you should be able to prevent the delay in the capture plane dequeue callback and ensure that each frame of the specific stream is processed by the hardware encoder before the next frame arrives.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

[etay]

@carolyuu

Gigabit Multimedia Serial Link

Hi carol

unfortunately, we are not using GMSL in our project, so most of your automatic LLM reply doesn’t really work for us.

we use two seperate programs made with the jetson multimedia api similar to 01_video_encode and run them simultaneously(this checks the “separate encoder instances” part), expecting it to look like the streams are encoded in true parallel. unfortunately, we experience the delay said above

so we wonder if there is something we miss out in when we set the encoder parameters that causes the delay.

you can use this page and the samples in the multimedia API to understand more

Jetson Linux API Reference: V4L2 Video Encoder | NVIDIA Docs

thanks anyways

etay

[DaneLLL]

Hi,
Please enable the option and see if there is further enhancement:

        --max-perf            Enable maximum Performance

And please also run $ sudo jetson_clocks. There is single hardware engine NVNEC handing multiple enoding tasks, so it is possible there is delay in concurrent requests. Please enable the engines at maximum throughput.