CUDA out of memory

[joshkimchi]

Hey, I am running the jetson container live llava script now and i am hit with CUDA out of memory error
ValueError: Error when loading parameters from params_shard_26.bin: [12:45:06] /opt/mlc-llm/3rdparty/tvm/src/runtime/cuda/cuda_device_api.cc:138: InternalError: Check failed: (e == cudaSuccess || e == cudaErrorCudartUnloading) is false: CUDA: out of memory

i run tegrastats along while i run the script but tegrastats doesnt show that RAM usage is filled
jetson-containers run $(autotag nano_llm)
python3 -m nano_llm.agents.video_query --api=mlc
–model Efficient-Large-Model/VILA-2.7b
–max-context-len 256
–max-new-tokens 32
–video-input /dev/video0
–video-output webrtc://@:8554/output

why do i get this error and how should i solve it please

[AastaLLL]

Hi,

Could you share the complete error with us?
Please also monitor the system status with tegrastats and share it with us.

Thanks.

[joshkimchi]

hello this is the error message:
15:48:13 | INFO | Press CTRL+C to quit
[gstreamer] gstCamera – onPreroll
[gstreamer] gstBufferManager recieve caps: video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)2:4:5:1
[gstreamer] gstBufferManager – recieved first frame, codec=raw format=yuyv width=1280 height=720 size=1843200
[cuda] allocated 4 ring buffers (1843200 bytes each, 7372800 bytes total)
[cuda] allocated 4 ring buffers (8 bytes each, 32 bytes total)
[gstreamer] gstreamer changed state from READY to PAUSED ==> mysink
[gstreamer] gstreamer message async-done ==> pipeline0
[gstreamer] gstreamer message latency ==> mysink
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> mysink
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> pipeline0
[cuda] allocated 4 ring buffers (2764800 bytes each, 11059200 bytes total)
[cuda] allocated 2 ring buffers (1382400 bytes each, 2764800 bytes total)
[gstreamer] gstEncoder – starting pipeline, transitioning to GST_STATE_PLAYING
[gstreamer] gstreamer changed state from NULL to READY ==> fakesink0
[gstreamer] gstreamer changed state from NULL to READY ==> queue0
[gstreamer] gstreamer changed state from NULL to READY ==> videotee
[gstreamer] gstreamer changed state from NULL to READY ==> capsfilter2
[gstreamer] gstreamer changed state from NULL to READY ==> rtph264pay0
[gstreamer] gstreamer changed state from NULL to READY ==> capsfilter1
[gstreamer] gstreamer changed state from NULL to READY ==> encoder
[gstreamer] gstreamer changed state from NULL to READY ==> mysource
[gstreamer] gstreamer changed state from NULL to READY ==> pipeline1
[gstreamer] gstreamer stream status CREATE ==> src
[gstreamer] gstreamer changed state from READY to PAUSED ==> queue0
[gstreamer] gstreamer changed state from READY to PAUSED ==> videotee
[gstreamer] gstreamer changed state from READY to PAUSED ==> capsfilter2
[gstreamer] gstreamer stream status ENTER ==> src
[gstreamer] gstreamer changed state from READY to PAUSED ==> rtph264pay0
[gstreamer] gstreamer changed state from READY to PAUSED ==> capsfilter1
[gstreamer] gstreamer changed state from READY to PAUSED ==> encoder
[gstreamer] gstreamer stream status CREATE ==> src
[gstreamer] gstreamer changed state from READY to PAUSED ==> mysource
[gstreamer] gstreamer changed state from READY to PAUSED ==> pipeline1
[gstreamer] gstreamer message new-clock ==> pipeline1
[gstreamer] gstreamer stream status ENTER ==> src
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> queue0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> videotee
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> capsfilter2
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> rtph264pay0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> capsfilter1
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> encoder
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> mysource
[gstreamer] gstreamer message qos ==> v4l2src0
[gstreamer] gstEncoder – new caps: video/x-raw, width=1280, height=720, format=(string)I420, framerate=30/1
[gstreamer] gstreamer message latency ==> encoder
[gstreamer] gstreamer message stream-start ==> pipeline1
[gstreamer] gstreamer fakesink0 taglist, encoder=(string)x264, encoder-version=(uint)163, maximum-bitrate=(uint)4096000, nominal-bitrate=(uint)4096000;
[gstreamer] gstreamer changed state from READY to PAUSED ==> fakesink0
[gstreamer] gstreamer message async-done ==> pipeline1
[gstreamer] gstreamer message latency ==> fakesink0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> fakesink0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> pipeline1
NvMapMemAllocInternalTagged: 1075072515 error 12
NvMapMemHandleAlloc: error 0
NvMapMemAllocInternalTagged: 1075072515 error 12
NvMapMemHandleAlloc: error 0
[15:48:16] /opt/mlc-llm/3rdparty/tvm/src/runtime/memory/pooled_allocator.h:65: Warning: PooledAllocator got InternalError during allocation: InternalError: Check failed: (e == cudaSuccess || e == cudaErrorCudartUnloading) is false: CUDA: out of memory
[15:48:16] /opt/mlc-llm/3rdparty/tvm/src/runtime/memory/pooled_allocator.h:66: Warning: Trying to release all unused memory and reallocate…
NvMapMemAllocInternalTagged: 1075072515 error 12
NvMapMemHandleAlloc: error 0
CUDA error: out of memory

it appears at different times, sometimes when loading the model and sometimes when the gstreamer is loading. looks to be an issue with the memory of it.

here is the tegrastats
10-27-2025 15:48:42 RAM 6020/7620MB (lfb 1x4MB) SWAP 151/20194MB (cached 0MB) CPU [2%@1728,0%@1728,0%@1728,0%@1728,11%@1728,0%@1728] GR3D_FREQ 0% [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] VDD_IN 7791mW/7924mW VDD_CPU_GPU_CV 1737mW/1797mW VDD_SOC 2649mW/2670mW
10-27-2025 15:48:43 RAM 6022/7620MB (lfb 1x4MB) SWAP 151/20194MB (cached 0MB) CPU [3%@1728,4%@1728,1%@1728,4%@1728,11%@1728,0%@1728] GR3D_FREQ 3% [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] VDD_IN 7989mW/7927mW VDD_CPU_GPU_CV 1855mW/1800mW VDD_SOC 2689mW/2671mW
10-27-2025 15:48:44 RAM 6021/7620MB (lfb 1x4MB) SWAP 151/20194MB (cached 0MB) CPU [2%@1728,2%@1728,0%@1728,0%@1728,10%@1728,0%@1728] GR3D_FREQ 0% [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] VDD_IN 7870mW/7924mW VDD_CPU_GPU_CV 1737mW/1797mW VDD_SOC 2649mW/2670mW
10-27-2025 15:48:45 RAM 6022/7620MB (lfb 1x4MB) SWAP 151/20194MB (cached 0MB) CPU [2%@1728,4%@1728,1%@1728,1%@1728,11%@1728,0%@1728] GR3D_FREQ 0% [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] VDD_IN 7910mW/7924mW VDD_CPU_GPU_CV 1816mW/1798mW VDD_SOC 2649mW/2669mW

[AastaLLL]

@joshkimchi

NvMapMemAllocInternalTagged: 1075072515 error 12
NvMapMemHandleAlloc: error 0
NvMapMemAllocInternalTagged: 1075072515 error 12
NvMapMemHandleAlloc: error 0

It looks like you hit an error related to the r36.4.7 upgrade.
We are actively working on fixing this issue and will keep you updated.

For more information, please check the topic below:

"unable to allocate CUDA0 buffer" after Updating Ubuntu Packages

After Ubuntu update it said, “Error: 500 Internal Server Error: llama runner process has terminated: error loading model: unable to allocate CUDA0 buffer llama_model_load_from_file_impl: failed to load model” I typically would run ollama from jetson-containers, and up until the recent update it worked fine. I tried to keep separation with the host not having the base system including packages that were used for models, but this effort seemed not to help. Since it was mentioned already that r…

Thanks.

[joshkimchi]

hey, ok. will be waiting for the update of issue! thanks @AastaLLL

[AastaLLL]

Hi,

Thanks for your patience.

There is some update about the root cause shared in the comment below:

"unable to allocate CUDA0 buffer" after Updating Ubuntu Packages

Hi, @all Thank you all for the testing and sharing. We are really sorry about the inconvenience that the r36.4.7 brings. Although our internal team is still working on the issue, here are some updates about the issue that we can share with you: The recent update (r38.2.1->r38.2.2, r36.4.4->r36.4.7, 35.6.2->r35.6.3) contains a security fix for CVE-2025-33182 & CVE-2025-33177: The patches can be found in the below comment (r35.6.3 version): The security fix adds a mechanism to prevent the…

Please note that we are still working on the fix.
Will keep you updated.

Thanks.