"Fix a reference leak of hw_frames_ctx and prepare for QSV" commit broke AMD Radeon VA-API KMS usage #804
Description
Is there an existing issue for this?
- I have searched the existing issues
Is your issue described in the documentation?
- I have read the documentation
Is your issue present in the nightly release?
- This issue is present in the nightly release
Describe the Bug
The pull request from #736 causes a startup crash on AMD Radeon cards using VA-API and KMS grabbing.
Git bisect pinpoints following commit:
$ git bisect good
44ad28ebf4a17779e483971e80c260c74e972244 is the first bad commit
commit 44ad28ebf4a17779e483971e80c260c74e972244
Author: Cameron Gutman <[email protected]>
Date: Tue Jan 10 13:52:15 2023 -0600
Fix a reference leak of hw_frames_ctx and prepare for QSV (#736)
src/platform/common.h | 9 +++-
src/platform/linux/cuda.cpp | 17 ++++---
src/platform/linux/vaapi.cpp | 11 +++--
src/platform/macos/nv12_zero_device.cpp | 2 +-
src/platform/macos/nv12_zero_device.h | 2 +-
src/platform/windows/display_vram.cpp | 87 ++++++++++++++++++++-------------
src/video.cpp | 33 ++++++-------
7 files changed, 92 insertions(+), 69 deletions(-)
The crash happens on a fedora 36 machine with an AMD Radeon RX6900XT, using VA-API and only when using KMS grabbing (ie. with setcap done).
Crash output:
./sunshine-0.17.0
[vt_software] -- [auto]
[amd_quality] -- [balanced]
[origin_web_ui_allowed] -- [lan]
[dwmflush] -- [enabled]
[gamepad] -- [x360]
[origin_pin_allowed] -- [pc]
[resolutions] -- [[
352x240,
480x360,
858x480,
1280x720,
1920x1080,
2560x1080,
3440x1440,
1920x1200,
3860x2160,
3840x1600
]]
[key_rightalt_to_key_win] -- [disabled]
[vt_realtime] -- [enabled]
[fec_percentage] -- [10]
[nv_coder] -- [auto]
[amd_coder] -- [auto]
[upnp] -- [disabled]
[nv_preset] -- [p4]
[nv_tune] -- [ull]
[vt_coder] -- [auto]
[amd_rc] -- [vbr_latency]
[hevc_mode] -- [0]
[nv_rc] -- [cbr]
[fps] -- [[10,30,60,90,120]]
[output_name] -- [0]
[min_log_level] -- [2]
[2023:01:18:15:10:38]: Error: Environment variable WAYLAND_DISPLAY has not been defined
[2023:01:18:15:10:38]: Info: Detecting connected monitors
[2023:01:18:15:10:38]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2023:01:18:15:10:38]: Info: Trying encoder [nvenc]
[2023:01:18:15:10:38]: Info: Screencasting with KMS
[2023:01:18:15:10:38]: Info: Found monitor for DRM screencasting
[2023:01:18:15:10:38]: Info: Color coding [Rec. 601]
[2023:01:18:15:10:38]: Info: Color range: [JPEG]
[2023:01:18:15:10:38]: Error: Failed to create a CUDA device: Operation not permitted
[2023:01:18:15:10:38]: Info: Screencasting with KMS
[2023:01:18:15:10:38]: Info: Found monitor for DRM screencasting
[2023:01:18:15:10:38]: Info: Color coding [Rec. 601]
[2023:01:18:15:10:38]: Info: Color range: [JPEG]
[2023:01:18:15:10:38]: Error: Failed to create a CUDA device: Operation not permitted
[2023:01:18:15:10:38]: Info: Screencasting with KMS
[2023:01:18:15:10:38]: Info: Found monitor for DRM screencasting
[2023:01:18:15:10:38]: Info: Color coding [Rec. 601]
[2023:01:18:15:10:38]: Info: Color range: [JPEG]
[2023:01:18:15:10:38]: Error: Failed to create a CUDA device: Operation not permitted
[2023:01:18:15:10:38]: Info: Screencasting with KMS
[2023:01:18:15:10:38]: Info: Found monitor for DRM screencasting
[2023:01:18:15:10:38]: Info: Color coding [Rec. 601]
[2023:01:18:15:10:38]: Info: Color range: [JPEG]
[2023:01:18:15:10:38]: Error: Failed to create a CUDA device: Operation not permitted
[2023:01:18:15:10:38]: Info: Encoder [nvenc] failed
[2023:01:18:15:10:38]: Info: Trying encoder [vaapi]
[2023:01:18:15:10:38]: Info: Screencasting with KMS
[2023:01:18:15:10:38]: Info: Found monitor for DRM screencasting
libva info: VA-API version 1.14.0
libva info: Trying to open /usr/lib64/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_14
libva info: va_openDriver() returns 0
libva info: VA-API version 1.14.0
[2023:01:18:15:10:38]: Info: Color coding [Rec. 601]
[2023:01:18:15:10:38]: Info: Color range: [JPEG]
libva info: Trying to open /usr/lib64/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_14
libva info: va_openDriver() returns 0
zsh: segmentation fault (core dumped) ./sunshine-0.17.0
Backtrace:
(gdb) bt
#0 0x00007fffe67c3b70 in ?? ()
#1 0x00007fffef26cae7 in vlVaDestroySurfaces () from /usr/lib64/dri/radeonsi_drv_video.so
#2 0x00007ffff7f1a02b in vaDestroySurfaces () from /lib64/libva.so.2
#3 0x0000000000e0124b in vaapi_buffer_free (opaque=0x26fdf80, data=<optimized out>) at libavutil/hwcontext_vaapi.c:463
#4 0x0000000000df5c97 in buffer_pool_flush (pool=0x26fe480) at libavutil/buffer.c:302
#5 av_buffer_pool_uninit (ppool=<optimized out>) at libavutil/buffer.c:332
#6 0x0000000000dfbeb3 in hwframe_ctx_free (opaque=<optimized out>, data=<optimized out>) at libavutil/hwcontext.c:230
#7 0x0000000000df57e9 in buffer_replace (src=0x0, dst=<optimized out>) at libavutil/buffer.c:133
#8 av_buffer_unref (buf=<optimized out>) at libavutil/buffer.c:144
#9 0x0000000000dfa11c in av_frame_unref (frame=<optimized out>) at libavutil/frame.c:493
#10 av_frame_unref (frame=0x26ed580) at libavutil/frame.c:477
#11 0x0000000000dfa23a in av_frame_free (frame=0x7fffffffcf98) at libavutil/frame.c:116
#12 0x0000000000d64670 in free_frame (frame=0x26ed580) at /home/kvo/source/Sunshine/src/platform/linux/graphics.cpp:868
#13 0x0000000000d5cb66 in util::Destroy<AVFrame*, void, &(free_frame(AVFrame*))>::operator() (this=0x2573220, p=0x26ed580) at /home/kvo/source/Sunshine/src/utility.h:800
#14 0x0000000000d5c492 in util::uniq_ptr<AVFrame, util::Destroy<AVFrame*, void, &(free_frame(AVFrame*))> >::reset (this=0x2573218, p=0x0) at /home/kvo/source/Sunshine/src/utility.h:501
#15 0x0000000000d5e949 in util::uniq_ptr<AVFrame, util::Destroy<AVFrame*, void, &(free_frame(AVFrame*))> >::~uniq_ptr (this=0x2573218, __in_chrg=<optimized out>) at /home/kvo/source/Sunshine/src/utility.h:496
#16 0x0000000000d5e6b4 in va::va_t::~va_t (this=0x25731f0, __in_chrg=<optimized out>) at /home/kvo/source/Sunshine/src/platform/linux/vaapi.cpp:278
#17 0x0000000000d5eaee in va::va_vram_t::~va_vram_t (this=0x25731f0, __in_chrg=<optimized out>) at /home/kvo/source/Sunshine/src/platform/linux/vaapi.cpp:411
#18 0x0000000000d5eee6 in std::_Destroy<va::va_vram_t> (__pointer=0x25731f0) at /usr/include/c++/12/bits/stl_construct.h:151
#19 0x0000000000d5ee9a in std::allocator_traits<std::allocator<void> >::destroy<va::va_vram_t> (__p=0x25731f0) at /usr/include/c++/12/bits/alloc_traits.h:648
#20 0x0000000000d5ed59 in std::_Sp_counted_ptr_inplace<va::va_vram_t, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0x25731e0) at /usr/include/c++/12/bits/shared_ptr_base.h:613
#21 0x0000000000b660c1 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x25731e0) at /usr/include/c++/12/bits/shared_ptr_base.h:346
#22 0x0000000000b6a479 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7fffffffd188, __in_chrg=<optimized out>) at /usr/include/c++/12/bits/shared_ptr_base.h:1071
#23 0x0000000000cc8226 in std::__shared_ptr<platf::hwdevice_t, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x7fffffffd180, __in_chrg=<optimized out>) at /usr/include/c++/12/bits/shared_ptr_base.h:1524
#24 0x0000000000cca741 in std::__shared_ptr<platf::hwdevice_t, (__gnu_cxx::_Lock_policy)2>::reset (this=0x7fffffffd2f0) at /usr/include/c++/12/bits/shared_ptr_base.h:1642
#25 0x0000000000cc925b in video::session_t::~session_t (this=0x7fffffffd2e0, __in_chrg=<optimized out>) at /home/kvo/source/Sunshine/src/video.cpp:326
#26 0x0000000000cd7d51 in std::_Optional_payload_base<video::session_t>::_M_destroy (this=0x7fffffffd2e0) at /usr/include/c++/12/optional:287
#27 0x0000000000cd2425 in std::_Optional_payload_base<video::session_t>::_M_reset (this=0x7fffffffd2e0) at /usr/include/c++/12/optional:318
#28 0x0000000000ccc4b8 in std::_Optional_payload<video::session_t, false, false, false>::~_Optional_payload (this=0x7fffffffd2e0, __in_chrg=<optimized out>) at /usr/include/c++/12/optional:439
#29 0x0000000000cc9b2c in std::_Optional_base<video::session_t, false, false>::~_Optional_base (this=0x7fffffffd2e0, __in_chrg=<optimized out>) at /usr/include/c++/12/optional:510
#30 0x0000000000cc9b48 in std::optional<video::session_t>::~optional (this=0x7fffffffd2e0, __in_chrg=<optimized out>) at /usr/include/c++/12/optional:705
#31 0x0000000000cc0754 in video::validate_config (disp=std::shared_ptr<platf::display_t> (use count 1, weak count 0) = {...}, encoder=..., config=...) at /home/kvo/source/Sunshine/src/video.cpp:1562
#32 0x0000000000cc0c3b in video::validate_encoder (encoder=...) at /home/kvo/source/Sunshine/src/video.cpp:1585
#33 0x0000000000cc28fb in video::init () at /home/kvo/source/Sunshine/src/video.cpp:1760
#34 0x0000000000b5b528 in main (argc=1, argv=0x7fffffffe578) at /home/kvo/source/Sunshine/src/main.cpp:322
Originally posted by @KarlVogel in #736 (comment)
Expected Behavior
No response
Additional Context
No response
Host Operating System
Linux
Operating System Version
6.1.6-100.fc36.x86_64
Architecture
64 bit
Sunshine commit or version
Package
None
GPU Type
AMD
GPU Model
Radeon 6900XT
GPU Driver/Mesa Version
22.1.7
Capture Method (Linux Only)
KMS
Relevant log output
No response