-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Add capture using WinRT Windows.Graphics.Capture API. #2149
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
cgutman
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for this contribution!
One other thing that WGC apparently provides is support for cross-adapter capture (capturing outputs attached to GPU 1 to a texture on GPU 2 for encoding). Did you happen to try that scenario? If not, that's fine - I was just curious.
This comment was marked as resolved.
This comment was marked as resolved.
|
Does UAC elevation and service mode work as expected? I've read many microsoft sources that pretty much told that this capture approach is not compatible with unattended use cases Source: robmikh/Win32CaptureSample#24 and robmikh/Win32CaptureSample#54 |
|
Thanks for the quick reviews! I'll be addressing code and GitHub actions issue in, uh, a bit.
@cgutman: I have not tried this scenario and while I did see this somewhere, I figured it was a little out of scope considering that, as far as I can tell, Sunshine is consumer software.
@TheElixZammuto: I think the headless scenario will work, but I need to test it after I make requested changes. Privilege elevation probably won't, but the existing API supports it. I designed this feature to be opt-in-only and to fall back if unsupported, so I hope that there's sufficient insurance for this. In no way do I advocate replacing the existing display capture code with Windows.Graphics.Capture. I quite like the idea of it being a beta feature that can be enabled in sunshine.conf but not the user interface. I also hope its presence lowers the bar for more knowledgeable contributors to improve this capture backend. |
WGC actually can capture the UAC secure desktop, even from an unelevated Sunshine.exe process, so it's actually better than DXGI DDA in that regard. It can even capture the login screen after locking the PC. However, you are correct that it doesn't work with our service model, at least not without additional changes. WGC works using per-user services which are activated via COM. The specific user service that runs the capture backend is Fortunately, the fact that elevation doesn't seem to actually provide any benefit to WGC (unlike DDA) means we do have some options here to support WGC from a service context. The first and simplest is to try impersonating using the user token from If impersonation doesn't work out, we'd need to use a WGC helper process that we would spawn into the user session and share textures with. This is definitely much more complex than the impersonation solution, but it may not be too horrible since we're already using shared textures between capture and encoding today.
Cross-adapter encoding is good to have on hybrid graphics systems, which are quite typical these days (most CPUs include an iGPU and basically all laptops do). Cross-adapter encoding lets users encoding on the dGPU even when the display is physically connected to the iGPU. We have to use a bunch of hacks (ddprobe.exe manually setting our graphics preference) to work around the cross-adapter encoding limitation for DDA on hybrid graphics systems to even be able to stream at all.
I actually would like to move to WGC as the default on OSes where support is good (Win11/2022+). It has a ton of advantages over the DXGI solution, including:
Obviously it could still have massive Achilles heel that sinks it for our usecase, but it looks quite compelling on the surface. In any case, none of this aspirational stuff matters for right now. I'm mainly just writing it down somewhere to not forget any of it. You should leave it as opt-in for this PR as you have it now and don't worry about the impersonation stuff. |
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
|
I finally tested this, and notice a fairly substantial performance regression on your updated branch vs the first commit (b843c12) tested on its own; my guess is the Testing 60fps client and server with the server rendering content at 60fps, the client can only runs at ~40fps maximum. My system is not too ancient (Ryzen 5700X and RX 6600), and I'm only testing at 768p, but this system usually has no trouble keeping 60fps at 4K with the standard capture path. Added some quick debug: This shows a LOT of zero count timeouts on the latest branch (and the host is definitely rendering content at 60fps, so the volume of timeouts I'm seeing should not be happening). I can raise the capture to 60fps via one of the following steps (performed separately): |
|
@psyke83 Thank you for reporting this. I noticed it too, but I thought it might be a personal problem (i.e. "It only doesn't work on my machine") The issue no longer appears for me after applying the commit "When the frame timeout is zero, use the most recently produced frame instead of waiting for a new one." If you get a chance, I'd be curious to hear your thoughts on the actual change and if it works for you too! |
|
Capture is now running at 60fps on my system - nice work! Unfortunately, whilst the incoming framerate stays at ~60, there is some noticeable periodic stuttering in games running at 60fps that isn't present in the regular capture path... but I may have discovered the solution already. My GPU/driver combo doesn't support HAGS, meaning that Sunshine is using realtime gpu scheduling on my system. This normally works well (especially to prevent stuttering when the GPU is heavily loaded), but it seems to conversly cause stuttering with your new WGC capture path. Manually setting the priority to high seems to result in capture as smooth as the DDAPI path (but I need to test more thoroughly to say that the stuttering is 100% alleviated). I would recommend you and any other testers to check realtime vs high gpu scheduling with this capture path, just in case the issue is not unique to my specific setup. Thanks, and again - great work! |
src/platform/windows/display_wgc.cpp
Outdated
| release_frame(); | ||
|
|
||
| AcquireSRWLockExclusive(&frame_lock); | ||
| if (timeout.count() > 0 && SleepConditionVariableSRW(&frame_present_cv, &frame_lock, timeout.count(), 0) == FALSE) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This needs to first check if a frame is available before sleeping on the condition variable. Unlike an event, a condition variable doesn't have stored signalled/not-signalled state. It only wakes waiters that are asleep at the time WakeConditionVariable() is called. This means you always want to have it gated by a check that happens under your lock (see Microsoft's sample code)
Likewise, on the producer side, you don't have to call WakeConditionVariable() when produced_frame is already non-null, because you're not making a state transition where the consumer could possibily be waiting (the predicate is already true).
This is probably what's killing your performance. If the consumer isn't waiting at the time the producer gets a frame (N), that frame is going to be completely lost because the consumer is going to go to sleep on the CV even though there's already a frame ready for it. Only once asleep will the consumer get properly woken up with the next new frame (N+1).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for this.
To make a long story short, today I learned that condition variables don't actually store signaled state. With your suggested changes, I still no longer see the performance degradation observed earlier! I wonder if the stuttering @psyke83 observed is still present.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as spam.
This comment was marked as spam.
|
@roob0 thank you for sharing the script. But i want to stream lock screen , at least the screen when windows is shutting down without disconnection. I hope there is a way. |
|
@Kobain-Seo Yes, It perfectly streams the lockscreen! I made this script for this purpouse. You can unlock PC from the client with moonlight, then reconnect (always in moonlight) after the pc is unlocked. |
|
@Kobain-Seo I usually put my pc in hybrid sleep (and cutting off power when I want to close it, so I don't press "Shutdown"). However, if you want to shutdown your pc in the "classic" way , in switch ddx's scheduler activity you can add the Event in Activator with System, User32, ID 1074. I'm testing the scripts in several scenario (lock-on, lockon and sleep, shutdown) and always work ;) |
|
@cgutman I find this simple solution for swicthing from service to the exe and from ddx to wgc (and viceversa). Perhaps a reconnection could be added in Moonlight (avoiding the disconnection) but, of course, a more sophisticated solution would be better. This is only a temporary "solution", waiting for the service to be "fixed". |
Can't make it work, i've never used autoit + task scheduler so I don't know if I didi it correctly, can you help me here? maybe making a tutorial, or via DM |
|
Also hoping for an improvement on that part. :) Can i use the same settings and configfile with that script. |
|
@Vaskyy don't use AutoIt, just create 2 config files, one with wgc setting like
Run the batch file again to reverse the settings 😊, if you want to use dxgi, don't forget to |
|
Damn that took me alot of time, bc im absolutely not advanced in any cmd/ps stuff. i did create an dxgi.bat: and also an wgc.bat: But now the wgc.bat is opening a cmd prompt, that if i close it, will terminate the thing of course. Also isnt it possible to autostart the sunshine.exe (not the service) to be able to use WGC always? |
I actually managed to do this BUT, and is a big BUT, you have to give up some home privacy, if you set sunshine.exe on the start forlder on windows Start Menu sunshine will autostart at widnows LOGON, (not boot up), although you cant set it up before it would be useless on WGC since you can't unlock the pc on WGC, so, my solution? autologon, pc boots and goes staight into windows with sunshine auto starting and letting you use always WGC... another workaround is to unlock the pc with parsec and then change to sunshine, I actually do this with another decive I have at home to turn my pc ON |
Funny because since I use virtual display driver, no one see my PC will know my PC was on unlocked haha they just think display is always off, you need change the cableing OS might output to it or you need to adjust in Moonlight remotely. |
there you go, try my workaround, is not ideal as a service, but in practice it does the job. |
|
@Crasthiff , @Vaskyy, @Kobain-Seo Hi, I decided to make this simply utility: https://github.com/roob-p/SunshineCaptureSwitcher |
|
@roob-p you don't need to modify the |
Ah, got it, thanks for the tip. |
|
@ReenigneArcher I updated the utility so now it uses capture=wgc without modifying Sunshine.conf. |
No, that's not possible right now. It could be possible, but not much value since you can terminate Sunshine using options that Windows (or whatever OS) provides though.
If I understand the question, no that's also not possible right now. |
Ok, thanks!
Yes, I meant that, about forcing auto-reconnection when a session is stopped. Would be a neat feature! |
|
Probably everything in your tool, and any features required to make it a better experience, should just be integrated into Sunshine (or the WGC method just made to work with the service). I did have a PR that attempted to remedy the situation (similar to your tool). #3344 If you (or anyone else) want to make an attempt, I've made a roadmap entry about this issue here: LizardByte/roadmap#65 |
|
@ReenigneArcher |
This comment was marked as spam.
This comment was marked as spam.
@Crasthiff Hi, thanks for the support and your idea. I had already been thinking about it, but your comment pushed me to give it a shot at handling the UAC problem. I already tried detecting the UAC prompt and switching to DDX. It works, but it’s a bit inconvenient: it requires disconnecting and reconnecting the stream. And going back to WGC after switching to DDX to handle the UAC prompt means reconnecting twice. However, it works, so maybe I’ll upload a new release in the next few days. Regarding Apollo, I just did some testing without changing the code, and I know it can work the same way. I think the service name also needs to be updated ( I’ll either release a version for Apollo, or an installer that supports both Sunshine and Apollo, with an option to choose which one to use (it’s better not to have both installed at the same time). |
dude, you're awesome, it'd be neat to be able to cheange api on the fly, but that's up to the devs of sun/apollo, meanwhile, keep it up, you just kick *ss. |
|
@Crasthiff Hi, I just uploaded the new release. It now supports Apollo (just set |
|
@Nonary Nice to hear, following with interest! |
I found a way to run sunshine via bat without the console popping up, it will appear as if it is run via a service. startSunshine.bat Add this to your bat file and sunshine can run in the background. |
|
@Earchaut You can run Sunshine in WGC mode by launching |
@roob-p Bro, I just read this... YOU ROCK SO MUCH!!!! it took you only 5 days to solve this. You're just amaing, keep it up! also...
@Nonary this is awesome new too, you guys are what the doctor prescribed ♥ |
Description
This adds a capture backend on Windows that uses the Windows.Graphics.Capture API. This is a more recent API that's capable of capturing the Xbox Game Bar, which the DXGI Desktop Duplication API seemingly cannot do.
To accomplish this, I used a slightly different build environment known as MSYS UCRT64. I also bumped the compilation standard to C++20 so that the GNU C++ compiler can handle the coroutine definitions present in WinRT headers. The UCRT64 and MinGW64 environment appear to be binary-compatible, and Sunshine's dependencies translate smoothly across this boundary. The build documentation in this change reflects these modifications.
This change can be enabled by setting
capture = wgcin sunshine.conf on Windows servers. When this setting is off, no functionality change should be observable.I've observed CPU and memory usage trends over a long streaming period and there don't appear to be any new leaks.
Issues Fixed or Closed
Fixes #832 (manually verified)
Type of Change
.github/...)Checklist
Branch Updates
LizardByte requires that branches be up-to-date before merging. This means that after any PR is merged, this branch
must be updated before it can be merged. You must also
Allow edits from maintainers.