Since then I have done some testing and concluded, that the fact that that patch even worked on my system was a miracle (and that my original approach is indeed completely flawed). I have tested it on several systems and it has worked only under sway on AMD/Intel GPUs. It has not worked on sway with an Nvidia GPU. This not as relevant, though, as the way in which it fails.

On some systems it is partially transparent, but then becomes black, when focused. When unfocused it becomes transparent again. This and the fact that in nsWindow::Create (link) everything is done to obtain a transparent window lead me to believe that firefox successfully manages to create a transparent window, as such it should be capable of rendering transparent content, but I have not managed to convince it to do so on any other systems (likely due to some specific quirks in sway's codebase that are not present anywhere else, but that is just speculation), not even the tab and toolbars, which work even on the current firefox release build of my distro (122.0.1) with a certain userChrome.css on sway but not on xwayland (or X) or any other wayland compositor I tried. There it usually shows a gray background.

Moreover I have edited the lines mentioned previously to return a hardcoded color to see which one is relevant and it seems that the first one enforces the opaque background (link), despite the fact that firefox should have a transparent context. That if, as far as I understand, checks exactly that and ensures that the background is opaque when the context (presentation shell?) is opaque. But the context should be transparent (because the window is) and yet it is not. As of now I am yet unsure where this disconnect happens between the main window being created as transparent and its contents detecting it as opaque.

This is what I will be looking into next, trying to find a an approach that hopefully works on all systems I can test it on.

Any suggestions where I should be looking for such things in the codebase @emilio?

Yeah, on wayland at least we create an opaque region for the whole window except the window corners, see https://searchfox.org/mozilla-central/rev/90cdd721bc108f735b8f1f9f9715eda7e66e5e47/widget/gtk/MozContainerWayland.cpp#498

In the end it seems to work both on wayland and X, although it was a little finicky on an intel iGPU + nvidia GPU laptop. It still does not seem to work on xwayland, though.

EDIT: did some further testing and updated this comment with the new results.

Code analysis found 5 defects in diff 826889:

• 1 defect found by clang-tidy
• 2 defects found by clang-format
• 2 defects found by clang-format (Mozlint)

You can run this analysis locally with:

• ./mach static-analysis check --outgoing (C/C++)
• ./mach clang-format -p dom/ipc/BrowserParent.cpp widget/gtk/nsWindow.cpp
• ./mach lint --warnings --outgoing

For your convenience, here is a patch that fixes all the clang-format defects (use it in your repository with hg import or git apply -p0).

If you see a problem in this automated review, please report it here.

You can view these defects in the Diff Detail section of Phabricator diff 826889.

Code analysis found 1 defect in diff 826890:

• 1 defect found by clang-tidy

1 defect unresolved and 4 defects closed compared to the previous diff 826889.

You can run this analysis locally with:

• ./mach static-analysis check --outgoing (C/C++)

If you see a problem in this automated review, please report it here.

You can view these defects in the Diff Detail section of Phabricator diff 826890.

Unfortunately this seems to work only on sway. On Weston and Hyprland, for example, it is bugged. On Hyprland if I move a window over it it just gets imprinted on the firefox window multiple times and the partially transparent background accumulates into full black. On weston all transparent parts simply become black. What I'm suspecting is happening is that the compositor treats the window as if it were opaque and as such it overdraws it instead of redrawing. Perhaps the parts of the codebase that request the window explicitly make it opaque and then this patch makes it transparent anyway and that confuses the compositor? Should I open a new separate issue or should this be solved here as well?

This same problem was happening before, when I did not remove the opaque regions. On sway it worked even then and it works now because it seems not to care about the wl_surface_set_opaque_region hint and fully ignores it and apparently just redraws everything instead of overdrawing on opaque windows. Weston and Hyprland, however, seem to be using the hint to optimize drawing, hence this problem, probably. This is my speculation.

In D201346#7166798, @kravantokh wrote:

Unfortunately this seems to work only on sway. On Weston and Hyprland, for example, it is bugged. On Hyprland if I move a window over it it just gets imprinted on the firefox window multiple times and the partially transparent background accumulates into full black. On weston all transparent parts simply become black. What I'm suspecting is happening is that the compositor treats the window as if it were opaque and as such it overdraws it instead of redrawing. Perhaps the parts of the codebase that request the window explicitly make it opaque and then this patch makes it transparent anyway and that confuses the compositor? Should I open a new separate issue or should this be solved here as well?

Is the opaque region wrong? After bug 1891414 we should set it correctly... It'd be good to see why that would not be working.

I am terribly sorry. It was yet another rookie mistake on my part. I played around some more with the source code and then did a hard reset and pull to sync it up with the remote and then ran moz-phab patch, thinking that it would apply it against the latest changes only to realize now that it put me on a different branch and reverted to the previous version of the code, where this was not yet repaired.

Anyway, now I managed to rebase the commit against the latest ones, that include bug 1891414 and now it works perfectly under both weston and Hyprland.

Thank you a lot for your help, patience and tolerance while guiding me through writing this patch correctly.

As such this patch implements all necessary changes to allow users to make firefox transparent

Could you please elaborate on what use case this serves? What kind of users would we expect to touch this pref, to what end?

The main use case for this change is simply aesthetics. A considerable number of users (from what I have seen) like having a transparent terminal and I have met the idea of having the browser be able to do the same in several places. This is mainly about choice: allowing users to opt for this transparent look, should they so desire. In a context where transparent terminals are used having a transparent browser would just make perfect sense, aesthetically speaking.

Along the years there has been at least one attempt to achieve this with a bug in firefox's gif renderer (that has since been fixed). Links: actual config and discussions about it.
Here are some relevant posts I have found: archlinux forums, r/FirefoxCSS, r/firefox.

Looking at these I considered them to be enough reason to implement this change besides the original feature request having been open for over 24 years.

As for the audience, I would expect mostly power users to reach for this option: people who are likely to be using a userChrome.css file anyway. An example of such a set of people would be the r/FirefoxCSS community or part of this other subreddit about good-looking linux configurations (transparent aesthetics are quite frequent over there).
These people would likely have no difficulty in potentially configuring an already existing page recoloring extension such as Dark Reader or this other one or even just straight up writing a custom one to automatically recolor all pages to transparent.