Skip to content

Comments

treewide: use non-binary wrapper for wrapGAppsHook where appropriate#172320

Merged
thiagokokada merged 8 commits intoNixOS:masterfrom
xaverdh:binary-wrapper-fixes
May 10, 2022
Merged

treewide: use non-binary wrapper for wrapGAppsHook where appropriate#172320
thiagokokada merged 8 commits intoNixOS:masterfrom
xaverdh:binary-wrapper-fixes

Conversation

@xaverdh
Copy link
Contributor

@xaverdh xaverdh commented May 10, 2022

The use of --add-flags in this derivation does not work with binary
wrappers, which wrapGAppsHook uses since #164163.

Description of changes

See discussion here

Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 22.05 Release Notes (or backporting 21.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
    • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
  • Fits CONTRIBUTING.md.

The use of --add-flags in this derivation does not work with binary
wrappers, which wrapGAppsHook uses since NixOS#164163.
@xaverdh xaverdh marked this pull request as ready for review May 10, 2022 10:57
@xaverdh
Copy link
Contributor Author

xaverdh commented May 10, 2022

Ok, the test passed and I am successfully running signal from this branch on my system.

@ncfavier
Copy link
Member

I think we can fix the other programs in this PR as well

@xaverdh
Copy link
Contributor Author

xaverdh commented May 10, 2022

I think we can fix the other programs in this PR as well

will do that!

xaverdh added 3 commits May 10, 2022 13:05
The use of --add-flags in this derivation does not work with binary
    wrappers, which wrapGAppsHook uses since NixOS#164163.
The use of --add-flags in this derivation does not work with binary
    wrappers, which wrapGAppsHook uses since NixOS#164163.
The use of --add-flags in this derivation does not work with binary
    wrappers, which wrapGAppsHook uses since NixOS#164163.
@ofborg ofborg bot requested review from Mic92 and equirosa May 10, 2022 11:16
@ofborg ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. 10.rebuild-linux: 1 This PR causes 1 package to rebuild on Linux. labels May 10, 2022
@xaverdh
Copy link
Contributor Author

xaverdh commented May 10, 2022

I did build brave, but not vscode and discord (due to the unfree license). Still I think they should work.

From a quick grep through nixpkgs, these are indeed all derivations that use wrapGAppsHookand mention NIXOS_OZONE_WL. But there are probably other similar usages of --add-flags..

@ncfavier
Copy link
Member

ncfavier commented May 10, 2022

Yes, but none of those I could find (grep -e "--(add-flags|prefix|suffix|set|argv0).*\\\\('')?\\\$") are using wrapGAppsHook, so the fix is less urgent.

@ncfavier
Copy link
Member

Please change the PR's title

@xaverdh xaverdh changed the title signal-desktop: use non-binary wrapper for wrapGAppsHook treewide: use non-binary wrapper for wrapGAppsHook where appropriate May 10, 2022
@ofborg ofborg bot removed the 10.rebuild-linux: 1 This PR causes 1 package to rebuild on Linux. label May 10, 2022
@xaverdh
Copy link
Contributor Author

xaverdh commented May 10, 2022

Yes, but none of those I could find (grep -e "--(add-flags|prefix|suffix|set|argv0).*\\\\('')?\\\$") are using wrapGAppsHook, so the fix is less urgent.

I think e.g. cryptomator is also broken, unless I am mistaken?

@ncfavier
Copy link
Member

It doesn't look like it expands any variables at runtime.

@xaverdh
Copy link
Contributor Author

xaverdh commented May 10, 2022

Will the ~ get expanded by the java runtime itself?

@ncfavier
Copy link
Member

At least it won't get expanded by Bash, because of the single quotes.

@ncfavier
Copy link
Member

ncfavier commented May 10, 2022

But this will. Let's add tlaplusToolbox to this PR, please.

@xaverdh
Copy link
Contributor Author

xaverdh commented May 10, 2022

Right, still there appears to be an issue with the --add-flags "--class-path '$out/share/cryptomator/libs/*'"
part. I get Error: --class-path requires class path specificationon master but it works when I inject the non-binary wrapper.

@ncfavier
Copy link
Member

Ah, I guess the binary implementation of --add-flags doesn't parse quotes.

@xaverdh
Copy link
Contributor Author

xaverdh commented May 10, 2022

There are also cases like this where the implementation is "leaked" to users, but Its not very likely somebody relies on injecting shell code there?

@ncfavier
Copy link
Member

It looks like we also need to fix gmtp (probably just remove quotes around $out/share) and bluej.

@ncfavier
Copy link
Member

There's a missing ) in pkgs/applications/science/logic/tlaplus/toolbox.nix

The use of --add-flags in this derivation does not work with binary
    wrappers, which wrapGAppsHook uses since NixOS#164163.
@drupol
Copy link
Contributor

drupol commented May 10, 2022

I just tested vscode using this patch and now it works.

xaverdh added 3 commits May 10, 2022 15:17
The use of --add-flags in this derivation does not work with binary
    wrappers, which wrapGAppsHook uses since NixOS#164163.
The use of --add-flags in this derivation assumed quotes to be expanded, which the binary
         wrapper (which wrapGAppsHook uses since NixOS#164163) will not do.
The use of --add-flags in this derivation assumed quotes to be expanded, which the binary
         wrapper (which wrapGAppsHook uses since NixOS#164163) will not do.
@xaverdh xaverdh force-pushed the binary-wrapper-fixes branch from e59f2e9 to 757f518 Compare May 10, 2022 13:18
@ofborg ofborg bot added 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. and removed 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels May 10, 2022
@drupol
Copy link
Contributor

drupol commented May 10, 2022

Do you think we can commit this as-is so we unblock people using vscode and other software impacted by this issue ?

@thiagokokada thiagokokada merged commit c4956c8 into NixOS:master May 10, 2022
@thiagokokada
Copy link
Contributor

Done.

@drupol
Copy link
Contributor

drupol commented May 10, 2022

Thanks ! Any clue on how to force a new evaluation on hydra?

@thiagokokada
Copy link
Contributor

Thanks ! Any clue on how to force a new evaluation on hydra?

No, but not sure if this is a good idea. I mean, almost every PR here is important for those that need it, if we had to restart hydra evals on each PR merged we would never get anywhere.

@ncfavier
Copy link
Member

ncfavier commented May 10, 2022

Started a new evaluation. I think vscode + signal + discord is past the threshold of "important for almost everyone".

@drupol
Copy link
Contributor

drupol commented May 10, 2022

Thank you :)

autoPatchelfHook
maven
makeWrapper
(wrapGAppsHook.override { makeBinaryWrapper = makeWrapper; })
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is the override needed for cryptomator?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#172320 (comment)

We could also remove the single quotes, but then we would be implicitly relying on the binary wrapper being in scope, which isn't great.

Copy link
Member

@dotlambda dotlambda May 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So makeBinaryWrapper ends up in cryptomator's nativeBuildInputs without? That should not happen.

Copy link
Member

@dotlambda dotlambda May 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The derivation doesn't use gappsWrapperArgs.

Copy link
Member

@ncfavier ncfavier May 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So makeBinaryWrapper ends up in cryptomator's nativeBuildInputs without? That should not happen.

Yes, because it's a propagated build input of the wrapGAppsHook (that is how setup hooks work). I agree that's not great, but I don't know how we should fix it. This is a limitation of Bash more than anything.

The derivation doesn't use gappsWrapperArgs.

It doesn't need to. It first creates the cryptomator executable as a wrapper over java, and then wrapGAppsHook wraps that wrapper automatically in the fixup phase.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makeBinaryWrapper is also a setup hook, i.e. a Bash script that defines functions. Functions aren't scoped in Bash, so we would need to do something like

wrapGApp() (
    local program="$1"
    shift 1
    source @makeWrapper@
    wrapProgram "$program" "${gappsWrapperArgs[@]}" "$@"
)

in order not to expose the functions to the global scope. Maybe it could work.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But makeWrapper also depends on dieHook, so we'd also need to source that explicitly... it's kind of awkward.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could call one of the bash functions makeBinaryWrapper and use a boolean argument for wrapGAppsHook

Copy link
Member

@ncfavier ncfavier May 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could call one of the bash functions makeBinaryWrapper

I'm doing that in #172366, but the goal of the makeBinaryWrapper hook is to be a drop-in replacement for makeWrapper, so we can't just drop the makeWrapper function.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then let's call the other makeShellWrapper so people can decide which one they want to use if they feel like it. That would make this override superfluous.

@bergkvist
Copy link
Member

@thiagokokada - wordexp looks interesting. As for patterns like ${VAR:-VALUE} and ${VAR:+VALUE}, it would be nice if we had some replacement for these in the binary wrappers.

I think

--set X '${Y:-$Z}'

would be equivalent to this (assuming wordexp is able to expand '$Y' and '$Z' at runtime)

--set X '$Y'
--set-default X '$Z'

Need to think a bit more about ${VAR:+VALUE} though. It might make sense to introduce some new "primitive" to make something equivalent to this possible in binary wrappers. I'm also curious about the performance impact of using wordexp.

@thiagokokada
Copy link
Contributor

thiagokokada commented May 11, 2022

@thiagokokada - wordexp looks interesting. As for patterns like ${VAR:-VALUE} and ${VAR:+VALUE}, it would be nice if we had some replacement for these in the binary wrappers.

I think

--set X '${Y:-$Z}'

would be equivalent to this (assuming wordexp is able to expand '$Y' and '$Z' at runtime)

--set X '$Y'
--set-default X '$Z'

Need to think a bit more about ${VAR:+VALUE} though. It might make sense to introduce some new "primitive" to make something equivalent to this possible in binary wrappers. I'm also curious about the performance impact of using wordexp.

Huh... Actually re-reading wordexp documentation it really looks like what we want (it expand env vars too, I thought it only expanded things like ~). So yeah, looks exactly what we want. Needs more testing though.

I also wouldn't worry too much about the performance. I think this would be better to be behind a flag anyway, since I think having it enabled by default (at least for now) can cause some issues.

@bergkvist Can you open an issue for this? I think it would be useful to concentrate the discussion there, instead of a closed PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants