Skip to content

fetchurl: fix handling of fallback URLs#470643

Merged
ShamrockLee merged 1 commit intoNixOS:masterfrom
trofi:fetchurl-urls
Dec 15, 2025
Merged

fetchurl: fix handling of fallback URLs#470643
ShamrockLee merged 1 commit intoNixOS:masterfrom
trofi:fetchurl-urls

Conversation

@trofi
Copy link
Contributor

@trofi trofi commented Dec 14, 2025

Without the change fallback to urls = [] does not work as $urls evaluates on ony first entry. Use
${urls[@]} instead.

Example derivation that is fixed in master is
xterm.src:

Before:

$ nix build --no-link -f. xterm.src --rebuild -L
xterm> structuredAttrs is enabled
...
xterm> trying ftp://ftp.invisible-island.net/xterm/xterm-403.tgz
...
xterm> curl: (67) Access denied: 550
xterm> error: cannot download xterm-403.tgz from any mirror

After:

$ nix build --no-link -f. xterm.src --rebuild -L
xterm> structuredAttrs is enabled
...
xterm> trying ftp://ftp.invisible-island.net/xterm/xterm-403.tgz
...
xterm> curl: (67) Access denied: 550
...
xterm> trying https://invisible-mirror.net/archives/xterm/xterm-403.tgz
xterm>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
xterm>                                  Dload  Upload   Total   Spent    Left  Speed
xterm> 100  1577k 100  1577k   0     0 703670     0   0:00:02  0:00:02 --:--:-- 703866

Things done

  • Built on platform:
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • Tested, as applicable:
  • Ran nixpkgs-review on this PR. See nixpkgs-review usage.
  • Tested basic functionality of all binary files, usually in ./result/bin/.
  • Nixpkgs Release Notes
    • Package update: when the change is major or breaking.
  • NixOS Release Notes
    • Module addition: when adding a new NixOS module.
    • Module update: when the change is significant.
  • Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

Add a 👍 reaction to pull requests you find important.

Without the change fallback to `urls = []` does not
work as `$urls` evaluates on ony first entry. Use
`${urls[@]}` instead.

Example derivation that is fixed in `master` is
`xterm.src`:

Before:

    $ nix build --no-link -f. xterm.src --rebuild -L
    xterm> structuredAttrs is enabled
    ...
    xterm> trying ftp://ftp.invisible-island.net/xterm/xterm-403.tgz
    ...
    xterm> curl: (67) Access denied: 550
    xterm> error: cannot download xterm-403.tgz from any mirror

After:

    $ nix build --no-link -f. xterm.src --rebuild -L
    xterm> structuredAttrs is enabled
    ...
    xterm> trying ftp://ftp.invisible-island.net/xterm/xterm-403.tgz
    ...
    xterm> curl: (67) Access denied: 550
    ...
    xterm> trying https://invisible-mirror.net/archives/xterm/xterm-403.tgz
    xterm>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
    xterm>                                  Dload  Upload   Total   Spent    Left  Speed
    xterm> 100  1577k 100  1577k   0     0 703670     0   0:00:02  0:00:02 --:--:-- 703866
@nixpkgs-ci nixpkgs-ci bot requested a review from philiptaron December 14, 2025 07:31
@nixpkgs-ci nixpkgs-ci bot added 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 6.topic: fetch Fetchers (e.g. fetchgit, fetchsvn, ...) labels Dec 14, 2025
Copy link
Contributor

@rhelmot rhelmot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missed 2 spots

@nixpkgs-ci nixpkgs-ci bot added the 12.approvals: 1 This PR was reviewed and approved by one person. label Dec 14, 2025
@ShamrockLee
Copy link
Contributor

ShamrockLee commented Dec 14, 2025

Thank you for fixing (one of) the leftover of PR #464475!

Could we add a test case to guard the use of urls and catch future failures? It could be something like { urls = [ "https://example.com/source" "<actual URL>" ]; }

fi
fi
done
urls="$urls2"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is essentially equivalent to urls=("$urls2"). Even though it behaves like a plain variable, it is probably not ideal in the long run.

Nevertheless, it is still a good workaround.

Pinging @wolfgangwalther @MattSturgeon for input.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since spaces are invalid in URLs, this should be OK in practice, right?

Copy link
Contributor

@MattSturgeon MattSturgeon Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I initially looked, my reaction was “let's just use arrays correctly and consistently.” But yes, spaces being invalid makes it mostly a non-issue in practice.

One potential subtlety is what error you get when you pass an invalid URL with a space... If we handle arrays correctly, then the error will come from passing curl an invalid URL. With the current implementation, the error will instead show up as passing curl multiple URL fragments.

While I'd like to see this implementation cleaned up, for me it doesn't block this PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My main concern is how would users access urls in postFetch. This fix allows accessing it as "$urls", which is compatible with pre-structured-attrs behavior, but

  • People and tools would expect urls be passed as a Bash array when [[ -n "${__structuredAttes-}" ]].
  • declare -p urls will show it's a Bash array, but it cnnot be accessed as such.
  • If users try to access urls in netrcPhase, they need to Access it as a Bash array.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I'd like to see this implementation cleaned up, for me it doesn't block this PR.

I agree. Let's merge it first.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was wrong about the "behave like a plain variable" part.

This fix is bugged -- it doesn't work the way we have thought.

$ foo=(a b)
$ declare -p foo
declare -a foo=([0]="a" [1]="b")
$ foo="${foo[*]}"
$ declare -p foo
declare -a foo=([0]="a b" [1]="b")
$ declare foo="${foo[*]}"
$ declare -p foo
declare -a foo=([0]="a b b" [1]="b")

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are showing how foo="${foo[*]}" doesn't work as expected. But we don't do urls="${urls2[*]}", we do urls="$urls2". So I don't see the connection, yet.

Copy link
Contributor

@ShamrockLee ShamrockLee Dec 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we don't do urls=\"${urls2[*]}\", we do urls=\"$urls2\".

The problemis the same. It is essentially urls[0]=$urls2. We thought we were rewrititng urls, but it is just urls[0].

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I found the mistake in my test-case. It's not exactly the same behavior as above, but yes only the first element is replaced with the other first element.

fi
fi
done
urls="$urls2"
Copy link
Contributor

@MattSturgeon MattSturgeon Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I initially looked, my reaction was “let's just use arrays correctly and consistently.” But yes, spaces being invalid makes it mostly a non-issue in practice.

One potential subtlety is what error you get when you pass an invalid URL with a space... If we handle arrays correctly, then the error will come from passing curl an invalid URL. With the current implementation, the error will instead show up as passing curl multiple URL fragments.

While I'd like to see this implementation cleaned up, for me it doesn't block this PR.

@nixpkgs-ci nixpkgs-ci bot added 12.approvals: 2 This PR was reviewed and approved by two persons. and removed 12.approvals: 1 This PR was reviewed and approved by one person. labels Dec 15, 2025
@ShamrockLee ShamrockLee added this pull request to the merge queue Dec 15, 2025
@nixpkgs-ci nixpkgs-ci bot added 12.approvals: 3+ This PR was reviewed and approved by three or more persons. and removed 12.approvals: 2 This PR was reviewed and approved by two persons. labels Dec 15, 2025
Merged via the queue into NixOS:master with commit 860a844 Dec 15, 2025
45 of 48 checks passed
@trofi trofi deleted the fetchurl-urls branch December 15, 2025 20:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: fetch Fetchers (e.g. fetchgit, fetchsvn, ...) 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. 12.approvals: 3+ This PR was reviewed and approved by three or more persons.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants