Skip to content

introduce the zig fetch subcommand and symlink support in zig packages#17363

Merged
andrewrk merged 6 commits intomasterfrom
tar-symlinks
Oct 3, 2023
Merged

introduce the zig fetch subcommand and symlink support in zig packages#17363
andrewrk merged 6 commits intomasterfrom
tar-symlinks

Conversation

@andrewrk
Copy link
Member

@andrewrk andrewrk commented Oct 2, 2023 •
edited
Loading 

zig fetch [options] <url>
zig fetch [options] <path>

Fetches a package which is found at <url> or <path> into the global cache directory, printing the package hash to stdout.

Closes #16972
Related to #14280

Additionally, this branch:

Merge Checklist

  • support symlinks in Package/git.zig
  • more kinds of symlink metadata in std.tar eagerly awaiting @vesim987's upcoming PR for this
  • handle Diagnostics from std.tar

@andrewrk
Copy link
Member Author

andrewrk commented Oct 2, 2023

cc @ianprime0509 - would you be willing to add the git.zig logic to add symlink support here?

@ianprime0509
Copy link
Contributor

ianprime0509 commented Oct 2, 2023

Awesome! Sure, I'll take a closer look at this after work and send a patch to add support for this in Git.

@andrewrk
std.tar: support symlinks
a5144d1 
closes #16678
@andrewrk
compiler: extract package hashing logic to separate file
a435298 
There are no functional changes in this commit.
@andrewrk
std.fs: give readLink an explicit error set
309c532
@andrewrk
introduce the 'zig fetch' command + symlink support
ef9966c 
zig fetch [options] <url>
zig fetch [options] <path>

Fetches a package which is found at <url> or <path> into the global
cache directory, printing the package hash to stdout.

Closes #16972
Related to #14280

Additionally, this commit:

* Adds uncompressed .tar support to package fetching
* Introduces symlink support to package fetching
@andrewrk
zig fetch: enhanced error reporting
2118118 
* Package: use std.tar diagnostics to give detailed error messages
* std.tar: add diagnostic for unsupported file type
@andrewrk
Copy link
Member Author

andrewrk commented Oct 3, 2023

more kinds of symlink metadata in std.tar

eagerly awaiting @vesim987's upcoming PR for this

@andrewrk andrewrk enabled auto-merge October 3, 2023 00:22
@andrewrk
Copy link
Member Author

andrewrk commented Oct 3, 2023

Note that zig fetch on a directory will be a bit useless until #14311 is resolved, because the package will potentially include files from version control which would not make it into a tarball.

@ianprime0509
Copy link
Contributor

ianprime0509 commented Oct 3, 2023

@andrewrk not sure what would be most convenient for you (maybe I should just send a follow-up PR once this is merged), but here's a patch to integrate this with git.zig as well: https://gist.github.com/ianprime0509/f097b73a77e0c9412fe4a11824bd169d

This is able to fetch the dependency in this PR: kristoff-it/orca-ui-hello-zig#1

⬢[ian@toolbox orca-ui-hello-zig]$ ~/src/zig/build/stage4/bin/zig build
Fetch Packages... orca [9MiB] /var/home/ian/src/tmp/orca-ui-hello-zig/build.zig.zon:8:20: error: dependency is missing hash field
            .url = "git+https://git.handmade.network/hmn/orca.git#c4a1ad52a07bae2e6710c38f41a3c64ed8f93136",
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
note: expected .hash = "1220a913fdaac3962e4331bd71d729b70da9ecfa2a959d62d6c491f18b4619cdca61",

Sample when the underlying filesystem doesn't support symlinks:

⬢[ian@toolbox orca-ui-hello-zig]$ ZIG_GLOBAL_CACHE_DIR=$HOME/src/tmp/fat.mnt ~/src/zig/build/stage4/bin/zig build
Fetch Packages... orca [8MiB] Checkout... /var/home/ian/src/tmp/orca-ui-hello-zig/build.zig.zon:8:20: error: unable to unpack packfile
            .url = "git+https://git.handmade.network/hmn/orca.git#c4a1ad52a07bae2e6710c38f41a3c64ed8f93136",
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
note: unable to create symlink from 'src/ext/wasm3/platforms/android/app/src/main/cpp/m3' to '../../../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/arduino/lib/wasm3' to '../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/bluepill/lib/wasm3/src' to '../../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/esp32-idf-wasi/main/wasm3' to '../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/esp32-pio/lib/wasm3/src' to '../../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/esp8266/lib/wasm3' to '../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/fomu/src/wasm3' to '../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/hifive1/lib/wasm3' to '../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/particle/lib/wasm3/src' to '../../../../../source': AccessDenied
note: unable to create symlink from 'src/ext/wasm3/platforms/embedded/wm_w600/wasm3' to '../../../source': AccessDenied
note: unable to create symlink from 'tests/files/data/jail/dir_escape' to '..': AccessDenied
note: unable to create symlink from 'tests/files/data/jail/file_escape' to '../regular.txt': AccessDenied
note: unable to create symlink from 'tests/files/data/symlink' to 'regular.txt': AccessDenied

@andrewrk
Copy link
Member Author

andrewrk commented Oct 3, 2023

If you're fine with it, I can cherry-pick from your zig fork into this branch

@ianprime0509
Copy link
Contributor

ianprime0509 commented Oct 3, 2023

That sounds like an easier idea, I just pushed it up here: https://github.com/ianprime0509/zig/tree/tar-symlinks

@andrewrk
Copy link
Member Author

andrewrk commented Oct 3, 2023

Commit looks great 👍

@andrewrk andrewrk merged commit df4853a into master Oct 3, 2023
@andrewrk andrewrk deleted the tar-symlinks branch October 3, 2023 10:33
@kassane kassane mentioned this pull request Oct 9, 2023
Open
1 task
@kassane kassane mentioned this pull request Nov 1, 2023
Open
@ianprime0509 ianprime0509 mentioned this pull request Nov 5, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

build.zig.zon improve workflow to update dependency hash std.tar: handle tar files with symbolic links

2 participants

@andrewrk @ianprime0509