Improving the Rust packaging ecosystem #45483
Replies: 4 comments 3 replies
-
|
@alecbcs FYI |
Beta Was this translation helpful? Give feedback.
-
|
also @trws |
Beta Was this translation helpful? Give feedback.
-
|
Sadly pretty much everything you mention as a problem is accurate. I did some exploration to see if we could do something better, but never got to a point of being able to put anything up as a PR. The last place I ended up was thinking we would either want to generate all the dependency packages and merge them periodically (effective, low(ish) changes, massively noisy) or work out how to add a virtual package repository for cargo dependencies. In the end, we need cargo dependencies to be represented as packages somehow, then to assemble them into a fake vendor or workspace directory so they appear to have been built by cargo as part of that build. It's messy, but basically what some linux distros do, and not too hard to set up. So it's actually posted somewhere long-term, here's the minimal version of import hashlib
import os
import shutil
import sys
import tomli
import urllib
import urllib.request
with open("Cargo.lock", 'rb') as lockf:
versions = tomli.load(lockf)
crates = []
os.makedirs("vendor", exist_ok=True)
cratesio = "https://crates.io/api/v1/crates"
for pkg in versions['package']:
if 'source' in pkg:
if pkg['source'] == "registry+https://github.com/rust-lang/crates.io-index":
url = f"{cratesio}/{pkg['name']}/{pkg['version']}/download"
print(f"{pkg['name']}: {url}")
crate_out = f"{pkg['name']}-{pkg['version']}.crate"
crate_out_full = "vendor/" + crate_out
crates.append(crate_out)
with urllib.request.urlopen(url) as u:
crate = u.read()
with open(crate_out_full, "wb") as of:
of.write(crate)
new_cs = hashlib.sha256(crate).hexdigest()
if pkg['checksum'] != new_cs:
raise RuntimeError(f"Hash mismatch on crate? downloaded: {new_cs} lock: {pkg['checksum']}")
shutil.unpack_archive(crate_out_full, filter='tar', extract_dir='vendor', format="gztar")
crate_dir = f"vendor/{pkg['name']}-{pkg['version']}"
os.makedirs(crate_dir, exist_ok=True)
with open(f"{crate_dir}/.cargo-checksum.json", "w") as f:
f.write(f'{{"package":"{pkg["checksum"]}","files":{{}}}}')
else:
print("huh, found a git one")
sys.exit(1) |
Beta Was this translation helpful? Give feedback.
-
|
@climbfuji this may be of interest to you as well based on the py-maturin pr. |
Beta Was this translation helpful? Give feedback.
-
|
If it's of any use to anyone, I can get around this issue with go- and rust-based packages using "spack develop" and prefetching the deps: spack develop [email protected]
mkdir $SPACK_ENV/cbindgen/.cargo
echo '[source.crates-io]
replace-with = "vendored-sources"
[source.vendored-sources]
directory = "vendor"' > $SPACK_ENV/cbindgen/.cargo/config.toml
cd $SPACK_ENV/cbindgen
cargo vendor
# Later:
spack installspack develop [email protected]
export GOMODCACHE=$SPACK_ENV/go-mod-cache
cd $SPACK_ENV/gh
go mod download
# Later:
export GOMODCACHE=$SPACK_ENV/go-mod-cache
spack install ghI started to go down a rabbit hole of setting up gh's go deps as resources in package.py, but boy what a rabbit hole that turned out to be... (plausible but pretty messy) |
Beta Was this translation helpful? Give feedback.
-
|
Also of possible use: it occurred to me there's nothing stopping us from prefetching packages and their corresponding dependencies. As a probably easier solution than above, here's a script that picks out CargoPackage packages for a concretized environment, fetches the code, and uses 'cargo fetch' to download all dependencies to $CARGO_HOME: https://github.com/AlexanderRichert-NOAA/spack-helpers/blob/main/fetch_cargo_deps.py |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
In trying to create the firefox package (#45083) I had to create with some rust-based dependencies, specifically cbindgen (#45393). I noticed a few things about the rust packages that make them particularly difficult to work with and wanted to see if anyone has any tips. If not, this discussion may serve as a place to generate some ideas for improvements to the
CargoPackageand other aspects of the rust ecosystem.The three main issues complement each other in a way that mask the underlying problem that basically end up bypassing Spack's dependency management.
Cargo, Rust's package manager, will resolve, download, and "install"/stage all of its dependencies a(orutomatically as needed. It also caches these outside of the build tree. Since things will usually "just work," packagers may not notice that a package has (often far more) dependencies. This, of course, breaks once you are inside an air-gapped environment. It was the cause of the issue discussed in #35277 (since
py-cryptographydepends on rust), which ties into some of the auditing tools discussed in the Airgap thread: #44573).When a package maintainer tries to add dependencies more explicitly they will find two other problems. There are lots and lots of Rust packages that are not defined in Spack, even some of the most common ones. This will improve over time, but since things have more-or-less worked, there's a lot to do. Looking at the
Cargo.lockfiles for even some simple packages shows quite a lot of dependencies. It seems like a tool similar topip2spackwould help a great way to help expedite this.Second, the
CargoPackageclass is missing functionality that ends up requiring customization for many packages. My limited understanding of Rust tells me that the default behavior doesn't support virtual manifests which leads to the issue discussed in: rust-lang/cargo#7599. There's an example of a solution to this here: https://stackoverflow.com/questions/61189179/found-a-virtual-manifest-at-path-instead-of-a-package-manifest. For this issue, it seems like having a customizablecargo_pathvariable (orcargo_paths) that defaults to"."would at least provide a way to override without writing a custom build method. There may be a way to derive this from theCargo.tomlfile however, which would be even better.I'm happy to make potential changes to improve this but don't know enough about the Rust/Cargo ecosystem to know if they actually make sense
Beta Was this translation helpful? Give feedback.
All reactions