This is great, and actually it is a massive gain on time!

I'm curious about what would be the result of this (parallelism) vs concurrency. In theory, concurrency works better for I/O operations so I think ThreadPoolExecutor could work good here (or maybe just asyncio tasks)

In any case, this is massive improvement :)

This is great, and actually it is a massive gain on time!

I'm curious about what would be the result of this (parallelism) vs concurrency. In theory, concurrency works better for I/O operations so I think ThreadPoolExecutor could work good here (or maybe just asyncio tasks)

In any case, this is massive improvement :)

👍

something that can be experimented later.

tested in a real PR here. i've added the parallel build on top of #606 and made a draft pr. i've had to adjust the pipelines a bit and, eventually, ended up cherrypicking install_slices.py into the branch from which i'm making the PR from, but the files are identical to the ones in dcafc48

in the future, can you make a branch named something other than 'main' to PR from? (ideally <jira-ticket>/<whatever>, e.g. ROCKS-2300/libgcc-13-14-dev-25.04) it was a bit of a pain to juggle my local branches to check out the pr

hmmm... if you compare the install times in the CI between the normal version (#606) and the parallel installation test (#607) they take ca. 20min and ca. 6h (timeout) so i think either i've done something wrong in #606, although there is no git diff ... between this pr and #607 install_slices.py, and the only diff in install-slices.yaml is to coax it into running the workflow from the PR branch as opposed to defaulting to canonical/chisel-releases/main

EDIT:

i cannot get to the bottom of this(!) 😕 i've tried to reproduce the installation locally with this script (run from the root of #607). i've limited the CPUs to mimmic a bit more constrained environment of the CI, but i still got the expected result of about 660s with parallel execution and 1980s withhout. less than 3x speedup, but still a speedup, not the drastic slowdown we see in CI. i think the CI is doing some kindof resource sharing between the 14 jobs (7 arch's 2 chisel versions matrix) we're throwing at it, and the context switch with the additional jobs is too much.

EDIT 2:

turns out all was working correctly, just the way i've set up #607 it was installing all the slices as opposed to just the delta. i'll leave the comment as a chain of thought, but not applicable anymore. 👍

#!/usr/bin/bash

set -e

if [ "$1" = "--run" ]; then
    docker run \
        -it --rm \
        -v.:/mnt \
        --cpus 4 \
        --cpuset-cpus 0-3 \
        ubuntu:25.04 bash \
        /mnt/test_install_slices.sh
    exit 0
fi


export DEBIAN_FRONTEND=noninteractive
export TZ=Etc/UTC

cd /mnt
apt update
apt install -yq $(cat .github/scripts/install-slices/deb-requirements.txt)
apt install -yq python3-pip
pip install -r .github/scripts/install-slices/requirements.txt --break-system-packages

apt install -yq golang
go install "github.com/canonical/chisel/cmd/chisel@latest"
ln -s "$(go env GOPATH)/bin/chisel" /usr/bin/chisel

ARCH=$(uname -m)
[ "$ARCH" = "x86_64" ] && ARCH="amd64"
[ "$ARCH" = "aarch64" ] && ARCH="arm64"

# Trim to the first 50 SDFs for testing purposes
SDFS=$(find slices -type f -name "*.yaml")
SDFS=$(echo "$SDFS" | head -n 50)

date +%s > /mnt/start_time.txt

python3 .github/scripts/install-slices/install_slices.py \
    --arch "${ARCH}" \
    --release ./ \
    --ensure-existence \
    --ignore-missing \
    --workers 20 \
    ${SDFS[@]}

date +%s > /mnt/end_time.txt

DELTA=$(($(cat /mnt/end_time.txt) - $(cat /mnt/start_time.txt)))
echo "NEW: $DELTA seconds"

so, as an aside, i've been thinking what takes so much time in this CI anyway, and i've managed to speed it up by another 70%, down to about 210s on the test case above, and it never went above 50% CPU usage on any of the cores while previously being nearly saturated (aka, not CPU bound anymore).

we run the validation inside of the chisel on each and every run we do, but we're always using the same release, so we really need to validate it only once. hence the trick is to install chisel like this: 😅

# clone v1.2.0 of chisel
git config --global advice.detachedHead false
git clone --branch v1.2.0 https://github.com/canonical/chisel.git

cd chisel

# remove validation of release
git apply - << 'EOF'
diff --git a/internal/setup/setup.go b/internal/setup/setup.go
index 18f25f0..35ae23d 100644
--- a/internal/setup/setup.go
+++ b/internal/setup/setup.go
@@ -172,10 +172,6 @@ func ReadRelease(dir string) (*Release, error) {
 		return nil, err
 	}
 
-	err = release.validate()
-	if err != nil {
-		return nil, err
-	}
 	return release, nil
 }

EOF

# build chisel
go build -o chisel ./cmd/chisel
ln -s "$(pwd)/chisel" /usr/bin/chisel

rather than doing this ☝️ in the ci though, i would say that this is a strong supporting argument/use-case for a PR to chisel with some kind --skip-validation