Skip to content

Commit 6f2f61c

Browse files
fix(cli): remove the OpenShell first-poll settle wait
fullsend run waited up to 12s after sandbox creation (openShellFirstPollSettle) to work around NVIDIA/OpenShell#3809: on OpenShell 0.1.2 the supervisor's first settings poll reloaded the policy and cut in-flight connections. The upstream fix (NVIDIA/OpenShell#3819) ships in 0.1.3+, and fullsend now pins 0.1.4 in .github/scripts/openshell-version.sh, so every run paid the wait for nothing. - Delete internal/cli/openshell_settle.go and its test. - runAgent: drop the sandboxReadyAt anchor and the 9b-1 wait block. The 9b-2 label is left as is. - hack/gitlab-runner-vm: bump the fallback OPENSHELL_VERSION (used only when the pin file is absent) from 0.1.2 to 0.1.4 in create-gcp-vm.sh, create-openshift-vm.sh and setup.sh. - Correct the runner-VM README and setup.sh header: prepare.sh installs the job image's OpenShell only when it equals the host pin and fails the job otherwise, so a stale VM pin does not self-heal. After a bump, re-run create-*-vm.sh --resume NUMBER from an updated checkout. Note: pre-commit could not fetch remote hook repos (TLS verification failed in the sandbox), so the hooks were run directly: whitespace, EOF, conflict, CRLF checks, shellcheck 0.10.0, gofmt, go vet, lint-broken-symlinks and lychee all passed; gitleaks was covered by scan-secrets. go test ./internal/cli/ has six failures that also fail on main in this sandbox (TestStartFetchService_* cannot dial localhost; TestPostScriptEnv_NoSchemaAppended* see the sandbox's FULLSEND_OUTPUT_SCHEMA and pass with it unset). Closes fullsend-ai#8531
1 parent 07bb0ea commit 6f2f61c

7 files changed

Lines changed: 9 additions & 190 deletions

File tree

‎hack/gitlab-runner-vm/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -312,7 +312,7 @@ VM that was configured and served jobs.
312312
| `RUNNER_TAG` | no | `fullsend-gitlab-runner` | Runner tag for job matching |
313313
| `DRAIN_TIMEOUT_SEC` | no | `600` | Delete mode only: cap in seconds on draining in-flight jobs (SIGQUIT + wait for `runner-*` containers to exit) before deleting the VM. On cap overrun, the script warns and proceeds. See `--no-drain` to skip draining entirely |
314314
| `RUNNER_ACCESS_LEVEL` | no | `not_protected` | `ref_protected` restricts the runner to protected branches and tags, so merge-request pipelines on unprotected source refs never match and sit `pending`. In project mode, `not_protected` means any job on any branch of the project can run; in group mode, any tag-matched job on any branch of any project invited into the group tree runs on this VM (see Security below) |
315-
| `OPENSHELL_VERSION` | no | from `.github/scripts/openshell-version.sh` | OpenShell version used at **VM provision** (Renovate-tracked). Per-job `prepare.sh` re-reads the job image's `openshell --version` and upgrades the host CLI + supervisor when they differ, so a stale VM pin cannot stick. |
315+
| `OPENSHELL_VERSION` | no | from `.github/scripts/openshell-version.sh` | OpenShell version used at **VM provision** (Renovate-tracked); it becomes the host pin. Per-job `prepare.sh` reads the job image's `openshell --version` and installs that version (CLI + supervisor) only when it equals this host pin. A job image on any other version fails the job ("this host only trusts the Renovate-pinned ..."). After an OpenShell bump, re-run `create-gcp-vm.sh` or `create-openshift-vm.sh` with `--resume NUMBER` from an updated checkout (it copies the new pin file and re-runs `setup.sh`) before the VM runs runner images built from the bump. |
316316
| `GITLAB_RUNNER_VERSION` | no | `19.2.1` | gitlab-runner version |
317317
| `REGISTRATION_TOKEN` | setup only | — | GitLab runner registration token |
318318

‎hack/gitlab-runner-vm/create-gcp-vm.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -176,7 +176,7 @@ if [ -f "${_openshell_version_sh}" ]; then
176176
source "${_openshell_version_sh}"
177177
fi
178178
# Fallback only when the pin file is absent; keep in step with openshell-version.sh.
179-
OPENSHELL_VERSION="${OPENSHELL_VERSION:-0.1.2}"
179+
OPENSHELL_VERSION="${OPENSHELL_VERSION:-0.1.4}"
180180
PREFIX="fullsend-gitlab-runner"
181181
# Boot disk size in GiB (gcloud "GB" is GiB). Matches the ~30 GiB guest disks
182182
# of the OpenShift runner fleet (#8163).

‎hack/gitlab-runner-vm/create-openshift-vm.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,7 @@ if [ -f "${_openshell_version_sh}" ]; then
129129
source "${_openshell_version_sh}"
130130
fi
131131
# Fallback only when the pin file is absent; keep in step with openshell-version.sh.
132-
OPENSHELL_VERSION="${OPENSHELL_VERSION:-0.1.2}"
132+
OPENSHELL_VERSION="${OPENSHELL_VERSION:-0.1.4}"
133133
TEMPLATE="${SCRIPT_DIR}/vm.yaml"
134134
PREFIX="fullsend-gitlab-runner"
135135

‎hack/gitlab-runner-vm/setup.sh‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -37,9 +37,11 @@
3737
# GITLAB_RUNNER_VERSION — gitlab-runner version to install (default: 19.2.1)
3838
#
3939
# Note: The OpenShell version sourced here is the *provisioning* pin
40-
# (.github/scripts/openshell-version.sh, Renovate-tracked). Per-job prepare.sh
41-
# re-reads the job image's openshell --version and upgrades the host when they
42-
# differ, so a VM that was created on an older pin still matches the job.
40+
# (.github/scripts/openshell-version.sh, Renovate-tracked) and becomes the host
41+
# pin. Per-job prepare.sh installs the job image's openshell --version only when
42+
# it equals this host pin; a job image on any other version fails the job. After
43+
# an OpenShell bump, re-run create-*-vm.sh --resume NUMBER from an updated
44+
# checkout before the VM runs runner images built from the bump.
4345
set -euo pipefail
4446

4547
GITLAB_URL="${GITLAB_URL:-}"
@@ -58,7 +60,7 @@ if [ -f "${_openshell_version_sh}" ]; then
5860
source "${_openshell_version_sh}"
5961
fi
6062
# Fallback only when the pin file is absent; keep in step with openshell-version.sh.
61-
OPENSHELL_VERSION="${OPENSHELL_VERSION:-0.1.2}"
63+
OPENSHELL_VERSION="${OPENSHELL_VERSION:-0.1.4}"
6264

6365
# Source the executor's gateway helpers (wait_for_openshell_gateway,
6466
# user_systemctl) so configure_per_job_gateway can wait for the seed start

‎internal/cli/openshell_settle.go‎

Lines changed: 0 additions & 58 deletions
This file was deleted.

‎internal/cli/openshell_settle_test.go‎

Lines changed: 0 additions & 115 deletions
This file was deleted.

‎internal/cli/run.go‎

Lines changed: 0 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1923,9 +1923,6 @@ func runAgent(ctx context.Context, agentName, fullsendDir, outputBase, targetRep
19231923
printer.StepFail("Failed to create sandbox")
19241924
return fmt.Errorf("creating sandbox: %w", err)
19251925
}
1926-
// Anchor for waitForOpenShellFirstPoll (OpenShell #3809) before the first
1927-
// network request.
1928-
sandboxReadyAt := time.Now()
19291926
finalizeSandboxSpan(sandboxSpan, nil)
19301927

19311928
if len(runScopedProviders) > 0 {
@@ -2420,13 +2417,6 @@ func runAgent(ctx context.Context, agentName, fullsendDir, outputBase, targetRep
24202417
}
24212418
}
24222419

2423-
// 9b-1. Let OpenShell 0.1.2's first settings poll pass before any network
2424-
// request; it terminates connections in flight (OpenShell #3809). Remove
2425-
// with openShellFirstPollSettle once the pin includes the upstream fix.
2426-
if _, err := waitForOpenShellFirstPoll(ctx, sandboxReadyAt, printer); err != nil {
2427-
return fmt.Errorf("waiting for the sandbox's first policy poll: %w", err)
2428-
}
2429-
24302420
// 9b-2. Pre-flight GitHub API connectivity check.
24312421
// Validates that the sandbox can reach api.github.com through the proxy
24322422
// before starting the agent. Without this, agents that depend on gh CLI

0 commit comments

Comments
 (0)