Skip to content

Commit 8cfc874

Browse files
jyaunchesclaude
andcommitted
feat(ci): add ephemeral Brev E2E test infrastructure
Add a GitHub Actions workflow and orchestration script that spins up a fresh Brev CPU instance, bootstraps it with NemoClaw (Node.js, Docker, OpenShell, gateway, sandbox), runs E2E tests, and tears down the instance afterward. Triggered via workflow_dispatch with inputs for branch, PR number, and test suite selection. Also supports workflow_call for reuse from nightly or other workflows. New files: - scripts/e2e-brev-test.sh: ephemeral box lifecycle (create, bootstrap, test, delete) with trap-based cleanup - .github/workflows/hardware-e2e.yaml: workflow with PR check run reporting and comment posting Key design decisions: - Ephemeral instances (not persistent) — no state drift, no idle cost - CPU-only (--cpu 4x16) — OpenShell gateway works without GPU - Brev CLI pinned to v0.6.310 (v0.6.322 removed --cpu flag) - Headless auth via brev login --token using refresh token - Pre-creates onboarding marker to skip interactive first-run prompt - Sets npm_config_prefix for non-root user compatibility Known limitations: - Brev refresh tokens expire (~24 hours) — BREV_API_TOKEN secret needs periodic refresh until a long-lived token solution is found - brev-setup.sh installs vLLM unnecessarily on CPU boxes (~3 min waste) Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
1 parent e1097a6 commit 8cfc874

3 files changed

Lines changed: 282 additions & 0 deletions

File tree

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
name: hardware-e2e
5+
6+
on:
7+
workflow_dispatch:
8+
inputs:
9+
branch:
10+
description: "Branch to test"
11+
required: true
12+
default: "main"
13+
pr_number:
14+
description: "PR number (for status reporting, optional)"
15+
required: false
16+
default: ""
17+
test_suite:
18+
description: "Test suite to run"
19+
required: true
20+
default: "full"
21+
type: choice
22+
options:
23+
- full
24+
- credential-sanitization
25+
- all
26+
workflow_call:
27+
inputs:
28+
branch:
29+
required: true
30+
type: string
31+
pr_number:
32+
required: false
33+
type: string
34+
default: ""
35+
test_suite:
36+
required: true
37+
type: string
38+
default: "full"
39+
secrets:
40+
BREV_API_TOKEN:
41+
required: true
42+
NVIDIA_API_KEY:
43+
required: true
44+
45+
permissions:
46+
contents: read
47+
checks: write
48+
pull-requests: write
49+
50+
concurrency:
51+
group: hardware-e2e
52+
cancel-in-progress: true
53+
54+
jobs:
55+
hardware-e2e:
56+
if: github.repository == 'NVIDIA/NemoClaw'
57+
runs-on: ubuntu-latest
58+
timeout-minutes: 30
59+
steps:
60+
- name: Checkout target branch
61+
uses: actions/checkout@v6
62+
with:
63+
ref: ${{ inputs.branch }}
64+
65+
- name: Create check run (pending)
66+
if: inputs.pr_number != ''
67+
env:
68+
GH_TOKEN: ${{ github.token }}
69+
run: |
70+
PR_SHA=$(gh pr view ${{ inputs.pr_number }} --json headRefOid -q .headRefOid)
71+
CHECK_RUN_ID=$(gh api repos/${{ github.repository }}/check-runs \
72+
-f name="Hardware E2E (${{ inputs.test_suite }})" \
73+
-f head_sha="$PR_SHA" \
74+
-f status="in_progress" \
75+
-f "output[title]=Running on ephemeral Brev instance" \
76+
-f "output[summary]=Tests in progress..." \
77+
--jq '.id')
78+
echo "CHECK_RUN_ID=$CHECK_RUN_ID" >> "$GITHUB_ENV"
79+
echo "PR_SHA=$PR_SHA" >> "$GITHUB_ENV"
80+
81+
- name: Install Brev CLI
82+
run: |
83+
# Pin to v0.6.310 — v0.6.322 removed --cpu flag and defaults to GPU instances
84+
curl -fsSL -o /tmp/brev.tar.gz "https://github.com/brevdev/brev-cli/releases/download/v0.6.310/brev-cli_0.6.310_linux_amd64.tar.gz"
85+
tar -xzf /tmp/brev.tar.gz -C /usr/local/bin brev
86+
chmod +x /usr/local/bin/brev
87+
88+
- name: Run ephemeral Brev E2E
89+
env:
90+
BREV_API_TOKEN: ${{ secrets.BREV_API_TOKEN }}
91+
NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }}
92+
GITHUB_TOKEN: ${{ github.token }}
93+
INSTANCE_NAME: e2e-pr-${{ inputs.pr_number || github.run_id }}
94+
TEST_SUITE: ${{ inputs.test_suite }}
95+
run: bash scripts/e2e-brev-test.sh
96+
97+
- name: Update check run (completed)
98+
if: always() && inputs.pr_number != '' && env.CHECK_RUN_ID != ''
99+
env:
100+
GH_TOKEN: ${{ github.token }}
101+
run: |
102+
CONCLUSION=${{ job.status == 'success' && 'success' || 'failure' }}
103+
gh api repos/${{ github.repository }}/check-runs/${{ env.CHECK_RUN_ID }} \
104+
-X PATCH \
105+
-f status="completed" \
106+
-f conclusion="$CONCLUSION" \
107+
-f "output[title]=Hardware E2E (${{ inputs.test_suite }}): ${CONCLUSION}" \
108+
-f "output[summary]=See workflow run for details."
109+
110+
- name: Post PR comment
111+
if: always() && inputs.pr_number != ''
112+
env:
113+
GH_TOKEN: ${{ github.token }}
114+
run: |
115+
if [ "${{ job.status }}" = "success" ]; then
116+
EMOJI="✅"
117+
STATUS="PASSED"
118+
else
119+
EMOJI="❌"
120+
STATUS="FAILED"
121+
fi
122+
BODY="${EMOJI} **Hardware E2E** (${{ inputs.test_suite }}): **${STATUS}** on branch \`${{ inputs.branch }}\` — [See logs](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})"
123+
gh pr comment ${{ inputs.pr_number }} --repo ${{ github.repository }} --body "$BODY"
124+
125+
- name: Upload test logs
126+
if: failure()
127+
uses: actions/upload-artifact@v4
128+
with:
129+
name: hardware-e2e-logs
130+
path: /tmp/brev-e2e-*.log
131+
if-no-files-found: ignore

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ Thumbs.db
1616
# Project-specific
1717
.claude/
1818
draft_newsletter_*
19+
specs/
1920
vdr-notes/
2021

2122
# Security: secrets, credentials, and keys

‎scripts/e2e-brev-test.sh‎

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
#!/usr/bin/env bash
2+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
# SPDX-License-Identifier: Apache-2.0
4+
#
5+
# Ephemeral Brev E2E test runner.
6+
# Creates a fresh Brev instance, bootstraps it, runs tests, tears it down.
7+
#
8+
# Required env vars:
9+
# BREV_API_TOKEN — Brev refresh token for headless auth
10+
# NVIDIA_API_KEY — passed to VM for inference config during onboarding
11+
# GITHUB_TOKEN — passed to VM for OpenShell binary download
12+
# INSTANCE_NAME — Brev instance name (e.g. pr-156-test)
13+
#
14+
# Optional env vars:
15+
# TEST_SUITE — which test to run: full (default), credential-sanitization, all
16+
# BREV_CPU — CPU spec (default: 4x16)
17+
18+
set -euo pipefail
19+
20+
RED='\033[0;31m'
21+
GREEN='\033[0;32m'
22+
YELLOW='\033[1;33m'
23+
NC='\033[0m'
24+
25+
info() { echo -e "${GREEN}[brev-e2e]${NC} $1"; }
26+
warn() { echo -e "${YELLOW}[brev-e2e]${NC} $1"; }
27+
fail() { echo -e "${RED}[brev-e2e]${NC} $1"; exit 1; }
28+
29+
# ── Validate inputs ──────────────────────────────────────────────
30+
[ -n "${BREV_API_TOKEN:-}" ] || fail "BREV_API_TOKEN not set"
31+
[ -n "${NVIDIA_API_KEY:-}" ] || fail "NVIDIA_API_KEY not set"
32+
[ -n "${GITHUB_TOKEN:-}" ] || fail "GITHUB_TOKEN not set"
33+
[ -n "${INSTANCE_NAME:-}" ] || fail "INSTANCE_NAME not set"
34+
35+
TEST_SUITE="${TEST_SUITE:-full}"
36+
BREV_CPU="${BREV_CPU:-4x16}"
37+
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
38+
REPO_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
39+
LOG_FILE="/tmp/brev-e2e-${INSTANCE_NAME}.log"
40+
REMOTE_EXIT_CODE=0
41+
42+
# ── Cleanup trap — always delete the instance ────────────────────
43+
cleanup() {
44+
info "Cleaning up: deleting instance $INSTANCE_NAME..."
45+
brev delete "$INSTANCE_NAME" 2>/dev/null || warn "Failed to delete instance (may not exist)"
46+
info "Log saved to $LOG_FILE"
47+
}
48+
trap cleanup EXIT
49+
50+
# ── Step 1: Authenticate ────────────────────────────────────────
51+
info "Authenticating with Brev..."
52+
# Pre-create onboarding marker to skip interactive first-run prompt
53+
mkdir -p "$HOME/.brev"
54+
echo '{"step":1,"hasRunBrevShell":true,"hasRunBrevOpen":true}' > "$HOME/.brev/onboarding_step.json"
55+
56+
LOGIN_OUTPUT=$(brev login --token "$BREV_API_TOKEN" 2>&1) || {
57+
echo "$LOGIN_OUTPUT"
58+
fail "Brev login failed"
59+
}
60+
info "Authenticated"
61+
62+
# ── Step 2: Create instance ──────────────────────────────────────
63+
info "Creating instance $INSTANCE_NAME (cpu: $BREV_CPU)..."
64+
brev create "$INSTANCE_NAME" --cpu "$BREV_CPU" --detached 2>&1 || fail "Failed to create instance"
65+
info "Instance creation started"
66+
67+
# ── Step 3: Wait for SSH ─────────────────────────────────────────
68+
info "Waiting for SSH..."
69+
brev refresh 2>/dev/null || true
70+
71+
MAX_ATTEMPTS=60
72+
WAIT_SECONDS=5
73+
for i in $(seq 1 "$MAX_ATTEMPTS"); do
74+
if ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=no -o LogLevel=ERROR "$INSTANCE_NAME" "echo ok" >/dev/null 2>&1; then
75+
info "SSH ready (attempt $i/$MAX_ATTEMPTS)"
76+
break
77+
fi
78+
if [ "$i" -eq "$MAX_ATTEMPTS" ]; then
79+
fail "SSH not ready after $MAX_ATTEMPTS attempts"
80+
fi
81+
# Refresh SSH config periodically in case IP changed
82+
if [ $((i % 5)) -eq 0 ]; then
83+
brev refresh 2>/dev/null || true
84+
fi
85+
sleep "$WAIT_SECONDS"
86+
done
87+
88+
# ── Step 4: Sync code ───────────────────────────────────────────
89+
info "Syncing code to $INSTANCE_NAME..."
90+
REMOTE_HOME=$(ssh -o LogLevel=ERROR "$INSTANCE_NAME" 'echo $HOME')
91+
REMOTE_DIR="${REMOTE_HOME}/nemoclaw"
92+
ssh -o LogLevel=ERROR "$INSTANCE_NAME" "mkdir -p $REMOTE_DIR"
93+
rsync -az --delete \
94+
--exclude node_modules --exclude .git --exclude dist --exclude .venv \
95+
"$REPO_DIR/" "$INSTANCE_NAME:$REMOTE_DIR/"
96+
info "Code synced to $REMOTE_DIR"
97+
98+
# ── Step 5: Bootstrap ───────────────────────────────────────────
99+
info "Bootstrapping VM (Node.js, Docker, OpenShell)..."
100+
ssh -o LogLevel=ERROR "$INSTANCE_NAME" "cd $REMOTE_DIR && \
101+
export NVIDIA_API_KEY='${NVIDIA_API_KEY}' && \
102+
export GITHUB_TOKEN='${GITHUB_TOKEN}' && \
103+
export NEMOCLAW_NON_INTERACTIVE=1 && \
104+
export NEMOCLAW_SANDBOX_NAME=e2e-test && \
105+
bash scripts/brev-setup.sh" 2>&1 | tee -a "$LOG_FILE"
106+
info "Bootstrap complete"
107+
108+
# ── Step 6: Run tests ───────────────────────────────────────────
109+
info "Running test suite: $TEST_SUITE"
110+
111+
run_test() {
112+
local test_script="$1"
113+
local test_name="$2"
114+
info "Running $test_name..."
115+
ssh -o LogLevel=ERROR "$INSTANCE_NAME" "cd $REMOTE_DIR && \
116+
export npm_config_prefix=\$HOME/.local && \
117+
export PATH=\$HOME/.local/bin:\$PATH && \
118+
export NVIDIA_API_KEY='${NVIDIA_API_KEY}' && \
119+
export NEMOCLAW_NON_INTERACTIVE=1 && \
120+
export NEMOCLAW_SANDBOX_NAME=e2e-test && \
121+
bash $test_script" 2>&1 | tee -a "$LOG_FILE"
122+
return "${PIPESTATUS[0]}"
123+
}
124+
125+
case "$TEST_SUITE" in
126+
full)
127+
run_test "test/e2e/test-full-e2e.sh" "full E2E" || REMOTE_EXIT_CODE=$?
128+
;;
129+
credential-sanitization)
130+
run_test "test/e2e/test-credential-sanitization.sh" "credential sanitization" || REMOTE_EXIT_CODE=$?
131+
;;
132+
all)
133+
run_test "test/e2e/test-full-e2e.sh" "full E2E" || REMOTE_EXIT_CODE=$?
134+
if [ "$REMOTE_EXIT_CODE" -eq 0 ]; then
135+
run_test "test/e2e/test-credential-sanitization.sh" "credential sanitization" || REMOTE_EXIT_CODE=$?
136+
fi
137+
;;
138+
*)
139+
fail "Unknown TEST_SUITE: $TEST_SUITE"
140+
;;
141+
esac
142+
143+
# ── Done ─────────────────────────────────────────────────────────
144+
if [ "$REMOTE_EXIT_CODE" -eq 0 ]; then
145+
info "All tests passed"
146+
else
147+
warn "Tests failed (exit code: $REMOTE_EXIT_CODE)"
148+
fi
149+
150+
exit "$REMOTE_EXIT_CODE"

0 commit comments

Comments
 (0)