Skip to content

examples: crabbox runner for pr_diff tasks#53

Closed
zozo123 wants to merge 1 commit into
huggingface:mainfrom
zozo123:feat/crabbox-islo-example
Closed

examples: crabbox runner for pr_diff tasks#53
zozo123 wants to merge 1 commit into
huggingface:mainfrom
zozo123:feat/crabbox-islo-example

Conversation

@zozo123

@zozo123 zozo123 commented May 28, 2026 •
edited
Loading

Copy link
Copy Markdown

Summary

  • Adds examples/crabbox/ — a small Python adapter (runner.py, stdlib only) + a crabbox.sh shim — that scores Repo2RLEnv pr_diff tasks on any crabbox sandbox provider via a single --provider flag.
  • Supported via a PROVIDER_CONFIG table: local-container (docker) · islo · e2b · modal · daytona · namespace-devbox · tensorlake. Per-provider flag names are not consistent across crabbox (--islo-image, --e2b-template, --daytona-snapshot, --local-container-image, --namespace-image, …); the table makes the mapping explicit.
  • Unsupported providers (aws, azure, gcp, hetzner, proxmox, ssh) raise a ValueError listing the supported names and explaining why VM providers need a pre-baked image (separate, larger lift).
  • No changes to src/. No new runtime deps. Lives entirely under examples/ + a single test file. Diff: +863 / −0, one commit.

Live demo

▶ https://zozo123.github.io/repo2rlenv-crabbox-demo/ — architecture, the runs below, the full pytest output.

Test plan

  • 20 unit tests in tests/test_examples_crabbox.py parametrize over every supported provider, monkeypatch subprocess.run, and assert the exact crabbox argv constructed (image-flag, workdir-flag, --keep, --allow-env, no-sentinel error path, preflight rejections for missing crabbox / missing ISLO_API_KEY / missing task files, image_ref honored from task.toml, repo/ref passed as positional bash args so a poisoned task.toml cannot shell-inject). No network, no crabbox binary required.
  • 1 live islo smoke in the same file, gated on ISLO_API_KEY + crabbox on PATH (skipped in CI by default). Pulls pallets__click-3466 from the public reference dataset, scores the oracle, asserts final_reward == 1.0.
  • uv run pytest -q — 743 passed, 5 skipped (4 pre-existing gated tests + the live islo one).
  • uv run ruff check . — clean.
  • uv run ruff format --check . — clean.
  • uv build — sdist + wheel OK.

Verified end-to-end on a live islo.dev sandbox

run wall final_reward
single, oracle (pallets__click-3466) ~51 s 1.0
single, empty diff (agent gave up) ~50 s 0.0
batch, 3 tasks -j 3 (click ×2 + chalk) ~57 s 1.0 each

Design notes

  • Reward exfil: islo delegates exec, so crabbox --download / --capture-stdout aren't supported on that provider. The remote script emits a sentinel and cats /logs/verifier/reward.json; the host parses it from the subprocess pipe. Portable across every supported provider.
  • Shell-injection safe: task.repo / task.ref come from a third-party file (task.toml). They're passed as positional $1 / $2 arguments to bash -lc, never interpolated into the script body — a malicious repo = "foo; rm -rf /" cannot escape its variable expansion.
  • image_ref honored: the wrapper defaults --image to task.image_ref (the value pinned by the task.toml), only falling back to python:3.12-slim if the task didn't specify one. A pr_runtime task pinning a bootstrap image is now scored on that image rather than silently downgrading.
  • Preflight checks: missing crabbox binary, missing ISLO_API_KEY (when provider=islo), and missing task files (task.toml / environment/Dockerfile / tests/test.sh / agent patch) all fail fast with actionable messages before any sandbox is leased.
  • Empty patch: [ -s agent.diff ] guards the git apply so "agent gave up" scores its baseline instead of crashing.
  • /workspace rewrite: test.sh hardcodes /workspace as the repo root; islo also uses /workspace as its sync mount. The rewrite is sed -E 's#(^|[^A-Za-z0-9_/])/workspace([^A-Za-z0-9_]|$)#\1/repo\2#g' so it only matches the whole-token path, not strings that merely contain /workspace.
  • Why examples/, not src/: per CLAUDE.md ("No repo2rlenv run / no parallel sandbox runtime — for full tasks, users run harbor run. We're synthesis-only."), this is a downstream-consumer recipe.

Out of scope (happy to follow up if there's interest)

  • pr_runtime / commit_runtime / cve_patches and the other sandbox pipelines — they build a per-repo Docker image during repo2rlenv bootstrap and need docker-in-docker (or a pre-pushed registry image).
  • Promoting this into a first-class --env crabbox backend in Harbor — that belongs upstream in harbor-framework/harbor.
  • VM providers (aws, hetzner, etc.) — currently rejected with a helpful error; supporting them would mean a separate "warmup" path that bakes Python + git into a custom AMI.

@zozo123 zozo123 changed the title examples: crabbox-islo runner for pr_diff tasks examples: crabbox runner for pr_diff tasks May 28, 2026
@zozo123

zozo123 commented May 29, 2026

Copy link
Copy Markdown
Author

Quick status — this is ready for review:

Live demo → https://zozo123.github.io/repo2rlenv-crabbox-demo/ (the same architecture diagram + the three verified runs from the PR description, with the captured reward.json).

CI gate mirrored locally (the three jobs in ci.yml):

$ uv run ruff check .                  → All checks passed!
$ uv run ruff format --check .         → 114 files already formatted
$ uv run pytest -q                     → 738 passed, 5 skipped in 11.18s
$ uv build                             → sdist + wheel OK

The 5 skipped are pre-existing (private HF e2e, hub-build smoke, ghcr live probe, …) plus the live-islo smoke I added — it requires ISLO_API_KEY + crabbox on PATH and is opt-in by design.

Branch is current with main (no rebase needed). Diff is +754 / −0, isolated to examples/crabbox/ + one test file in tests/. No src/ changes, no new runtime deps.

Happy to address any feedback or follow up with a pr_runtime adapter in a separate PR if there's interest.

@zozo123
examples: add crabbox runner for pr_diff tasks
727d33f 
Score Repo2RLEnv pr_diff tasks on any crabbox sandbox provider via a
single --provider flag — local-container (docker), islo, e2b, modal,
daytona, namespace-devbox, tensorlake. Per-provider crabbox flag names
aren't consistent (--islo-image, --e2b-template, --daytona-snapshot,
--namespace-image, ...); a PROVIDER_CONFIG table makes the mapping
explicit. Unsupported providers raise ValueError listing the supported
names and noting why VM providers (aws, azure, gcp, hetzner, proxmox,
ssh) need a separate adapter.

Files:
- examples/crabbox/runner.py — stdlib-only Python (tomllib, argparse,
  concurrent.futures). Single-task + parallel batch (--all -j N -> CSV)
  modes. Honors task.toml's reproducibility.image_ref; --allow-env for
  forwarding ANTHROPIC_API_KEY etc. to the verifier's LLM-judge.
- examples/crabbox/crabbox.sh — 3-line shim that execs runner.py.
- examples/crabbox/README.md — provider matrix, quickstart, scope.
- tests/test_examples_crabbox.py — 21 unit tests parametrized over
  every supported provider (monkeypatched subprocess, no network or
  binary required); preflight error paths (missing crabbox / missing
  ISLO_API_KEY / missing task files); image_ref-default honored;
  task.repo / task.ref passed as positional bash args so a poisoned
  task.toml can't shell-inject. One live islo smoke gated on
  ISLO_API_KEY + crabbox on PATH (opt-in, skipped in CI).

Reward exfil is over a sentinel-delimited stdout because islo's
delegate-exec mode doesn't support crabbox --download / --capture-stdout.
Verified end-to-end on a live islo.dev sandbox: oracle diff -> 1.0,
empty diff -> 0.0, 3-task batch j=3 -> 1.0 each in ~57s wall.

ruff check + ruff format --check clean. pytest -q: 743 passed,
5 skipped (4 pre-existing + the gated live-islo smoke).
@zozo123
zozo123 force-pushed the feat/crabbox-islo-example branch from 2b05de7 to 727d33f Compare June 3, 2026 17:12
@zozo123

zozo123 commented Jun 3, 2026

Copy link
Copy Markdown
Author

Updated the branch (force-pushed) with a self-review pass — squashed to a single commit and addressed five issues a careful reviewer would surface:

  • Shell-injection safe: task.repo / task.ref from a third-party task.toml are now passed as positional $1 / $2 to bash -lc instead of being f-string-interpolated into the script body. A poisoned dataset combined with --allow-env ANTHROPIC_API_KEY can no longer exfiltrate the forwarded key. Covered by test_run_task_passes_repo_and_ref_as_positional_args.
  • task.image_ref honored: was silently overridden by a hardcoded python:3.12-slim. Now defaults to the task's recorded image. Covered by test_run_task_honors_task_image_ref_by_default.
  • git add PATH-strip fixed: the second subprocess.run([\"git\", \"add\", \"-A\"], env={**env}) previously dropped PATH, breaking on macOS Homebrew (/opt/homebrew/bin/git). All three git invocations now share {**os.environ, **GIT_IDENTITY}.
  • Preflight checks restored: missing crabbox binary / missing ISLO_API_KEY / missing task files now fail fast with actionable messages instead of opaque Python exceptions. Three new tests cover the rejection paths.
  • /workspace sed tightened: the path rewrite uses a token-aware pattern ((^|[^A-Za-z0-9_/])/workspace([^A-Za-z0-9_]|$)) so it can't corrupt strings that merely contain /workspace (e.g. a URL).

Tests: 20 unit + 1 gated live. pytest -q → 743 passed, 5 skipped. Branch still current with main.

@adithya-s-k

adithya-s-k commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator

Hey @zozo123, thank you for this, genuinely. I went through it carefully and it's excellent work: stdlib-only, zero src/ changes, injection-safe (repo/ref as positional args), honors the task's image_ref, the reward-exfil-over-stdout trick, and 20+ real tests asserting exact argv. The live islo runs are a great touch.

That said, I don't think it's quite the right fit for this repo, and that's an architecture call, not a quality one. Repo2RLEnv is deliberately synthesis-only and provider-agnostic: we delegate the runtime/consumption layer to Harbor rather than own it (CLAUDE.md: "No repo2rlenv run / no parallel sandbox runtime, we're synthesis-only"). A provider-routing runner, even under examples/, pulls us into a runtime surface, and one bound to a specific external stack (crabbox + islo). There's also overlap: Harbor already supports Docker / Modal / Daytona / E2B, so emitting Harbor format is exactly what lets us not maintain per-provider consumers.

The good news is you already named the right home in your "out of scope" section: a first-class --env crabbox backend upstream in harbor-framework/harbor. That's where it has real leverage, since every Harbor dataset benefits, not just ours.

Too good to bury in examples/. Thanks again 🙏

@zozo123

zozo123 commented Jun 10, 2026

Copy link
Copy Markdown
Author

Thanks @adithya-s-k — that's a completely fair call, and the kind of clear architectural boundary that keeps a project healthy. "Synthesis-only, delegate the runtime layer to Harbor" is exactly the right line, and a provider-routing runner sits on the wrong side of it no matter how small the diff or how carefully it stays out of src/.

You're also right that I'd already named the proper home for this: I'll port it upstream to harbor-framework/harbor as a first-class --env crabbox backend, where the provider-flag mapping table and the reward-exfil-over-stdout pattern can serve every Harbor dataset rather than one repo's examples. I'll link the Harbor PR back to this thread once it's up so anyone landing here can find the living version.

Closing this accordingly. Thanks for the careful read and the kind words about the injection-safety and test work — genuinely appreciated 🙏

@zozo123 zozo123 closed this Jun 10, 2026
@zozo123 zozo123 mentioned this pull request Jun 10, 2026
Open
@zozo123

zozo123 commented Jun 10, 2026

Copy link
Copy Markdown
Author

Following up on the promise above — the Harbor-side stack is up (it predates this thread, in fact):

@adithya-s-k — if you still think a crabbox backend has leverage for Harbor-format datasets, a short note to that effect on harbor#1745 would genuinely help the Harbor maintainers gauge demand. Thanks again for the redirect 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@zozo123 @adithya-s-k