pr_diff: Harbor-runnable env + 6-component reward + 100-env reference dataset - #40

PR descriptions frequently link back to the answer the agent is supposed
to produce — adding noise that lets a frontier model shortcut by fetching
the linked artifact. We now strip:

  - Closes / Fixes / Resolves #N (already; extended to multi: '#1, #2')
  - See / Refs / Follow-up to #N
  - Markdown issue links [#N](url)
  - Bare github.com /pull/, /issues/, /commit/ URLs
  - Co-authored-by / Signed-off-by / Reviewed-by / Acked-by trailer lines
  - "(#N)" squash-merge suffix on the title

12 new unit tests in tests/test_pipeline_pr_diff.py covering each pattern
+ the end-to-end _build_instruction shape. Full suite at 647 passing.

This is the v0.8.3 Arc 1 optimization landed alongside the full pr_diff
sweep.

Sergio's review on the PR surfaced two additional patterns that the
first-pass strip didn't handle cleanly:

1. `Closes [#1234](url)` — the markdown-link form of a Closes/Fixes/
   Resolves ref. The bare-#N regex didn't match, so the keyword was
   left orphaned in the output (only the [#N](url) part was stripped
   by the issue-link regex).

2. `[descriptive text](https://github.com/x/y/pull/N)` — markdown
   link whose URL points at a GH pull/issues/commit, with any link
   text (not just `[#N]`). The bare-URL regex stripped the URL but
   left empty `[text]()` brackets in the prose.

Two new regexes — _CLOSES_MD_RE / _REFS_MD_RE for the closes-style
markdown variants, _MD_GH_URL_RE for the descriptive-text variant —
run BEFORE the piece-wise regexes so composite patterns are stripped
whole rather than fragmented.

4 new unit tests cover both cases (single, multi-list, descriptive
text, see-with-markdown). Suite at 638 passing.

The whole point of Repo2RLEnv is verifiable RL envs that LLMs can solve.
v0.8.1's pr_diff was text-only — instruction.md + solution/patch.diff,
nothing harbor-runnable. The reward was supposed to be computed by the
consumer externally. That fails the bar.

This change makes pr_diff produce a fully Harbor-runnable env where the
verifier IS the SWE-RL-style sequence-similarity score against the
oracle diff. Same reward function as repo2rlenv.reward (in fact, the
verifier's embedded Python is kept in lockstep with reward.py — pure
stdlib, ~30 lines).

New helpers in src/repo2rlenv/pipelines/pr_diff.py:
- `build_pr_diff_environment_dockerfile(repo_url, base_commit, oracle_diff)`
  - FROM python:3.12-slim + git
  - shallow clone of the repo @ base_commit
  - base64-bakes the oracle diff into /verifier/oracle.patch
  - no bootstrap LLM — ~30s build per task
- `build_pr_diff_eval_script(base_commit)`
  - captures git diff <base_commit> after the agent's edits
  - embeds the diff-similarity Python (base64) — needs only python3 + git
  - writes the score to /logs/verifier/reward.txt

`PRDiffOptions.emit_harbor_env: bool = True` default-on. Set False to
get the v0.8.1 text-only output (consumer computes reward).

End-to-end smoke on pallets/click PR #3508:
- harbor run -a oracle → reward 1.000 (28s)
- harbor run -a claude-code -m anthropic/claude-sonnet-4-6 → reward 0.710 (4m32s)

Both via harbor run on the emitted task as-is. The oracle case proves
the verifier is wired correctly; the Sonnet case proves the agent path
works end-to-end with a real partial-credit reward.

6 new unit tests on the Dockerfile + eval-script builders (base64
encoding, special-char patches, difflib in inline Python, reward.txt
path). 1 existing test updated to reflect the new instruction wording.
Suite at 644 passing.

Replace the single-scalar diff-similarity reward with the SWE-RL-paper-
style multi-component approach. Lifts pr_diff from "coarse training
signal" to "evaluable single-task env" while keeping the cheap/multi-
language property.

New module: src/repo2rlenv/pipelines/_pr_diff_verifier.py
  - Pure-stdlib in-container verifier (read at gen time, base64-baked
    into tests/test.sh, decoded back to a file at run time).
  - Reviewable + unit-testable as ordinary Python — no more opaque
    base64-blob diffs in test.sh.

6 components (sum = 1.0 default weights):
  format_valid     0.05   (predicted text parses as unified diff)
  size_sanity      0.05   (min(o_loc, p_loc) / max — rampage guard)
  file_targeting   0.10   (F1, NOT Jaccard — recall matters more
                          than punishing extras; explained in module)
  region_overlap   0.20   (predicted hunks overlap oracle hunks
                          with 5-line slack — strongest spatial signal)
  similarity       0.20   (SequenceMatcher over +/- lines ONLY —
                          fixes v0.8.1 context-credit inflation)
  llm_judge        0.40   (Haiku scores semantic correctness;
                          graceful degradation on API failure,
                          remaining weights re-normalize)

Verifier outputs reward.txt (single float, harbor reads this) AND
reward.json (full breakdown for downstream inspection / re-weighting).

Gen-time additions in pr_diff.py:
  - Quality filter: drops test-only / docs-only / revert / trivially-
    small diffs / instruction-too-thin candidates before emission.
  - Calibration baseline: empty-patch reward stamped in
    task.toml.metadata.reward_calibration.baseline_reward. Consumers
    compute calibrated = (raw - baseline) / (1 - baseline) for
    cross-task comparability.
  - Difficulty bucket: trivial/small/medium/large by oracle LOC.
  - PRDiffOptions.min_loc_changed: int = 3 (filter knob).

Dockerfile now pre-installs claude-code via npm at build time (apt + npm
layers cacheable across all pr_diff tasks). This makes harbor's
claude-code agent-setup robust at 25-parallel-container scale — no more
per-container curl-install flakiness.

End-to-end smoke on pallets/click PR #3508:
  - harbor run -a oracle: reward 1.000 (8s with cached layers)
  - harbor run -a claude-code -m claude-sonnet-4-6 --ve ANTHROPIC_API_KEY=$X:
    final 0.98 (5m), all 5 deterministic = 1.0, llm_judge = 0.95

Test coverage: 37 new tests in tests/test_pr_diff_verifier.py covering
each component, judge graceful degradation, weight redistribution.
Suite at 681 passing.

Pilot at concurrency=25 (then 12) had 100% Sonnet failure with
AgentSetupTimeoutError after 360s. Root cause: harbor's claude-code
adapter runs `curl claude.ai/install.sh | bash` in EVERY container at
agent-setup time. N parallel ~80MB downloads saturate local bandwidth
and time out.

Fix: pre-install claude-code in the Dockerfile at BUILD time using the
same install.sh route harbor uses. Docker's content-addressable layer
cache means the install runs ONCE per image-build and is shared across
all per-task images (the install RUN line is identical between tasks).
When harbor's adapter re-runs the install script per-container, it
finds the binary already in /root/.local/bin and short-circuits the
network step.

Also drops the nodejs/npm route which was a dead end — harbor's
adapter only uses npm on Alpine images; on python:3.12-slim it always
uses curl install.sh regardless of pre-installed binaries.

Single-task smoke after the fix: 0.858 reward in 3m57s (was 4m56s
before the bake; same task, all 6 components fire including
llm_judge=0.92).

Earlier commit (69d1631) baked claude-code into the task's
environment/Dockerfile via curl claude.ai/install.sh at build time.
That was wrong: the task spec is supposed to be agent-agnostic. The
Dockerfile defines the ENVIRONMENT the task runs in (repo, verifier,
tools) — it shouldn't pre-install a specific agent vendor's binary.

Consequences of the bake:
- ~80MB of claude-code in every image, even when the agent is
  openhands / codex / aider / etc.
- A specific Anthropic CLI on PATH could interfere with what other
  agents do (unlikely but possible).
- Cross-vendor contamination — the dataset implicitly assumes
  claude-code is the runner.

The original problem the bake was supposed to solve: at concurrency≥8,
N parallel `curl claude.ai/install.sh` calls during harbor's claude-code
agent-setup saturate local bandwidth and time out. Correct fixes:

1. Lower concurrency to ≤5 (harbor's own default is 4). Confirmed
   working in the latest smoke: 0.975 reward in 2m33s with no bake.
2. Use --max-retries 2 (already in run_pilot.py).
3. File an upstream harbor issue for install-script caching.

Single-task smoke after the revert: 0.975, 2m33s. The 100-env
reference dataset has been re-generated and re-pushed to HF Hub with
the clean Dockerfile.

Three correctness fixes flagged by an independent audit of PR #40:

1. Self-contained Dockerfile fast path was too broad. It skipped the
   image-push step for ANY non-local FROM ref, including private or
   unqualified images (my-bootstrap:latest, company/base:dev) that a
   consumer couldn't actually pull/rebuild. Narrowed to an explicit
   allowlist of known-public Docker Hub bases (python/node/golang/...).
   Anything else falls through to the normal push path, which surfaces
   a clear error rather than silently publishing a broken dataset.

2. The fast path left stale reproducibility metadata. The emitter seeds
   every Dockerfile task with mode=local_only, image_visibility=private;
   the fast path returned without rewriting it, so published pr_diff
   tasks advertised "private/local_only" despite being publicly
   rebuildable. Now rewrites each task.toml to mode=inline_dockerfile,
   image_visibility=public, with the Dockerfile sha256 stamped for
   traceability.

3. Private repos with emit_harbor_env=True silently produced envs that
   fail at consumer build time (the inlined `git clone` is
   unauthenticated). Now fails fast at run() with a clear message
   pointing at emit_harbor_env=False for text-only output.

Also: corrected the verifier module docstring (said "5-component" +
listed pre-retune weights) and removed four unused _NORMALIZE_RE_*
constants left over from an earlier normalization path.

Tests: +3 (public-base fast path rewrites metadata; non-allowlisted
image surfaces an error; private+emit_harbor_env fails fast). The
private e2e test now passes emit_harbor_env=False explicitly.