Skip to content

Commit f53bbd1

Browse files
tlongwell-blockEva
andauthored
fix(bench): mention the orchestrator by pubkey when posting the task (#5136)
## Problem The harness posts each trial's task via `buzz messages send`, relying on `@<orchestrator-id>` name resolution. Task text is untrusted payload: when it contains @-tokens of its own, the CLI's mention resolver tries to resolve them as channel members, fails, and refuses to send — killing the trial with `RuntimeLaunchError` before the agent ever saw the task. Live occurrence: TB 2.1's `large-scale-text-editing` task embeds Vim macros (`:%normal! @a`). In the tb21-solo-1 run the trial died at launch: ``` RuntimeLaunchError: buzz messages send ... exited 1: {"error":"user_error","message":"mention '@A' does not match a current channel member; retry with --mention <pubkey>"} ``` Any TB task whose statement contains @-syntax is silently zeroed this way. ## Fix Pass the orchestrator's pubkey as an explicit `--mention` when posting the task. The CLI demotes unresolved @-tokens in the text to presentation-only when any explicit identity is supplied, so delivery still targets exactly the orchestrator and every @-token in the task statement becomes inert. The harness already holds the orchestrator's `AgentCredential` (it writes that pubkey into the worker roster tables), so no persistence is needed — fresh key per trial, fresh `--mention` per trial. Verified both halves against a live relay: a fenced `@a` without `--mention` still hard-fails (the resolver is not markdown-aware); the same content with an explicit `--mention` sends clean with `mention_pubkeys` containing only the target. ## Testing - `benchmarks/harbor-buzz-orchestra`: full pytest suite — 35 passed (34 baseline + new `test_send_mentions_by_pubkey_so_task_text_stays_inert`), ruff clean. Run against `origin/main` 769ac70 with exactly this patch applied. - `testbed`: full pytest suite — 23 passed, 1 skipped; ruff clean. ## Acceptance A task statement containing arbitrary @-tokens (Vim registers, emails, decorators) launches and delivers to the orchestrator instead of dying in `_send`. Originating Buzz thread: `buzz://message?channel=c3252dd2-0142-4e01-88c7-a2183c3960a5&id=74a65a0990fd2197882b66b5ea2707169d4a3dbd2020d1610c45150fb99f140b` Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
1 parent ad92335 commit f53bbd1

2 files changed

Lines changed: 52 additions & 7 deletions

File tree

‎benchmarks/harbor-buzz-orchestra/src/harbor_buzz_orchestra/container_runtime.py‎

Lines changed: 21 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -157,9 +157,17 @@ async def run(
157157
)
158158
# The task arrives exactly as it would in production Buzz: a
159159
# user prompt @mentioning the orchestrator. The harness never
160-
# speaks as any agent.
160+
# speaks as any agent. The orchestrator is mentioned by pubkey,
161+
# not by name resolution: task text is untrusted payload, and any
162+
# @-token inside it (e.g. Vim's `:%normal! @a`) would otherwise
163+
# fail member resolution and kill the trial before the agent
164+
# ever saw the task. An explicit --mention demotes unresolved
165+
# @-tokens in the text to presentation-only.
161166
await self._send(
162-
trial.user, trial, f"@{orchestrator.agent_id} {instruction}"
167+
trial.user,
168+
trial,
169+
f"@{orchestrator.agent_id} {instruction}",
170+
mention=orchestrator.nostr_pubkey,
163171
)
164172
final_message = await asyncio.wait_for(
165173
self._wait_for_done(environment, orchestrator, trial, agents + infra),
@@ -519,18 +527,24 @@ async def _verify_m1_output(
519527
)
520528

521529
async def _send(
522-
self, credential: AgentCredential, trial: TrialHandle, content: str
530+
self,
531+
credential: AgentCredential,
532+
trial: TrialHandle,
533+
content: str,
534+
*,
535+
mention: str | None = None,
523536
) -> None:
524-
await self._buzz_json(
525-
credential,
526-
trial,
537+
args = [
527538
"messages",
528539
"send",
529540
"--channel",
530541
trial.channel_id,
531542
"--content",
532543
content,
533-
)
544+
]
545+
if mention is not None:
546+
args += ["--mention", mention]
547+
await self._buzz_json(credential, trial, *args)
534548

535549
async def _buzz_json(
536550
self, credential: AgentCredential, trial: TrialHandle, *args: str

‎benchmarks/harbor-buzz-orchestra/tests/test_container_runtime.py‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -372,6 +372,37 @@ async def test_m1_output_probe_matches_grader_and_is_condition_scoped(
372372
assert bool(probed) == (condition == "M1-hello-world")
373373

374374

375+
async def test_send_mentions_by_pubkey_so_task_text_stays_inert(
376+
tmp_path, monkeypatch
377+
):
378+
"""Task text is untrusted payload: `:%normal! @a` in a task statement must
379+
not be fed to member-name resolution (it would fail and kill the trial).
380+
An explicit --mention pins delivery to the orchestrator's pubkey."""
381+
rt = runtime(tmp_path)
382+
orch = credential("orch-1", "orchestrator", "orch-model")
383+
trial = trial_handle((orch,))
384+
calls = []
385+
386+
async def buzz_json(credential, trial, *args):
387+
calls.append(args)
388+
return {}
389+
390+
monkeypatch.setattr(rt, "_buzz_json", buzz_json)
391+
392+
await rt._send(
393+
trial.user,
394+
trial,
395+
"@orch-1 run `:%normal! @a` on the file",
396+
mention=orch.nostr_pubkey,
397+
)
398+
assert calls[-1][-2:] == ("--mention", "pubkey-orch-1")
399+
400+
# Without an explicit mention the send is unchanged (name resolution).
401+
await rt._send(trial.user, trial, "plain content")
402+
assert "--mention" not in calls[-1]
403+
assert calls[-1][-2:] == ("--content", "plain content")
404+
405+
375406
async def test_wait_for_done_requires_orchestrator_authorship(tmp_path, monkeypatch):
376407
rt = runtime(tmp_path, poll_seconds=0)
377408
orch = credential("orch-1", "orchestrator", "orch-model")

0 commit comments

Comments
 (0)