Repository navigation
Commit f53bbd1
fix(bench): mention the orchestrator by pubkey when posting the task (#5136)
## Problem
The harness posts each trial's task via `buzz messages send`, relying on
`@<orchestrator-id>` name resolution. Task text is untrusted payload:
when it contains @-tokens of its own, the CLI's mention resolver tries
to resolve them as channel members, fails, and refuses to send — killing
the trial with `RuntimeLaunchError` before the agent ever saw the task.
Live occurrence: TB 2.1's `large-scale-text-editing` task embeds Vim
macros (`:%normal! @a`). In the tb21-solo-1 run the trial died at
launch:
```
RuntimeLaunchError: buzz messages send ... exited 1:
{"error":"user_error","message":"mention '@A' does not match a current channel member; retry with --mention <pubkey>"}
```
Any TB task whose statement contains @-syntax is silently zeroed this
way.
## Fix
Pass the orchestrator's pubkey as an explicit `--mention` when posting
the task. The CLI demotes unresolved @-tokens in the text to
presentation-only when any explicit identity is supplied, so delivery
still targets exactly the orchestrator and every @-token in the task
statement becomes inert. The harness already holds the orchestrator's
`AgentCredential` (it writes that pubkey into the worker roster tables),
so no persistence is needed — fresh key per trial, fresh `--mention` per
trial.
Verified both halves against a live relay: a fenced `@a` without
`--mention` still hard-fails (the resolver is not markdown-aware); the
same content with an explicit `--mention` sends clean with
`mention_pubkeys` containing only the target.
## Testing
- `benchmarks/harbor-buzz-orchestra`: full pytest suite — 35 passed (34
baseline + new `test_send_mentions_by_pubkey_so_task_text_stays_inert`),
ruff clean. Run against `origin/main` 769ac70 with exactly this patch
applied.
- `testbed`: full pytest suite — 23 passed, 1 skipped; ruff clean.
## Acceptance
A task statement containing arbitrary @-tokens (Vim registers, emails,
decorators) launches and delivers to the orchestrator instead of dying
in `_send`.
Originating Buzz thread:
`buzz://message?channel=c3252dd2-0142-4e01-88c7-a2183c3960a5&id=74a65a0990fd2197882b66b5ea2707169d4a3dbd2020d1610c45150fb99f140b`
Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>1 parent ad92335 commit f53bbd1
2 files changed
Lines changed: 52 additions & 7 deletions
File tree
- benchmarks/harbor-buzz-orchestra
- src/harbor_buzz_orchestra
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
157 | 157 | | |
158 | 158 | | |
159 | 159 | | |
160 | | - | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
161 | 166 | | |
162 | | - | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
163 | 171 | | |
164 | 172 | | |
165 | 173 | | |
| |||
519 | 527 | | |
520 | 528 | | |
521 | 529 | | |
522 | | - | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
523 | 536 | | |
524 | | - | |
525 | | - | |
526 | | - | |
| 537 | + | |
527 | 538 | | |
528 | 539 | | |
529 | 540 | | |
530 | 541 | | |
531 | 542 | | |
532 | 543 | | |
533 | | - | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
534 | 548 | | |
535 | 549 | | |
536 | 550 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
372 | 372 | | |
373 | 373 | | |
374 | 374 | | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
375 | 406 | | |
376 | 407 | | |
377 | 408 | | |
| |||
0 commit comments