relay: workflows created pre-#1369 are orphaned — undeletable, undisableable, and fire forever #1593

Activity

1. AaronGoldsmith commented on Jul 9, 2026

   

   AaronGoldsmith

   on Jul 9, 2026

   ContributorAuthor

   More actions

   Update 2026-07-09: the "escape hatch" doesn't work either — orphans are fully unreachable

   Correction to the "How we escaped" section above: we did not actually escape. The name-based deletion sweep appeared to succeed (accepted:true on every run) but relay logs show every attempt was a silent no-op, and the orphan crons kept firing on 07-08 and 07-09.

   A fifth bug: pre-#1369 rows have name = d-tag UUID string

   The ≤ relay-v0.1.1 create path set the row name from the event's name tag, falling back to the d-tag:

   // crates/buzz-relay/src/handlers/command_executor.rs (relay-v0.1.1)
   let workflow_name = extract_tag(event, "name")
       .or_else(|| extract_d_tag(event))

   …and buzz_sdk::build_workflow_def only emits d and h tags — no name tag. So every CLI-created workflow row from that era has name = "<uuid-string>", not the YAML name:.

   Consequences for the orphans:

   • Name-based lookup can never match — the human-readable name isn't in the DB.
   • Sending the stored UUID-string name is impossible — any UUID-shaped a-tag d-value passes Uuid::parse_str and routes to the UUID branch (DELETE … WHERE id = …), which misses because the row id is a different random v4.

   There is no event a client can sign that reaches these rows. Client-side remediation is impossible, which upgrades proposed fix #1 (orphan reconciliation) from "nice to have" to the only fix.

   Evidence (Datadog, service:sprout env:staging)

   • 21× NIP-09 a-tag deletion: no workflow 'sims-daily-pr-triage' found for owner (warn) — one for every name-based deletion attempt by three different owners on 07-07 and 07-09. All received accepted:true (bug Initial release — Sprout Nostr relay with enterprise extensions #2 in action — this silent-success behavior misled us into declaring victory twice).
   • Workflow SendMessage: posting kind 9 event continued firing on schedule: 07-08 09:00/17:00×2, 07-09 09:00.
   • Same ticks show additional orphans on this relay failing invisibly: 6× Workflow run failed: webhook error: channel is archived: 51f7899e…, 1× channel not found: 430f0276…, plus an orphan posting into an unrelated channel under a different owner. ≥10 orphan rows fire daily on sprout-oss staging alone.

   Reconciliation sweep sketch

   Given the name bug, orphans are cheaply identifiable without joining against events:

   -- rows whose name parses as a UUID that differs from the row id
   DELETE FROM workflows
   WHERE name ~* '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
     AND name <> id::text;

   (The stricter "id matches no live 30620 d-tag" join from proposed fix #1 is still the right general-purpose migration; the above catches the pre-#1369 CLI-created population specifically.)

   Until a sweep runs, the affected rows fire forever; our only working mitigation is archiving the destination channel (orphan runs into archived channels error out without posting — confirmed in logs above).
2. Orland00 commented on Jul 21, 2026

   

   Orland00

   on Jul 21, 2026

   More actions

   Confirmed both core bugs against current main, and I'm working on a PR for the going-forward half.

   The validation/execution owner mismatch is real and is the root of the "no workflow found for owner" no-ops. validate_standard_deletion_event (crates/buzz-relay/src/handlers/side_effects.rs) correctly authorizes against the a-tag pubkey (including the is_agent_owner allowance), but handle_a_tag_deletion then parses parts[1] into pubkey_hex and never uses it — both the UUID delete and the name fallback are scoped by actor_bytes (the deletion signer). So a human owner deleting an agent-owned workflow passes validation and then silently deletes nothing. The name fallback additionally inherits LIMIT 1 (no ORDER BY) from find_by_owner_and_name in crates/buzz-db/src/workflow.rs, so with N same-named rows it removes an arbitrary one.

   From a cost angle this class of bug is nasty: a scheduled workflow whose owner has lost every kill switch is recurring compute/API spend with no accountable off switch — and @AaronGoldsmith's update shows accepted: true on no-op deletions actively misleads operators into believing they've stopped it.

   PR scope I have in flight (going-forward fixes only):

   1. handle_a_tag_deletion executes against the a-tag owner (authz unchanged — validation already checks it; this aligns execution with validation).
   2. Name-based deletion removes all matching rows, not LIMIT 1.
   3. No-op deletions are surfaced in the relay response instead of a silent accepted: true (your fix proposal Initial release — Sprout Nostr relay with enterprise extensions #2).

   Question for maintainers on the reconciliation half: given the 2026-07-09 update (pre-#1369 rows have name = d-tag UUID string, so no client-signable event can ever reach them — client-side remediation is impossible), the orphan sweep is the only fix for existing rows. Would you prefer (a) a one-shot migration (0025_…), (b) a startup/admin reconciliation sweep, or (c) a documented scripts/maintenance/ SQL script (existing precedent)? Happy to implement whichever fits, as a separate PR — I'd rather not ship a row-deleting migration without maintainer buy-in on shape.

   — Orlando (Olive One)
3. Orland00

   added a commit that references this issue on Jul 21, 2026

   fix(relay): execute a-tag workflow deletions against the a-tag owner,…

   ...

   103b368
4. Orland00

   added a commit that references this issue on Jul 29, 2026

   fix(relay): execute a-tag workflow deletions against the a-tag owner,…

   ...

   b6af13b