Skip to content

Configurable orchestration workflows (user-defined; agent + human-in-the-loop; progress + artifacts) #3871

Open

Description

@dilipgv

Configurable Orchestration Workflows

Problem

Today buzz workflows runs a linear step list with a fixed action set
(send_message | send_dm | set_channel_topic | add_reaction | call_webhook | request_approval | delay).
There is no step that invokes an agent, no branching, no loops, and no first-class
run state for artifacts/progress. Multi-pass processes (e.g. the IMPL workflow: Architect →
Plan Critic → Builder → Inspector → CI → merge) therefore can't be expressed natively — they
require a bespoke controller agent per process. Every new process = new custom code.

Goal

A user-defined, configurable orchestration engine where a workflow is a declarative graph of
steps, each step assigned to a dedicated agent or a human, with the engine tracking run
progress, chat, and artifacts across steps — reusable across many process types
(IMPL, bug triage, review-only, "scan Slack/email → summarize → act").

In scope

  • Step assignees: a step targets a specific agent (by pubkey/role) OR a human (approval/input).
  • Agent-invocation step: a real step type that dispatches a prompt to an assignee and
    waits for a structured result (not fire-and-forget send_message).
  • Control flow: conditional branching + bounded loops (revise/rebuild) with a max-iteration cap.
  • Run state & artifacts: each run tracks status per step, inputs/outputs, and attached
    artifacts (plans, diffs, PR ids, summaries) that later steps can read.
  • Human-in-the-loop: approval/input gates that pause a run and resume on human action
    (extends today's request_approval).
  • Progress & chat surface: a run is observable — current step, history, per-step
    responses/artifacts — visible in-channel.
  • Templates: named, versioned workflow definitions users author and instantiate.

Out of scope (for v1)

  • Arbitrary code execution inside steps (keep steps declarative; escape hatch = call_webhook).
  • Cross-channel orchestration.
  • A visual builder (YAML/JSON first; UI later).

Proposed shape (for design, not final)

  • Definition: extend the workflow YAML with assignee (agent/human/role), a new
    invoke_agent action (prompt + expected structured output schema/verdict grammar), and
    on/when edges for branching + a loop construct with max_iterations.
  • Run model: persistent run object keyed by run id — steps[] with
    status, assignee, input, output, artifacts[], attempts. Verdicts are structured
    tokens (e.g. APPROVED/REVISE/CLEAN) the engine branches on.
  • Artifacts: typed references (PR event id, plan doc, message id, external URL) so steps
    hand real objects forward, not prose.
  • Assignment/dispatch: engine posts the step to its assignee and blocks the run on that
    step until a structured result or human action arrives; bounded loops re-dispatch.

Example workflows this must support

  1. IMPL — Architect(plan) → Plan Critic(loop) → Builder(draft PR) → Inspector(loop) → CI → captain approval+merge. (See thread root for full spec + WORKFLOW STAMP.)
  2. Bug triage — intake → classify(agent) → severity gate(human) → assign → track.
  3. Review-only — diff_posted → reviewer agent(s) → verdict → human sign-off.
  4. Scan → summarize → act — schedule → Slack/email scan(agent) → summarize(agent) →
    proposed actions → human approval → execute.

Key risks / decisions to settle in design

  • Loops & branching are the hard part — the current engine is linear. Adding an
    invoke_agent step that blocks on a structured result is the enabling primitive; loops
    and branches build on it. Needs a bounded-iteration cap to avoid silent stalls.
  • Where run state lives (relay events vs a store) and how artifacts are referenced durably.
  • Assignee resolution: agent-by-role vs by-pubkey; what happens when an assignee is offline.
  • Authorization: today call_webhook needs owner/admin — decide the permission model for
    agent-invocation and human-gate steps.
  • Backward compatibility with existing linear workflows.

Suggested milestones

  • M1: invoke_agent step + structured-result wait + run/artifact state (unlocks single-pass
    agent workflows: review-only, scan→summarize).
  • M2: branching + bounded loops + max_iterations (unlocks IMPL, bug triage).
  • M3: human-in-the-loop input steps beyond approval; progress/chat surfacing in-app.
  • M4: templates (author/version/instantiate) + assignee-by-role.

Open questions for the team

  • Definition format: keep YAML, or move to a richer schema?
  • Do we build this into the existing buzz workflows engine, or a new orchestration service
    that emits/consumes workflow events?
  • Priority order of the four example workflows — which lands first as the proving ground?

Activity

  1. RegardV commented on Jul 31, 2026

    @RegardV

    Strong +1 on step assignees specifically. One piece of evidence for that bullet that isn't in the issue yet: it's worth looking at how a workflow reaches an agent today, because the current mechanism is string matching on display names.

    resolve_mention_pubkeys in crates/buzz-relay/src/workflow_sink.rs reverse-parses @Name out of workflow message text into the p tags that ACP agent-wake (event_mentions_agent) is gated on. Its own doc comment is explicit that this is load-bearing:

    The client resolves mentions to p tags at compose time from an interactive autocomplete pick; the workflow path has only free text, so this reverse-parse defines the matching contract.

    And it documents the constraints that follow:

    • Members only. Candidates are the destination channel's members; global users are never matched.
    • Exact display name. No substring, prefix, or fuzzy matching.
    • Ambiguous names wake no one. If two or more members share the matched display name, no p tag is emitted for it — arbitrary selection would silently misroute and tagging all of them is a false-wake firehose.

    The code is careful and the conservatism is clearly deliberate — but it means dispatch today has properties a workflow author can't see:

    • renaming an agent silently breaks every workflow that targeted it;
    • two agents sharing a display name means the step wakes nobody, with no error;
    • an agent that isn't a member of the destination channel is never a candidate, so dispatch silently stops;
    • an agent with no display_name can't be woken at all — the candidate list is built with filter_map(|u| { let name = u.display_name?; ... }), so it's dropped before matching runs;
    • none of it is checkable when the workflow is authored — only at run time, by nothing happening.

    In each case the kind:9 is still emitted; only the p tag is missing. So the run looks successful and the agent simply never wakes, which is the hardest shape of failure to debug from the outside.

    A structured assignee (pubkey or role, as proposed) turns all four into authoring-time validation errors instead of silent runtime no-ops. That seems worth calling out as motivation independent of the orchestration engine itself — it's a correctness argument, not just an ergonomics one.

  2. HashMac commented on Aug 3, 2026

    @HashMac

    Adding field evidence for two requirements that should be explicit in the run contract: durable recovery after connectivity loss, and structured cross-channel handoffs.

    In a four-channel research mission with several managed agents, I observed three failure shapes:

    • an agent posted STARTED but produced no terminal result, leaving downstream reviewers waiting indefinitely;
    • a child task completed or requested revision in one channel, but nothing reliably woke the parent coordinator in another channel;
    • after a network/provider interruption, the in-progress turn could be stranded until a human noticed and mentioned the agent again.

    These are not prompt-quality problems. A workflow engine needs to distinguish “temporarily unreachable” from “terminally failed” and own the continuation.

    Current-main details make the gap concrete:

    • RunStatus has pending | running | waiting_approval | completed | failed | cancelled, but no retryable/paused state.
    • execute_from_step is an approval-resume path; there is no general lease-expiry or reconnect recovery path.
    • delay is an in-memory sleep capped at 270 seconds, with scheduled resume still noted as future work.
    • Desktop get_workflow_runs currently returns an empty list because authoritative run state is not yet reconstructible for the client.

    Suggested durable-run contract:

    1. Persist the run and step-attempt record before dispatch, with stable run_id, step_id, attempt_id, parent/child IDs and an idempotency key.
    2. Checkpoint after every accepted side effect and structured result: inputs, outputs, artifacts, current/next atomic action, retry count and last error.
    3. Give active attempts a renewable lease/heartbeat. Loss of network, relay, provider or worker moves the run to paused_network or retry_scheduled—not failed.
    4. On reconnect or process restart, reclaim expired leases and resume the same step from its checkpoint. Durable outbox replay must reuse the same signed event/message ID so recovery cannot duplicate a tool call or reply.
    5. Make agent results structured callback events carrying parent_run_id, child_run_id, source channel, step, verdict and artifact references. The parent workflow should subscribe by run ID, not rely on display-name mentions.
    6. Persist approval/authority gates with the run. Resume must never widen tool, channel, spending or external-action permissions.
    7. Use bounded retries with backoff and a visible terminal state. Every non-terminal run should expose its owner, last checkpoint, next action and next retry time.

    A minimal status model could be:

    queued | running | waiting_agent | waiting_human | paused_network | retry_scheduled | completed | failed_terminal | cancelled

    Cross-channel orchestration does not need a visual builder in v1. invoke_agent can accept an optional destination channel, while callback routing remains keyed to the parent run. That would let specialist channels keep their context without making a human copy status between them.

    Incident-shaped acceptance tests:

    • disconnect mid-agent turn and mid-publish, then reconnect;
    • restart the worker/relay/desktop with a running step;
    • complete a child step in another channel and verify the parent wakes;
    • verify exactly one result/message after retry;
    • retain a pending or denied approval across restart;
    • surface an invalid/offline assignee as a waiting/error state rather than a successful silent no-op.

    The same-ID outbound durability work in #4443 looks like a useful transport primitive, but transport recovery alone is not workflow recovery. I would suggest adding a durability milestone before branching/loops, then layering cross-channel callbacks onto the structured-result mechanism.

  3. loganrenz commented on Aug 4, 2026

    @loganrenz

    A supervised v0.5.4 delivery trial provides a concrete minimal workflow for M1/M2 and shows where chat-only orchestration currently leaks work to the human.

    Trial shape

    One team mention assigned:

    1. Claude B: implement a bounded issue in a fresh external worktree and open a draft GitHub PR.
    2. Codex B: wait for an immutable-SHA handoff, independently review that exact SHA, and return structured PASS or FAIL.
    3. On FAIL, Claude B amends and publishes a new exact SHA; Codex B re-reviews.
    4. Merge/deploy remain human-gated and out of scope.

    The workflow completed successfully against narduk-enterprises/operator-portal issue #62 / draft PR #87:

    • implementation SHA bdf115d… passed repository validation and GitHub CI;
    • reviewer returned FAIL because the commit lacked the required human Signed-off-by trailer and retained a dead test construct;
    • implementer amended to d03c29b…, reran focused gates, and force-with-lease pushed;
    • reviewer verified local/origin/live-PR SHA equality and returned PASS;
    • GitHub Build, E2E, and Required checks passed; no merge or deploy occurred.

    What the human still had to orchestrate

    • The initial team mention briefly woke both agents before the reviewer settled into its wait role.
    • The implementer stopped after an intermediate measurement rather than advancing through tests, push, and handoff; several separate wakeups were required.
    • A long-lived session later stopped producing observable activity for more than five minutes and required a manual Desktop Restart, after which it resumed from the existing worktree.
    • There was no authoritative run object saying implementing, validating, waiting_review, revision_required, or passed; that state lived in prose and operator attention.
    • The reviewer verdict was structured in practice (PASS / FAIL) but not an engine-owned callback that could dispatch the next step automatically.

    Minimal acceptance scenario suggested by this run

    A first invoke_agent workflow fixture could encode exactly this loop:

    implement -> validate -> immutable handoff -> review -> (FAIL: repair with new SHA, bounded retry) | (PASS: waiting_human)

    The run should persist agent identity, worktree/branch, current immutable SHA, validation receipts, PR/artifact URL, attempt count, and mutation authority. A stalled agent should become visibly retryable rather than silently depending on another mention. The PASS state must not imply merge or deploy authorization.

    This is consistent with the durable recovery evidence already added above, but contributes a completed code-delivery loop where an independent reviewer caught a real policy defect despite green CI. It demonstrates that invoke_agent plus structured verdicts and bounded retries can deliver immediate value before a visual workflow builder exists.

    No private channel content, credential values, or raw transcripts are included.

  4. samiriasbeck commented on Aug 10, 2026

    @samiriasbeck

    Field evidence from a supervised two-agent review exposed one semantic detail that would be useful to make explicit in M2: N cycles should mean exactly N completed confrontations, not N messages and not N total model turns.

    For a two-agent debate, one cycle is complete only after:

    1. participant A receives the current artifact and publishes a structured position;
    2. participant B receives A's position plus the same artifact revision and publishes a structured challenge/response;
    3. the orchestrator durably records both outputs, their source event IDs, artifact digest/revision, elapsed time, and the next cycle number.

    The run should fail closed rather than count a cycle when either participant is missing, the artifact digest changes without an explicit revision transition, a response is only transport-accepted, or the callback cannot be correlated to (run_id, step_id, attempt_id, cycle).

    Suggested bounded-debate shape for the existing proposal:

    - id: adversarial_review
      action: debate
      participants:
        - pubkey: <agent-a>
          role: proposer
        - pubkey: <agent-b>
          role: critic
      cycles: 5
      artifact: '{{steps.prepare.output.artifact}}'
      completion:
        require_all_participants: true
        verdict_schema: debate-verdict/v1
      timeout_per_turn: 10m

    I would keep this as syntax sugar over the invoke_agent + structured-callback primitive rather than a second execution engine. The durable run state should expose:

    • queued -> dispatched -> accepted -> running -> responded | failed | timed_out per participant turn;
    • monotonic cycle_current / cycle_target;
    • immutable input artifact reference + digest per cycle;
    • explicit terminal reason (completed_exact_cycles, human_stopped, participant_failed, timeout, policy_denied);
    • idempotency keys so reconnect/retry cannot double-count a confrontation;
    • channel UUID on every request/result and a relay-enforced same-channel constraint for v1.

    The same run contract can carry a channel context envelope instead of relying on prompt prose:

    {
      "channel_id": "<uuid>",
      "trigger_event_id": "<event-id>",
      "recent_event_ids": ["..."],
      "canvas_revision": "<event-id-or-null>",
      "summary_revision": "<event-id-or-null>",
      "task_refs": ["..."],
      "attachment_receipts": ["..."],
      "context_generated_at": "<timestamp>"
    }

    Important boundary: the context envelope must be assembled only from events the assignee can read in that channel. A workflow owner must not be able to use its standing authority to inject content from another private channel into the assignee's turn.

    Acceptance cases I would add before enabling loops:

    • exactly 5 cycles produces 10 correlated participant results, no more and no fewer;
    • participant B failure in cycle 3 leaves cycles 1-2 complete and cycle 3 failed, never reported as 3/5 complete;
    • reconnect after dispatch reuses the same attempt/idempotency key and produces one result;
    • renamed/duplicate display names are irrelevant because assignees are pubkeys;
    • an assignee removed from the channel before cycle 4 is denied and the run terminates visibly;
    • an attachment digest mismatch prevents dispatch and emits a terminal materialization error;
    • every state transition is observable without logging prompts, message bodies, credentials, or tool arguments.

    This builds directly on the invoke_agent, bounded-loop, artifact and durable-recovery direction already described here, while providing a concrete proving workflow for M1/M2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions