Skip to content

Commit f25a317

Browse files
authored
feat(bundle): content-addressed run bundles (#568)
* feat(config): record include provenance in the loader and add the workflow.bundle schema block * feat(bundle): add the run bundle collector and content-addressed manifest * feat(bundle): add the content-addressed store and deterministic archive staging * feat(cli): add conductor bundle build and the validate bundle-closure report * docs(examples): run bundle guide, example, and changelog fragment * fix(bundle): honor resolved plugin component switches and skill precedence in the collector * fix(bundle): harden logical-path validation, store self-healing, and manifest serialization * fix(config): preserve authored include paths in provenance * fix(bundle): close symlink and plugin dependency graphs * fix(bundle): make CAS publication portable and race-safe * docs(bundle): document portable store and symlink closure * fix(bundle): use fd-based Windows bundle locks * test(bundle): write archive fixture with canonical newlines
1 parent 20a9a00 commit f25a317

33 files changed

Lines changed: 6326 additions & 35 deletions

‎AGENTS.md‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -95,11 +95,12 @@ step-by-step checklist.
9595

9696
### Core Package Structure (`src/conductor/`)
9797

98-
- **cli/**: Typer-based CLI. Hot-path verbs stay flat (`run`, `resume`, `validate`, `show`, `stop`, `replay`, `update`, `doctor`, `guide`); the long tail is grouped under noun sub-apps registered via `app.add_typer(...)` — `registry`, `plugin` (→ `list` / `fetch`), `mcp` (→ `serve`), `checkpoint` (→ `list`), and `gate` (→ `respond`) — with `rich_help_panel=` organising the root `--help` into *Run & Recover* / *Author & Inspect* / *Environment* / *Interact* / *State* sections (rendered order; determined by first-occurrence order of commands in the Click command list). The old flat commands `checkpoints` and `gate-respond` remain as **hidden deprecated aliases** that print a stderr deprecation warning (noting removal in a future release) and forward to the new grouped commands via a shared impl (issue #275).
99-
- `app.py` - Main entry point, defines the Typer application, flat commands (including `guide`, rich_help_panel="Interact"), and the hidden `checkpoints`/`gate-respond` deprecation aliases. `run`, `resume`, and `validate` accept `--environment` (resolved environment name or file path). **`status` also lists recently-completed runs, not just live ones (R1)** — see the fleet.py bullet below for the identical change to `fleet list`; `status --json` keeps its existing `running` array unchanged and adds a sibling `completed` array, and `--live` restores the exact pre-change scope on both the table and the JSON payload.
98+
- **cli/**: Typer-based CLI. Hot-path verbs stay flat (`run`, `resume`, `validate`, `show`, `stop`, `replay`, `update`, `doctor`, `guide`); the long tail is grouped under noun sub-apps registered via `app.add_typer(...)` — `registry`, `plugin` (→ `list` / `fetch`), `mcp` (→ `serve`), `checkpoint` (→ `list`), `gate` (→ `respond`), and `bundle` (→ `build`) — with `rich_help_panel=` organising the root `--help` into *Run & Recover* / *Author & Inspect* / *Environment* / *Interact* / *State* sections (rendered order; determined by first-occurrence order of commands in the Click command list). The old flat commands `checkpoints` and `gate-respond` remain as **hidden deprecated aliases** that print a stderr deprecation warning (noting removal in a future release) and forward to the new grouped commands via a shared impl (issue #275).
99+
- `app.py` - Main entry point, defines the Typer application, flat commands (including `guide`, rich_help_panel="Interact"), sub-apps (`bundle` under "Author & Inspect"), and the hidden `checkpoints`/`gate-respond` deprecation aliases. `run`, `resume`, and `validate` accept `--environment` (resolved environment name or file path). `validate --environment` reports the offline Bundle Closure section (`_report_bundle_closure`, lazy-gated on `--environment` via `collect_bundle(allow_network=False)`). **`status` also lists recently-completed runs, not just live ones (R1)** — see the fleet.py bullet below for the identical change to `fleet list`; `status --json` keeps its existing `running` array unchanged and adds a sibling `completed` array, and `--live` restores the exact pre-change scope on both the table and the JSON payload.
100100
- `guide.py` - `guide_impl(text, port, token)` behind `conductor guide` (issue #400): resolves the dashboard port via `pid.py::scan_pid_files()` when `--port` is omitted (read-only, matching `app.py::status`'s reasoning), POSTs `{"text": ...}` to `POST /api/guidance`, and maps 403/409/422/connect-error the same way `cli/gate.py::_gate_respond_impl` does. `guide` is a flat top-level command (not a `guide respond` sub-app) since there is exactly one verb
101101
- `checkpoint.py` - `checkpoint` group (`checkpoint list`) + shared `_list_checkpoints_impl` (modeled on `registry.py`)
102102
- `gate.py` - `gate` group (`gate respond`) + shared `_gate_respond_impl` (modeled on `registry.py`). Also consumed directly by the Fleet Manager TUI's gate-resolve action (`conductor.fleet.tui.actions.resolve_gate`) — see the fleet bullet below.
103+
- `bundle.py` - `bundle` group (`bundle build`, issue #527): collects the workflow's complete statically knowable file closure and publishes it into the content-addressed bundle store. Every reference to `conductor.bundle.*` is a **lazy import inside `build_bundle()`'s body**, deliberately: `cli/app.py` imports every `cli/*.py` sub-app module on every `conductor` invocation, and `conductor.bundle` pulls in the collector, which in turn imports Jinja2, plugin/skill registries, and registry cache. A top-level import here would pay that collector/SDK import cost for every command, not just `bundle build`.
103104
- `registry.py` - `registry` group (`list` / `add` / `remove` / `set-default` / `update` / `show`)
104105
- `plugin.py` - `plugin` group (`list` / `fetch`). Deliberately no `update`: a floating ref self-updates and a pinned one is meant not to. `fetch` existing as a separate verb is what keeps `conductor validate` off the network
105106
- `mcp.py` - `mcp` group (`mcp serve`, issue #432): starts an MCP server over stdio exposing every configured registry's workflows as tools (`--registry`/`--allow`/`--deny`/`--workflow-dir` narrow the exposed set; `--toolsets`/`--max-direct-tools`/`--max-wait-seconds`/`--tool-prefix`/`--max-concurrent-runs`/`--introspect-full` control behavior). `no_args_is_help=True` like `checkpoint`/`gate` — this group has no bare-invocation default, unlike `fleet`. Every reference to `conductor.mcp.*` is a **lazy import inside `serve()`'s body**, deliberately: `conductor.mcp.__init__` (the existing MCP *client* package, `mcp/manager.py`) eagerly imports the full `mcp` SDK as a side effect of its own `__init__.py`, and `cli/app.py` imports every `cli/*.py` sub-app module on every `conductor` invocation — so a top-level import here would pay that SDK-import cost for every command, not just `mcp serve`. See `docs/mcp-server.md` for the user-facing guide and `src/conductor/mcp/serve/` below for the server itself.
@@ -112,11 +113,20 @@ step-by-step checklist.
112113
- `update.py` - Update check and version comparison. Upgrades are delegated to the install script (`install.ps1`/`install.sh`); in-process self-upgrade was removed because on Windows the running Python interpreter sits inside the venv `uv tool install --force` is trying to recreate, which fails with "Access is denied". `conductor update` prints the OS-appropriate install-script one-liner; `conductor update --apply` spawns the installer detached (Windows: new console window; POSIX: `os.execvpe` replace) and exits the current process so file locks release. It also prints `_print_extras_note` — the extras `install_hint.py::installed_extras()` reads out of the uv receipt — because `--force` rewrites the tool's entire requirement set and an upgrade that named no extras used to silently uninstall `[tui]`/`[aca]` (issue #441); both install scripts read the same receipt and rebuild the source as `conductor-cli[<extras>] @ <source>`. The startup hint is suppressed by `CONDUCTOR_NO_UPDATE_CHECK=1`, `--silent`, `--help`/`--version`, and the `update` subcommand itself.
113114

114115
- **config/**: YAML loading and Pydantic schema validation
115-
- `schema.py` - Pydantic models for all workflow YAML structures (WorkflowConfig, AgentDef, ParallelGroup, ForEachDef, StepExecutionConfig, ExecutableStepBase, WorkflowDefaults, etc.)
116-
- `loader.py` - YAML parsing with environment variable resolution (${VAR:-default}) and `!file` tag support
116+
- `schema.py` - Pydantic models for all workflow YAML structures (WorkflowConfig, AgentDef, ParallelGroup, ForEachDef, StepExecutionConfig, ExecutableStepBase, WorkflowDefaults, BundleConfig, etc.)
117+
- `loader.py` - YAML parsing with environment variable resolution (${VAR:-default}), `!file` and `!yamlfile` tag support, and include provenance recording (`IncludedFile`/`IncludedFilesGraph`/`load_config_with_graph`, per-load recorder reset).
117118
- `validator.py` - Cross-reference validation (agent names, routes, parallel groups, execution profile references)
118119
- `environment.py` - Execution environment documents: loading, discovery, and resolution. Discovers `.conductor/environments/<name>.yaml` (project ancestor walk up to git root) and `$CONDUCTOR_HOME/environments/<name>.yaml`. Whole-document shadowing (no merge, no inheritance). Plain YAML loading with no `${VAR}` expansion.
119120

121+
- **bundle/**: Content-addressed run bundle packaging for remote execution backends (issue #527 step 4).
122+
- `model.py` - Frozen Pydantic models for the content zone (`BundleManifest`, `BundleEntry`) and ephemeral descriptor metadata (`BundleDescriptor`, `BundleProvenance`, `GitProvenance`, `RegistryProvenance`, `PluginProvenance`, `BundleEnvironmentLink`). The content-zone-vs-provenance rule: `bundle_digest` is computed via `compute_bundle_digest` strictly over sorted entries and skill/plugin topologies; link and provenance fields live only in `BundleDescriptor` and are never stored in the content-addressed store, so content addressing is invariant to git commit, timestamps, or build machine.
123+
- `collector.py` - `collect_bundle(workflow_path, environment=..., allow_network=...)` collects the workflow's complete statically knowable file closure: include graph (`!file`/`!yamlfile`), Jinja2 template closures (`{% include %}`), local/registry sub-workflows, skills, plugins, and declared assets within authorized roots (`workflow.bundle.additional_roots`). Enforces 10,000-entry and 512 MiB caps, raising typed `BundleError`s (`BundleDynamicTemplateError`, `BundleRootEscapeError`, `BundleSymlinkEscapeError`, `BundleCapsError`, `BundleUnfetchedError`, `BundleCycleError`).
124+
- `store.py` - Content-addressed store under `$CONDUCTOR_HOME/cache/bundles/sha256-<hex>/` (the manifest keeps `sha256:<hex>`; the filesystem key is Windows-safe). `publish_bundle` holds a cross-process per-digest lock across validation, repair, staging, and publication; creates deterministic `bundle.tar.gz`; renames the directory atomically; and writes `bundle.json` last as readiness sentinel. Validates reuse via a parse-only sentinel check.
125+
- `archive.py` - `write_bundle_archive` serializes a staged bundle tree into a deterministic, byte-identical `.tar.gz` archive with sorted POSIX paths, zeroed user/group IDs, and pinned `mtime=0`.
126+
- `errors.py` - `BundleError` exception hierarchy.
127+
128+
- **digest.py**: `canonical_json_digest(payload)` — the single canonical JSON digest helper (sorted keys, compact separators, `ensure_ascii=True`, SHA-256 with `sha256:` prefix) used across `environment.py`, `bundle/model.py`, and `engine/run_manifest.py`. A stdlib-only leaf module.
129+
120130
- **plugins/**: Plugin resolution — the *plugin* as the unit of opt-in (issue #378)
121131
- `manifest.py` - The one definition of what a plugin root looks like. Recognises **both** `.claude-plugin/plugin.json` and `.github/plugin/plugin.json` (the latter was the gap: 12 of 13 plugins on an ordinary machine use it, and both resolve at runtime, so recognising only the former was Conductor's own bug). Also parses `mcpServers` in all three forms — a **string path** like `".mcp.json"` (what every MCP-shipping plugin actually writes, so handling only the inline object would find zero servers on all of them), an inline object, or a conventional root `.mcp.json`. Deliberately a **leaf** (imports only `plugins/errors.py`) so `skills/registry.py` can import it without closing a cycle
122132
- `agents.py` - Parses `agents/*.agent.md` into `PluginAgent` specs named `<plugin>:<agent>`. `tools:` entries are the *host CLI's* vocabulary (`read`, `ado/*`) and are forwarded verbatim; `user-invocable` is deliberately not mapped (it gates a human menu, not model dispatch)

‎changelog.d/+run-bundle.added.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
**Run bundles and offline closure reporting**: added `conductor bundle build`
2+
to package self-contained, content-addressed workflow closures into
3+
`$CONDUCTOR_HOME/cache/bundles/`, the `workflow.bundle` YAML block for declaring
4+
static assets and additional authorized roots, and the offline Bundle Closure
5+
section in `conductor validate --environment`.

‎docs/cli-reference.md‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ Complete command-line reference for Conductor.
1616
- [`conductor guide`](#conductor-guide)
1717
- [`conductor checkpoint list`](#conductor-checkpoint-list)
1818
- [`conductor validate`](#conductor-validate)
19+
- [`conductor bundle build`](#conductor-bundle-build)
1920
- [`conductor doctor`](#conductor-doctor)
2021
- [`conductor registry`](#conductor-registry)
2122
- [`conductor mcp serve`](#conductor-mcp-serve)
@@ -996,6 +997,66 @@ When validating workflow profile references, Conductor applies a three-level che
996997
**Warnings** (validation passes with notes):
997998
- **Undeclared dependencies in explicit mode** — agent prompt references `{{ a.output.val }}` but doesn't declare `a.output` in its `input:` list
998999

1000+
### Bundle Closure Report
1001+
1002+
When `--environment` is passed, `conductor validate` computes the workflow's complete offline file closure and prints the **Bundle Closure** report.
1003+
1004+
* **Complete Closure:** If all dependencies are available locally, validation prints the `bundle_digest`, total uncompressed file size, an entry count breakdown by origin kind (`workflow`, `include`, `jinja_include`, `subworkflow`, `subworkflow_registry`, `skill`, `plugin`, `asset`), authorized roots, and git repository dirty status.
1005+
* **Incomplete Closure (Offline Plugin Misses):** `conductor validate` never uses the network. If a remote plugin source declared in `runtime.plugin_sources` is not cached locally under `$CONDUCTOR_HOME/cache/plugins/`, validation succeeds with an `incomplete` status note, reports missing cache identities, and omits the digest.
1006+
* **Bundle Validation Errors:** Local file defects fail validation with exit code 1. These include dynamic Jinja includes (`BundleDynamicTemplateError`), root escapes (`BundleRootEscapeError`, `BundleSymlinkEscapeError`), resource limit violations (`BundleCapsError`), missing declared roots, or circular dependencies.
1007+
1008+
## `conductor bundle build`
1009+
1010+
Build an immutable, content-addressed run bundle for a workflow.
1011+
1012+
Collects the complete statically knowable file closure of the workflow, including external file references (`!file`, `!yamlfile`), Jinja2 template partials, local and registry sub-workflows, skills, plugins, and declared assets. Publishes the resulting tree and deterministic archive to the content-addressed store at `$CONDUCTOR_HOME/cache/bundles/sha256-<hex>/`. The report and manifest retain the canonical `sha256:<hex>` digest spelling; only the directory name uses a hyphen for Windows portability.
1013+
1014+
```bash
1015+
conductor bundle build <workflow.yaml | registry-ref> [OPTIONS]
1016+
```
1017+
1018+
### Arguments
1019+
1020+
| Argument | Description |
1021+
|----------|-------------|
1022+
| `workflow` | Path to a local workflow YAML file or a workflow registry reference (`name`, `name@registry`, `name@registry#ref`). |
1023+
1024+
### Options
1025+
1026+
| Option | Description |
1027+
|--------|-------------|
1028+
| `--environment <name\|PATH>` | Execution environment: a name resolved via `.conductor/environments/` (project then user level) or a path to an environment document. Defaults to the built-in local environment. |
1029+
1030+
### Exit Codes
1031+
1032+
* `0`: Bundle built successfully (or existing cached bundle reused) and content published to the store.
1033+
* `1`: Build failed due to schema errors, dynamic includes, root escapes, missing dependencies, or exceeded limits.
1034+
1035+
### Examples
1036+
1037+
```bash
1038+
# Build a bundle for a local workflow file
1039+
conductor bundle build workflow.yaml
1040+
1041+
# Build with a specific execution environment
1042+
conductor bundle build workflow.yaml --environment demo
1043+
1044+
# Build directly from a workflow registry reference
1045+
conductor bundle build 'qa-bot@official#v1.2.3'
1046+
```
1047+
1048+
### Store Layout
1049+
1050+
Bundles are stored under `$CONDUCTOR_HOME/cache/bundles/sha256-<hex>/`:
1051+
1052+
* `bundle.json`: The serialized `BundleManifest` (readiness sentinel, written last).
1053+
* `bundle.tar.gz`: The reproducible gzip archive containing all files and `.bundle/manifest.json`.
1054+
* `tree/`: The staged directory hierarchy with POSIX-normalized paths (`tree/main/`, `tree/roots/`, etc.).
1055+
1056+
### CI Priming
1057+
1058+
`conductor bundle build` serves as the CI priming step for workflow dependencies. Unlike `conductor validate` (which runs offline), `conductor bundle build` fetches and caches any remote plugin sources declared in `runtime.plugin_sources`. After running `conductor bundle build` during a CI preparation stage, subsequent workflow runs and validations can execute in isolated, offline environments.
1059+
9991060
## `conductor doctor`
10001061

10011062
Report provider and environment diagnostics — a safe, read-only health check

‎docs/configuration.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1042,6 +1042,34 @@ setting — pass `--keep-last` to override the configured value for a single
10421042
invocation, or `--dry-run` to preview what would be deleted without
10431043
actually deleting anything.
10441044

1045+
## Bundle Cache Directory (`~/.conductor/cache/bundles/`)
1046+
1047+
When building content-addressed run bundles (`conductor bundle build`), Conductor stores the resulting package in a local cache directory:
1048+
1049+
```
1050+
~/.conductor/cache/bundles/
1051+
```
1052+
1053+
If `CONDUCTOR_HOME` is set, the cache path defaults to `$CONDUCTOR_HOME/cache/bundles/` instead.
1054+
1055+
### Structure
1056+
1057+
Each bundle is stored under a filesystem-safe encoding of its full content
1058+
digest. The manifest keeps `sha256:<hex>`, while the directory name replaces
1059+
the colon with a hyphen for Windows compatibility:
1060+
1061+
```
1062+
$CONDUCTOR_HOME/cache/bundles/
1063+
sha256-<hex>/
1064+
bundle.json # Manifest sentinel certifying the staged tree
1065+
bundle.tar.gz # Deterministic gzip archive of the bundle
1066+
tree/ # Staged file hierarchy under normalized POSIX paths
1067+
```
1068+
1069+
### Retention and Lifecycle
1070+
1071+
In v1, bundle cache storage grows without automated garbage collection. Stored bundles remain in the cache until removed manually or cleared by CI teardown steps. Rebuilding a workflow whose file closure has not changed reuses the existing bundle directory immediately without re-staging files.
1072+
10451073
## See Also
10461074

10471075
- [Fleet Manager](fleet.md)

0 commit comments

Comments
 (0)