You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f25a317
Browse filesBrowse the repository at this point in the historyBrowse files
feat(bundle): content-addressed run bundles (#568)
* feat(config): record include provenance in the loader and add the workflow.bundle schema block
* feat(bundle): add the run bundle collector and content-addressed manifest
* feat(bundle): add the content-addressed store and deterministic archive staging
* feat(cli): add conductor bundle build and the validate bundle-closure report
* docs(examples): run bundle guide, example, and changelog fragment
* fix(bundle): honor resolved plugin component switches and skill precedence in the collector
* fix(bundle): harden logical-path validation, store self-healing, and manifest serialization
* fix(config): preserve authored include paths in provenance
* fix(bundle): close symlink and plugin dependency graphs
* fix(bundle): make CAS publication portable and race-safe
* docs(bundle): document portable store and symlink closure
* fix(bundle): use fd-based Windows bundle locks
* test(bundle): write archive fixture with canonical newlines
Copy file name to clipboardExpand all lines: AGENTS.md
+14-4Lines changed: 14 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -95,11 +95,12 @@ step-by-step checklist.
95
95
96
96
### Core Package Structure (`src/conductor/`)
97
97
98
-
-**cli/**: Typer-based CLI. Hot-path verbs stay flat (`run`, `resume`, `validate`, `show`, `stop`, `replay`, `update`, `doctor`, `guide`); the long tail is grouped under noun sub-apps registered via `app.add_typer(...)` — `registry`, `plugin` (→ `list` / `fetch`), `mcp` (→ `serve`), `checkpoint` (→ `list`), and `gate` (→ `respond`) — with `rich_help_panel=` organising the root `--help` into *Run & Recover* / *Author & Inspect* / *Environment* / *Interact* / *State* sections (rendered order; determined by first-occurrence order of commands in the Click command list). The old flat commands `checkpoints` and `gate-respond` remain as **hidden deprecated aliases** that print a stderr deprecation warning (noting removal in a future release) and forward to the new grouped commands via a shared impl (issue #275).
99
-
-`app.py` - Main entry point, defines the Typer application, flat commands (including `guide`, rich_help_panel="Interact"), and the hidden `checkpoints`/`gate-respond` deprecation aliases. `run`, `resume`, and `validate` accept `--environment` (resolved environment name or file path). **`status` also lists recently-completed runs, not just live ones (R1)** — see the fleet.py bullet below for the identical change to `fleet list`; `status --json` keeps its existing `running` array unchanged and adds a sibling `completed` array, and `--live` restores the exact pre-change scope on both the table and the JSON payload.
98
+
-**cli/**: Typer-based CLI. Hot-path verbs stay flat (`run`, `resume`, `validate`, `show`, `stop`, `replay`, `update`, `doctor`, `guide`); the long tail is grouped under noun sub-apps registered via `app.add_typer(...)` — `registry`, `plugin` (→ `list` / `fetch`), `mcp` (→ `serve`), `checkpoint` (→ `list`), `gate` (→ `respond`), and `bundle` (→ `build`) — with `rich_help_panel=` organising the root `--help` into *Run & Recover* / *Author & Inspect* / *Environment* / *Interact* / *State* sections (rendered order; determined by first-occurrence order of commands in the Click command list). The old flat commands `checkpoints` and `gate-respond` remain as **hidden deprecated aliases** that print a stderr deprecation warning (noting removal in a future release) and forward to the new grouped commands via a shared impl (issue #275).
99
+
-`app.py` - Main entry point, defines the Typer application, flat commands (including `guide`, rich_help_panel="Interact"), sub-apps (`bundle` under "Author & Inspect"), and the hidden `checkpoints`/`gate-respond` deprecation aliases. `run`, `resume`, and `validate` accept `--environment` (resolved environment name or file path). `validate --environment` reports the offline Bundle Closure section (`_report_bundle_closure`, lazy-gated on `--environment` via `collect_bundle(allow_network=False)`). **`status` also lists recently-completed runs, not just live ones (R1)** — see the fleet.py bullet below for the identical change to `fleet list`; `status --json` keeps its existing `running` array unchanged and adds a sibling `completed` array, and `--live` restores the exact pre-change scope on both the table and the JSON payload.
100
100
-`guide.py` - `guide_impl(text, port, token)` behind `conductor guide` (issue #400): resolves the dashboard port via `pid.py::scan_pid_files()` when `--port` is omitted (read-only, matching `app.py::status`'s reasoning), POSTs `{"text": ...}` to `POST /api/guidance`, and maps 403/409/422/connect-error the same way `cli/gate.py::_gate_respond_impl` does. `guide` is a flat top-level command (not a `guide respond` sub-app) since there is exactly one verb
101
101
-`checkpoint.py` - `checkpoint` group (`checkpoint list`) + shared `_list_checkpoints_impl` (modeled on `registry.py`)
102
102
-`gate.py` - `gate` group (`gate respond`) + shared `_gate_respond_impl` (modeled on `registry.py`). Also consumed directly by the Fleet Manager TUI's gate-resolve action (`conductor.fleet.tui.actions.resolve_gate`) — see the fleet bullet below.
103
+
-`bundle.py` - `bundle` group (`bundle build`, issue #527): collects the workflow's complete statically knowable file closure and publishes it into the content-addressed bundle store. Every reference to `conductor.bundle.*` is a **lazy import inside `build_bundle()`'s body**, deliberately: `cli/app.py` imports every `cli/*.py` sub-app module on every `conductor` invocation, and `conductor.bundle` pulls in the collector, which in turn imports Jinja2, plugin/skill registries, and registry cache. A top-level import here would pay that collector/SDK import cost for every command, not just `bundle build`.
-`plugin.py` - `plugin` group (`list` / `fetch`). Deliberately no `update`: a floating ref self-updates and a pinned one is meant not to. `fetch` existing as a separate verb is what keeps `conductor validate` off the network
105
106
-`mcp.py` - `mcp` group (`mcp serve`, issue #432): starts an MCP server over stdio exposing every configured registry's workflows as tools (`--registry`/`--allow`/`--deny`/`--workflow-dir` narrow the exposed set; `--toolsets`/`--max-direct-tools`/`--max-wait-seconds`/`--tool-prefix`/`--max-concurrent-runs`/`--introspect-full` control behavior). `no_args_is_help=True` like `checkpoint`/`gate` — this group has no bare-invocation default, unlike `fleet`. Every reference to `conductor.mcp.*` is a **lazy import inside `serve()`'s body**, deliberately: `conductor.mcp.__init__` (the existing MCP *client* package, `mcp/manager.py`) eagerly imports the full `mcp` SDK as a side effect of its own `__init__.py`, and `cli/app.py` imports every `cli/*.py` sub-app module on every `conductor` invocation — so a top-level import here would pay that SDK-import cost for every command, not just `mcp serve`. See `docs/mcp-server.md` for the user-facing guide and `src/conductor/mcp/serve/` below for the server itself.
@@ -112,11 +113,20 @@ step-by-step checklist.
112
113
- `update.py` - Update check and version comparison. Upgrades are delegated to the install script (`install.ps1`/`install.sh`); in-process self-upgrade was removed because on Windows the running Python interpreter sits inside the venv `uv tool install --force` is trying to recreate, which fails with "Access is denied". `conductor update` prints the OS-appropriate install-script one-liner; `conductor update --apply` spawns the installer detached (Windows: new console window; POSIX: `os.execvpe` replace) and exits the current process so file locks release. It also prints `_print_extras_note` — the extras `install_hint.py::installed_extras()` reads out of the uv receipt — because `--force` rewrites the tool's entire requirement set and an upgrade that named no extras used to silently uninstall `[tui]`/`[aca]` (issue #441); both install scripts read the same receipt and rebuild the source as `conductor-cli[<extras>] @ <source>`. The startup hint is suppressed by `CONDUCTOR_NO_UPDATE_CHECK=1`, `--silent`, `--help`/`--version`, and the `update` subcommand itself.
113
114
114
115
-**config/**: YAML loading and Pydantic schema validation
115
-
-`schema.py` - Pydantic models for all workflow YAML structures (WorkflowConfig, AgentDef, ParallelGroup, ForEachDef, StepExecutionConfig, ExecutableStepBase, WorkflowDefaults, etc.)
116
-
-`loader.py` - YAML parsing with environment variable resolution (${VAR:-default})and `!file` tag support
116
+
-`schema.py` - Pydantic models for all workflow YAML structures (WorkflowConfig, AgentDef, ParallelGroup, ForEachDef, StepExecutionConfig, ExecutableStepBase, WorkflowDefaults, BundleConfig, etc.)
117
+
-`loader.py` - YAML parsing with environment variable resolution (${VAR:-default}), `!file`and `!yamlfile` tag support, and include provenance recording (`IncludedFile`/`IncludedFilesGraph`/`load_config_with_graph`, per-load recorder reset).
-`environment.py` - Execution environment documents: loading, discovery, and resolution. Discovers `.conductor/environments/<name>.yaml` (project ancestor walk up to git root) and `$CONDUCTOR_HOME/environments/<name>.yaml`. Whole-document shadowing (no merge, no inheritance). Plain YAML loading with no `${VAR}` expansion.
119
120
121
+
-**bundle/**: Content-addressed run bundle packaging for remote execution backends (issue #527 step 4).
122
+
-`model.py` - Frozen Pydantic models for the content zone (`BundleManifest`, `BundleEntry`) and ephemeral descriptor metadata (`BundleDescriptor`, `BundleProvenance`, `GitProvenance`, `RegistryProvenance`, `PluginProvenance`, `BundleEnvironmentLink`). The content-zone-vs-provenance rule: `bundle_digest` is computed via `compute_bundle_digest` strictly over sorted entries and skill/plugin topologies; link and provenance fields live only in `BundleDescriptor` and are never stored in the content-addressed store, so content addressing is invariant to git commit, timestamps, or build machine.
123
+
-`collector.py` - `collect_bundle(workflow_path, environment=..., allow_network=...)` collects the workflow's complete statically knowable file closure: include graph (`!file`/`!yamlfile`), Jinja2 template closures (`{% include %}`), local/registry sub-workflows, skills, plugins, and declared assets within authorized roots (`workflow.bundle.additional_roots`). Enforces 10,000-entry and 512 MiB caps, raising typed `BundleError`s (`BundleDynamicTemplateError`, `BundleRootEscapeError`, `BundleSymlinkEscapeError`, `BundleCapsError`, `BundleUnfetchedError`, `BundleCycleError`).
124
+
-`store.py` - Content-addressed store under `$CONDUCTOR_HOME/cache/bundles/sha256-<hex>/` (the manifest keeps `sha256:<hex>`; the filesystem key is Windows-safe). `publish_bundle` holds a cross-process per-digest lock across validation, repair, staging, and publication; creates deterministic `bundle.tar.gz`; renames the directory atomically; and writes `bundle.json` last as readiness sentinel. Validates reuse via a parse-only sentinel check.
125
+
-`archive.py` - `write_bundle_archive` serializes a staged bundle tree into a deterministic, byte-identical `.tar.gz` archive with sorted POSIX paths, zeroed user/group IDs, and pinned `mtime=0`.
126
+
-`errors.py` - `BundleError` exception hierarchy.
127
+
128
+
-**digest.py**: `canonical_json_digest(payload)` — the single canonical JSON digest helper (sorted keys, compact separators, `ensure_ascii=True`, SHA-256 with `sha256:` prefix) used across `environment.py`, `bundle/model.py`, and `engine/run_manifest.py`. A stdlib-only leaf module.
129
+
120
130
-**plugins/**: Plugin resolution — the *plugin* as the unit of opt-in (issue #378)
121
131
-`manifest.py` - The one definition of what a plugin root looks like. Recognises **both**`.claude-plugin/plugin.json` and `.github/plugin/plugin.json` (the latter was the gap: 12 of 13 plugins on an ordinary machine use it, and both resolve at runtime, so recognising only the former was Conductor's own bug). Also parses `mcpServers` in all three forms — a **string path** like `".mcp.json"` (what every MCP-shipping plugin actually writes, so handling only the inline object would find zero servers on all of them), an inline object, or a conventional root `.mcp.json`. Deliberately a **leaf** (imports only `plugins/errors.py`) so `skills/registry.py` can import it without closing a cycle
122
132
-`agents.py` - Parses `agents/*.agent.md` into `PluginAgent` specs named `<plugin>:<agent>`. `tools:` entries are the *host CLI's* vocabulary (`read`, `ado/*`) and are forwarded verbatim; `user-invocable` is deliberately not mapped (it gates a human menu, not model dispatch)
@@ -996,6 +997,66 @@ When validating workflow profile references, Conductor applies a three-level che
996
997
**Warnings** (validation passes with notes):
997
998
-**Undeclared dependencies in explicit mode** — agent prompt references `{{ a.output.val }}` but doesn't declare `a.output` in its `input:` list
998
999
1000
+
### Bundle Closure Report
1001
+
1002
+
When `--environment` is passed, `conductor validate` computes the workflow's complete offline file closure and prints the **Bundle Closure** report.
1003
+
1004
+
***Complete Closure:** If all dependencies are available locally, validation prints the `bundle_digest`, total uncompressed file size, an entry count breakdown by origin kind (`workflow`, `include`, `jinja_include`, `subworkflow`, `subworkflow_registry`, `skill`, `plugin`, `asset`), authorized roots, and git repository dirty status.
1005
+
***Incomplete Closure (Offline Plugin Misses):**`conductor validate` never uses the network. If a remote plugin source declared in `runtime.plugin_sources` is not cached locally under `$CONDUCTOR_HOME/cache/plugins/`, validation succeeds with an `incomplete` status note, reports missing cache identities, and omits the digest.
1006
+
***Bundle Validation Errors:** Local file defects fail validation with exit code 1. These include dynamic Jinja includes (`BundleDynamicTemplateError`), root escapes (`BundleRootEscapeError`, `BundleSymlinkEscapeError`), resource limit violations (`BundleCapsError`), missing declared roots, or circular dependencies.
1007
+
1008
+
## `conductor bundle build`
1009
+
1010
+
Build an immutable, content-addressed run bundle for a workflow.
1011
+
1012
+
Collects the complete statically knowable file closure of the workflow, including external file references (`!file`, `!yamlfile`), Jinja2 template partials, local and registry sub-workflows, skills, plugins, and declared assets. Publishes the resulting tree and deterministic archive to the content-addressed store at `$CONDUCTOR_HOME/cache/bundles/sha256-<hex>/`. The report and manifest retain the canonical `sha256:<hex>` digest spelling; only the directory name uses a hyphen for Windows portability.
|`workflow`| Path to a local workflow YAML file or a workflow registry reference (`name`, `name@registry`, `name@registry#ref`). |
1023
+
1024
+
### Options
1025
+
1026
+
| Option | Description |
1027
+
|--------|-------------|
1028
+
|`--environment <name\|PATH>`| Execution environment: a name resolved via `.conductor/environments/` (project then user level) or a path to an environment document. Defaults to the built-in local environment. |
1029
+
1030
+
### Exit Codes
1031
+
1032
+
*`0`: Bundle built successfully (or existing cached bundle reused) and content published to the store.
1033
+
*`1`: Build failed due to schema errors, dynamic includes, root escapes, missing dependencies, or exceeded limits.
# Build directly from a workflow registry reference
1045
+
conductor bundle build 'qa-bot@official#v1.2.3'
1046
+
```
1047
+
1048
+
### Store Layout
1049
+
1050
+
Bundles are stored under `$CONDUCTOR_HOME/cache/bundles/sha256-<hex>/`:
1051
+
1052
+
*`bundle.json`: The serialized `BundleManifest` (readiness sentinel, written last).
1053
+
*`bundle.tar.gz`: The reproducible gzip archive containing all files and `.bundle/manifest.json`.
1054
+
*`tree/`: The staged directory hierarchy with POSIX-normalized paths (`tree/main/`, `tree/roots/`, etc.).
1055
+
1056
+
### CI Priming
1057
+
1058
+
`conductor bundle build` serves as the CI priming step for workflow dependencies. Unlike `conductor validate` (which runs offline), `conductor bundle build` fetches and caches any remote plugin sources declared in `runtime.plugin_sources`. After running `conductor bundle build` during a CI preparation stage, subsequent workflow runs and validations can execute in isolated, offline environments.
1059
+
999
1060
## `conductor doctor`
1000
1061
1001
1062
Report provider and environment diagnostics — a safe, read-only health check
When building content-addressed run bundles (`conductor bundle build`), Conductor stores the resulting package in a local cache directory:
1048
+
1049
+
```
1050
+
~/.conductor/cache/bundles/
1051
+
```
1052
+
1053
+
If `CONDUCTOR_HOME` is set, the cache path defaults to `$CONDUCTOR_HOME/cache/bundles/` instead.
1054
+
1055
+
### Structure
1056
+
1057
+
Each bundle is stored under a filesystem-safe encoding of its full content
1058
+
digest. The manifest keeps `sha256:<hex>`, while the directory name replaces
1059
+
the colon with a hyphen for Windows compatibility:
1060
+
1061
+
```
1062
+
$CONDUCTOR_HOME/cache/bundles/
1063
+
sha256-<hex>/
1064
+
bundle.json # Manifest sentinel certifying the staged tree
1065
+
bundle.tar.gz # Deterministic gzip archive of the bundle
1066
+
tree/ # Staged file hierarchy under normalized POSIX paths
1067
+
```
1068
+
1069
+
### Retention and Lifecycle
1070
+
1071
+
In v1, bundle cache storage grows without automated garbage collection. Stored bundles remain in the cache until removed manually or cleared by CI teardown steps. Rebuilding a workflow whose file closure has not changed reuses the existing bundle directory immediately without re-staging files.
0 commit comments