feat: marketplace-based version management (#514)#3
Closed
sergio-sisternes-epam wants to merge 6 commits intomainfrom
Closed
feat: marketplace-based version management (#514)#3sergio-sisternes-epam wants to merge 6 commits intomainfrom
sergio-sisternes-epam wants to merge 6 commits intomainfrom
Conversation
Add marketplace version schema, semver range resolution, and version-aware install/view/outdated flows for Issue microsoft#514. - Add VersionEntry model and versions[] field to MarketplacePlugin - Create semver range resolver (^, ~, >=, exact, compound ranges) - Version-aware install: resolve marketplace versions with #specifier syntax - Version-aware view: apm view plugin@marketplace versions shows version table - Version-aware outdated: marketplace deps checked against marketplace versions - Add version_spec field to LockedDependency for lockfile persistence - 108 new tests across 5 test files, all 3909 tests passing Co-authored-by: Copilot <[email protected]>
Add 'apm marketplace publish' command that reads apm.yml defaults, resolves git HEAD SHA, validates semver, detects version conflicts, and updates marketplace.json with new version entries. Supports --dry-run, --force, and auto-detection of single marketplace. Add 'apm marketplace validate' command with validator.py module for schema validation, semver format checks, duplicate version detection, and duplicate name detection. Includes --check-refs placeholder and --verbose per-plugin details. 46 new tests (21 publish + 25 validate), 3955 total passing. Co-authored-by: Copilot <[email protected]>
…(Phase 3) Add version pin cache (~/.apm/cache/marketplace/version-pins.json) that tracks version-to-ref mappings per plugin per marketplace. Warns when a previously pinned version's ref changes (possible ref swap attack). Advisory only -- never blocks installation. Add multi-marketplace shadow detection that checks all registered marketplaces for duplicate plugin names during resolution. Warns about potential name squatting when the same plugin exists in multiple marketplaces. Add security-critical provenance comments in install.py confirming discovered_via and marketplace_plugin_name are set for all marketplace dependencies. 35 new tests (25 immutability + 10 shadow), 3990 total passing. Co-authored-by: Copilot <[email protected]>
Update marketplaces guide with versioned plugins schema, semver range install syntax, version viewing, publish/validate commands, and security hardening sections (immutability + shadow detection). Update CLI reference with marketplace publish and validate subcommands, version specifier syntax in install/view/outdated commands. Add version_spec field to lockfile spec. Update apm-guide skill resources with new marketplace commands and version specifier table. Add changelog entries for all Phase 1-3 features under [Unreleased]. Co-authored-by: Copilot <[email protected]>
- Use auth-first for marketplace fetch (private org repos return 404 unauthenticated, not 403, so unauth_first swallowed the error) - Fall back to version_spec when version field is absent in lockfile for marketplace outdated checks (strip range prefixes like ^, ~) Co-authored-by: Copilot <[email protected]>
B1: Add resolved_version field to LockedDependency and lockfile YAML.
Populated during marketplace version resolution (3-tuple return from
resolve_marketplace_plugin). Enables lockfile-based version verification.
B2: Exit code 1 when all packages fail validation (was silently returning 0).
B3: Show 'Resolved version: X.Y.Z' in verbose install output for marketplace
packages alongside the resolved ref.
B4: Route 'apm view plugin@marketplace' (no subfield) to marketplace version
display instead of failing with 'not found in apm_modules'.
B5: Show 'best in range' upgrade in outdated output when latest version is
outside the pinned range (e.g., '2.0.0 (outside ^1.0.0; best in range: 1.2.0)').
B6: Use resolved_version as primary fallback in outdated, with robust regex
extraction from version_spec for compound ranges.
B7: Route security warnings (immutability, shadow detection) through CLI
CommandLogger via warning_handler callback, visible at -v level.
Co-authored-by: Copilot <[email protected]>
There was a problem hiding this comment.
Pull request overview
Adds marketplace-based version management to APM by introducing a versions[] schema in marketplace.json, a lightweight semver range resolver, version-aware marketplace resolution, and related CLI/lockfile/docs/test updates. This enables installing/viewing/updating/publishing versioned marketplace plugins while adding advisory security warnings (immutability + shadow detection).
Changes:
- Add semver range resolution and version-aware marketplace resolver flow (plus lockfile provenance fields).
- Add security advisories (version pin cache + cross-marketplace shadow detection) and new CLI commands (
marketplace publish,marketplace validate). - Add/expand unit tests and documentation for new marketplace versioning workflows.
Reviewed changes
Copilot reviewed 30 out of 30 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/test_view_versions.py | New tests for apm view NAME@MARKETPLACE versions display behavior (sorting, latest badge, fallbacks). |
| tests/unit/test_view_command.py | Tests apm view NAME@MARKETPLACE default routing to marketplace versions. |
| tests/unit/test_version_resolver.py | New tests for semver range parsing/resolution and specifier detection heuristics. |
| tests/unit/test_outdated_marketplace.py | New tests for marketplace-aware apm outdated behavior (ranges, fallbacks, tuple shape). |
| tests/unit/marketplace/test_versioned_resolver.py | New tests for parse_marketplace_ref #version_spec parsing and version-aware resolution + warning routing. |
| tests/unit/marketplace/test_version_pins.py | New tests for version pin cache load/save/check behavior and advisory fail-open semantics. |
| tests/unit/marketplace/test_shadow_detector.py | New tests for shadow detection behavior and resolver/install integration expectations. |
| tests/unit/marketplace/test_marketplace_validator.py | New tests for marketplace validation engine and apm marketplace validate output/exit behavior. |
| tests/unit/marketplace/test_marketplace_resolver.py | Update tests for parse_marketplace_ref tuple shape change (adds version_spec). |
| tests/unit/marketplace/test_marketplace_publish.py | New tests for apm marketplace publish command and helper behaviors. |
| tests/unit/marketplace/test_marketplace_models.py | Extend model tests for new VersionEntry and MarketplacePlugin.versions parsing/backward-compat. |
| tests/unit/marketplace/test_marketplace_install_integration.py | Update marketplace parsing expectations and add install behavior regression tests. |
| src/apm_cli/marketplace/version_resolver.py | New semver range engine (caret/tilde/comparators/compound) and specifier heuristic. |
| src/apm_cli/marketplace/version_pins.py | New on-disk advisory cache for version->ref pinning with atomic writes. |
| src/apm_cli/marketplace/validator.py | New marketplace validation checks (schema, versions format, duplicates, names). |
| src/apm_cli/marketplace/shadow_detector.py | New advisory scan for same plugin name across registered marketplaces. |
| src/apm_cli/marketplace/resolver.py | Extend marketplace ref parsing to support #version_spec; version-aware resolution; warning_handler support; immutability + shadow warnings. |
| src/apm_cli/marketplace/models.py | Add VersionEntry and MarketplacePlugin.versions parsing from manifest JSON. |
| src/apm_cli/marketplace/client.py | Switch marketplace fetch to auth-first to avoid silent 404 behavior for private repos. |
| src/apm_cli/deps/lockfile.py | Add lockfile fields version_spec and resolved_version with serialization support. |
| src/apm_cli/commands/view.py | Add marketplace versions display path for apm view NAME@MARKETPLACE / versions. |
| src/apm_cli/commands/outdated.py | Add marketplace-aware update checking and a Source column in output. |
| src/apm_cli/commands/marketplace.py | Add apm marketplace validate and apm marketplace publish (plus helpers). |
| src/apm_cli/commands/install.py | Wire marketplace version_spec/resolved_version into resolution + lockfile provenance; ensure nonzero exit on all-failed validation. |
| packages/apm-guide/.apm/skills/apm-usage/dependencies.md | Document marketplace semver # specifiers in skill reference. |
| packages/apm-guide/.apm/skills/apm-usage/commands.md | Add new marketplace commands and marketplace versioning usage examples. |
| docs/src/content/docs/reference/lockfile-spec.md | Document new lockfile field version_spec. |
| docs/src/content/docs/reference/cli-commands.md | Update CLI reference for marketplace versioning, view behavior, outdated source column, and new marketplace subcommands. |
| docs/src/content/docs/guides/marketplaces.md | Add guide sections for versioned plugins, view/publish/validate, and security model. |
| CHANGELOG.md | Add Unreleased changelog entries for marketplace version management feature set. |
Comment on lines
+270
to
+275
| try: | ||
| manifest = fetch_or_cache(source) | ||
| except MarketplaceFetchError as exc: | ||
| logger.error(str(exc)) | ||
| logger.progress("Check your network connection and try again.") | ||
| sys.exit(1) |
There was a problem hiding this comment.
The marketplace fetch path does not pass an AuthResolver/token into fetch_or_cache(). This will likely break apm view NAME@MARKETPLACE for private/org-scoped marketplace repos (the same reason marketplace client switched to auth-first). Consider constructing an AuthResolver here and forwarding it to fetch_or_cache (or plumb an auth_resolver parameter through display_versions/view).
Comment on lines
+334
to
+340
| console.print(table) | ||
| click.echo("") | ||
| click.echo(f" Install: apm install {plugin_name}@{marketplace_name}") | ||
| click.echo( | ||
| f" Pin: apm install {plugin_name}@{marketplace_name}" | ||
| f"#^{sorted_versions[0].version}" | ||
| ) |
There was a problem hiding this comment.
The pin hint always uses #^{sorted_versions[0].version}. If a plugin's versions list contains only non-semver entries (or if all semver parsing fails), this will emit an invalid spec like #^nightly. Consider only showing the #^... hint when there is at least one valid semver entry; otherwise show a raw ref pin hint (e.g., #<ref>) or omit the pin line.
Comment on lines
+256
to
+261
| from ..marketplace.errors import MarketplaceFetchError, PluginNotFoundError | ||
| from ..marketplace.models import MarketplaceSource | ||
| from ..marketplace.registry import get_marketplace_by_name | ||
| from ..marketplace.client import fetch_or_cache | ||
| from ..marketplace.version_resolver import _parse_semver, _SEMVER_RE | ||
|
|
There was a problem hiding this comment.
This function imports and relies on private helpers from version_resolver (_parse_semver, _SEMVER_RE). Since these are underscored, other modules depending on them makes refactors risky. Consider either (a) exposing a small public semver parsing/sorting helper in version_resolver, or (b) keeping the parsing logic local in view.py.
Suggested change
| from ..marketplace.errors import MarketplaceFetchError, PluginNotFoundError | |
| from ..marketplace.models import MarketplaceSource | |
| from ..marketplace.registry import get_marketplace_by_name | |
| from ..marketplace.client import fetch_or_cache | |
| from ..marketplace.version_resolver import _parse_semver, _SEMVER_RE | |
| import re | |
| from ..marketplace.errors import MarketplaceFetchError, PluginNotFoundError | |
| from ..marketplace.models import MarketplaceSource | |
| from ..marketplace.registry import get_marketplace_by_name | |
| from ..marketplace.client import fetch_or_cache | |
| semver_re = re.compile( | |
| r"^v?(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)" | |
| r"(?:-([0-9A-Za-z.-]+))?$" | |
| ) | |
| def parse_semver(version: str) -> Optional[tuple[int, int, int, int, str]]: | |
| match = semver_re.match(version.strip()) | |
| if not match: | |
| return None | |
| major, minor, patch, prerelease = match.groups() | |
| prerelease_rank = 0 if prerelease is None else -1 | |
| return ( | |
| int(major), | |
| int(minor), | |
| int(patch), | |
| prerelease_rank, | |
| prerelease or "", | |
| ) | |
| _SEMVER_RE = semver_re | |
| _parse_semver = parse_semver |
Comment on lines
+68
to
+76
| path = _pins_path(pins_dir) | ||
| if not os.path.exists(path): | ||
| return {} | ||
| try: | ||
| with open(path, "r") as fh: | ||
| data = json.load(fh) | ||
| if not isinstance(data, dict): | ||
| logger.debug("version-pins file is not a JSON object; ignoring") | ||
| return {} |
There was a problem hiding this comment.
load_version_pins() opens JSON without an explicit encoding. To avoid platform-dependent defaults (and to match how marketplace.json is handled elsewhere), open the pins file with encoding='utf-8' (and consider errors='replace' if you want to stay fully fail-open).
Comment on lines
+90
to
+98
| path = _pins_path(pins_dir) | ||
| tmp_path = path + ".tmp" | ||
| try: | ||
| os.makedirs(os.path.dirname(path), exist_ok=True) | ||
| with open(tmp_path, "w") as fh: | ||
| json.dump(pins, fh, indent=2) | ||
| os.replace(tmp_path, path) | ||
| except OSError as exc: | ||
| logger.debug("Failed to save version-pins: %s", exc) |
There was a problem hiding this comment.
save_version_pins() writes JSON without specifying encoding. Use encoding='utf-8' for deterministic cross-platform behavior. Also consider cleaning up the .tmp file on failure so a leftover temp file does not accumulate if os.replace fails.
| ) | ||
| return ValidationResult( | ||
| check_name="Versions", | ||
| passed=len(errors) == 0 and len(warnings) == 0, |
There was a problem hiding this comment.
validate_version_format() marks the check as passed=False when there are warnings (invalid semver), but the CLI currently treats warnings as non-fatal. This makes the meaning of passed ambiguous and can lead to inconsistent exit codes. Consider defining passed based on errors only (and keeping warnings informational), or updating the CLI to fail when passed is False.
Suggested change
| passed=len(errors) == 0 and len(warnings) == 0, | |
| passed=len(errors) == 0, |
Comment on lines
+714
to
+721
| versions = target.get("versions", []) | ||
|
|
||
| # Remove existing version entry when --force is active | ||
| if force: | ||
| versions = [v for v in versions if v.get("version") != version_str] | ||
|
|
||
| versions.append({"version": version_str, "ref": ref}) | ||
| target["versions"] = versions |
There was a problem hiding this comment.
_update_marketplace_file assumes versions is a list; if marketplace.json contains versions in an unexpected type (string/dict), append() will raise and the publish command will crash. Consider normalizing non-list values to an empty list (or raising a clearer ValueError) before mutating.
| | `is_dev` | boolean | MAY | `true` if the dependency was resolved through [`devDependencies`](../manifest-schema/#5-devdependencies). Omitted when `false`. Dev deps are excluded from `apm pack --format plugin` bundles. | | ||
| | `deployed_files` | array of strings | MUST | Every file path APM deployed for this dependency, relative to project root. | | ||
| | `source` | string | MAY | Dependency source. `"local"` for local path dependencies. Omitted for remote (git) dependencies. | | ||
| | `version_spec` | string | MAY | Original semver range from the install specifier (e.g., `"^2.0.0"`). Present only for marketplace dependencies installed with a version constraint. Used by `apm outdated` to evaluate updates within the pinned range. | |
There was a problem hiding this comment.
The lockfile now includes both version_spec and resolved_version for marketplace deps, but the spec table only documents version_spec. Consider documenting resolved_version as well (what it contains and when it is present) so readers can interpret marketplace entries correctly.
Suggested change
| | `version_spec` | string | MAY | Original semver range from the install specifier (e.g., `"^2.0.0"`). Present only for marketplace dependencies installed with a version constraint. Used by `apm outdated` to evaluate updates within the pinned range. | | |
| | `version_spec` | string | MAY | Original semver range from the install specifier (e.g., `"^2.0.0"`). Present only for marketplace dependencies installed with a version constraint. Used by `apm outdated` to evaluate updates within the pinned range. | | |
| | `resolved_version` | string | MAY | Exact marketplace version selected during resolution (e.g., `"2.3.1"`). Present for marketplace dependencies when APM resolved and pinned a concrete published version. | |
Comment on lines
+165
to
+167
| The `#` separator carries a version specifier when the plugin declares `versions`, or a raw git ref when it does not. Plugins without `versions` continue to work as before. | ||
|
|
||
| APM resolves the plugin name against the marketplace index, fetches the underlying Git repository at the resolved ref, and installs it as a standard APM dependency. The resolved source appears in `apm.yml` and `apm.lock.yaml` just like any direct dependency. |
There was a problem hiding this comment.
Docs state that the # suffix is treated as a raw git ref when a plugin does not declare versions. However, resolve_marketplace_plugin currently ignores version_spec entirely for unversioned plugins (tests also assert this). Please either update the docs to reflect the current behavior, or implement raw-ref override for unversioned plugins so the documented syntax works.
Suggested change
| The `#` separator carries a version specifier when the plugin declares `versions`, or a raw git ref when it does not. Plugins without `versions` continue to work as before. | |
| APM resolves the plugin name against the marketplace index, fetches the underlying Git repository at the resolved ref, and installs it as a standard APM dependency. The resolved source appears in `apm.yml` and `apm.lock.yaml` just like any direct dependency. | |
| The `#` separator carries a version specifier only when the plugin declares `versions`. For plugins without `versions`, APM ignores the suffix and installs the plugin from the marketplace entry as defined. | |
| APM resolves the plugin name against the marketplace index, selects the matching declared version when available, and installs the underlying source as a standard APM dependency. The resolved source appears in `apm.yml` and `apm.lock.yaml` just like any direct dependency. |
| @@ -702,7 +718,7 @@ def install(ctx, packages, runtime, exclude, only, update, dry_run, force, verbo | |||
| ) | |||
| # Short-circuit: all packages failed validation — nothing to install | |||
There was a problem hiding this comment.
This comment line contains a non-ASCII em dash (U+2014), but the project requires source files to stay within printable ASCII for Windows terminal compatibility. Replace it with a plain '-' or '--' in this block.
Suggested change
| # Short-circuit: all packages failed validation — nothing to install | |
| # Short-circuit: all packages failed validation -- nothing to install |
sergio-sisternes-epam
pushed a commit
that referenced
this pull request
Apr 22, 2026
* feat(policy): W1 foundations for install-time policy enforcement (microsoft#827) Wave 1 of issue microsoft#827 implementation. Lays the foundations the install pipeline gate (W2) will plug into. No behaviour change yet — install still does NOT enforce policy until W2 wires the gate phase. What's in: - policy_checks: new public seam run_dependency_policy_checks(deps, lockfile=, policy=, mcp_deps=, effective_target=) accepting a resolved dep set; old run_policy_checks(project_root, policy) is now a thin wrapper. Honours require_resolution: project-wins for version-pin mismatches only. Latent isinstance(allow, list) bug fixed for schema's Tuple[str, ...]. - policy/discovery: cache stores merged effective policy with chain metadata + fingerprint. Atomic writes via temp + os.replace, with pid+thread_id suffix to prevent concurrent-writer collision. MAX_STALE_TTL=7d ceiling on cache reuse. PolicyFetchResult expanded to express 9 outcomes (found, absent, cached_stale, cache_miss_fetch_fail, malformed, disabled, garbage_response, no_git_remote, empty). - diagnostics: CATEGORY_POLICY constant + per-category renderer wired into render_summary(). - command_logger: InstallLogger.policy_resolved/violation/disabled with per-class actionable error wording (auth/unreachable/malformed/ blocked). - tests/fixtures/policy/: 14 policy fixtures + 7 project fixtures (denied-direct, denied-transitive, required-missing, required-version-mismatch, mcp-denied, target-mismatch, unpacked-bundle) covering W4 live matrix scenarios L2/L4/L13 and rubber-duck findings I5/I6/I7/N14/C2. - docs: 12-section Install-time enforcement guide skeleton in both enterprise/policy-reference.md and packages/apm-guide skill mirror. 10 sections filled; sections 7 (snippets) and 10 (error table) stubbed for W3-docs-final once W2 lands and W4 captures live output. Tests: - tests/unit: 4878 passed (1 pre-existing unrelated MCP failure deselected). Includes 41 logger + 29 policy-seam + 38 cache + 21 fixture-load new tests. Refs: microsoft#827 Co-authored-by: Copilot <[email protected]> * feat(install): W2A policy enforcement at install time (microsoft#827) Wave 2A wires the three install-time enforcement sites planned for microsoft#827: 1. **Pipeline gate phase** (src/apm_cli/install/phases/policy_gate.py): New phase running between resolve and targets. Discovers org policy, resolves the inheritance chain via resolve_policy_chain, persists the merged effective policy + chain refs to cache (chain_refs threading per C1 amendment), then calls run_dependency_policy_checks against the resolved deps. Routes 9 discovery outcomes (found, absent, cached_stale, cache_miss_fetch_fail, malformed, disabled, garbage_response, no_git_remote, empty). Block-mode violations raise PolicyViolationError to halt the pipeline cleanly. 2. **--mcp branch preflight** (src/apm_cli/policy/install_preflight.py + commands/install.py:1091-1125): apm install --mcp does NOT enter the install pipeline. New shared helper run_policy_preflight() runs discovery + dep checks for any non-pipeline command site. Wired into --mcp BEFORE _run_mcp_install so denied servers never reach the integrator. Also exports PolicyBlockError for callers. 3. **install <pkg> snapshot+rollback** (commands/install.py): apm install <pkg> mutates apm.yml BEFORE the pipeline runs. We now snapshot apm.yml as raw bytes (not parsed YAML, to avoid round-trip drift on whitespace / key-order / comments), and on ANY pipeline failure (policy block, download error, etc.) restore byte-for-byte via tempfile + os.replace atomic write. Logs '[i] apm.yml restored to its previous state.' and exits non-zero. InstallContext gains policy_fetch, policy_enforcement_active, no_policy. Tests: +68 new tests, 4946 unit tests pass total. - test_policy_gate_phase.py: 27 (covers all 9 outcomes) - test_mcp_preflight_policy.py: 22 (escape hatches, allow/deny, transport, self-defined, trust_transitive, discovery outcomes, return shape) - test_install_pkg_policy_rollback.py: 19 (byte-equal restore, comments preserved, --no-policy bypass, download error rollback, snapshot unit tests) W2B (dry-run, target-aware, escape-hatch CLI flag) and C2 panel review follow. Refs: microsoft#827 Co-authored-by: Copilot <[email protected]> * feat(policy): W2B install enforcement - escape hatch, dry-run preview, target-aware check (microsoft#827) W2B completes the enforcement surface: * policy_target_check.py - new pipeline phase after targets that re-runs target/compilation checks with the resolved effective_target. Filters to TARGET_CHECK_IDS only to avoid double-emitting dep violations from the gate phase. Honors CLI --target override (I6 fix scenario). * --no-policy escape hatch on apm install / install <pkg> / install --mcp / update. APM_POLICY_DISABLE=1 env var equivalent. Both route through ctx.no_policy and emit always-visible warnings via InstallLogger.policy_disabled() noting that apm audit --ci still fails. * --dry-run policy preview. run_policy_preflight gains dry_run=True kwarg. Emits '[!] Would be blocked by policy: <dep> -- <reason>' (block) or '[!] Policy warning: <dep> -- <reason>' (warn) before the would-install table. Never raises, never mutates. Direct manifest deps only (resolver doesn't run in dry-run; documented limitation). InstallRequest, InstallService, InstallContext threaded with no_policy. LOC budget on install.py raised 1625 -> 1650 with documented rationale. Tests: 5003 unit pass (+57 W2B: 17 target_check + 24 no_policy_flag + 16 dry_run_policy). Full suite green vs main baseline. Co-authored-by: Copilot <[email protected]> * fix(policy): C2 panel fixes - transitive MCP enforcement, shared chain discovery, dry-run cap, drop apm update --no-policy (microsoft#827) C2 panel checkpoint surfaced 4 fixes (S1+B1+D2 BLOCKER/PASS-WITH-CONCERN, D1 DevX). All landed; full suite 5032 pass. S1 (Supply Chain BLOCKER) - transitive MCP enforcement: Transitive MCP servers from APM packages were bypassing install-time policy. The pipeline gate phase only sees direct apm.yml deps; transitive MCP servers are merged later via MCPIntegrator.collect_transitive() and written to runtime configs (.copilot/mcp.json, .cursor/mcp.json) with no policy check. This defeated microsoft#827 on the most security-critical dep category. Fix: second run_policy_preflight() call in commands/install.py after the transitive merge, before MCPIntegrator.install(). On block: abort MCP config writes, exit non-zero. APM packages remain installed (gate phase approved them). 15 new unit tests in test_transitive_mcp_policy.py. B1 (Architect, partial) - shared chain-aware discovery: Extract discover_policy_with_chain() into policy/discovery.py so both policy_gate.py and install_preflight.py walk the same inheritance chain. Closes the gap where --mcp / --dry-run paths could resolve a different effective policy than the pipeline path. Gate-phase keeps its 9-outcome routing; only the discovery seam moved. 10 new tests in test_chain_discovery_shared.py. D2 (DevX UX) - dry-run noise cap: install_preflight._DRY_RUN_PREVIEW_LIMIT = 5. Long deny lists now show 5 lines per severity bucket + tail '[!] ... and N more would be blocked by policy. Run apm audit for full report.' 4 new tests. D1 (DevX UX) - drop apm update --no-policy: apm update is the CLI self-updater (refreshes the apm binary), not a dependency refresh. The flag was accepted but unused. Removed the option and flipped the test to assert the flag is now rejected. LOC budget on install.py raised 1650 -> 1675 with documented justification. Tests: 5032 unit pass (+29 new: 15 transitive_mcp + 10 chain_discovery_shared + 4 dry_run_noise_cap). 1 pre-existing MCP test deselected. Co-authored-by: Copilot <[email protected]> * docs+test(policy): W3 - integration matrix, docs final fill, CHANGELOG, growth (microsoft#827) W3 phase complete. All 5 parallel workstreams landed. Tests: tests/integration/test_policy_install_e2e.py - 17 e2e scenarios I1..I17 Covers all 9 PolicyFetchResult outcomes + all 6 violation classes via CliRunner-driven full-pipeline flows. Mocks discover_policy_with_chain at both seams (policy_gate + install_preflight). Uses _build_policy() helper for frozen-dataclass safe construction. Docs: docs/src/content/docs/enterprise/policy-reference.md sec 7: 8 verbatim CLI snippets (success, block, warn, --no-policy, APM_POLICY_DISABLE, --dry-run with overflow tail, install <pkg> rollback, transitive MCP block) sec 10: outcome table (9 fetch outcomes) + violation table (6 classes) Added explicit JSON/SARIF non-goal callout (C1 amendment). packages/apm-guide/.apm/skills/apm-usage/governance.md Same content, leaner skill version, links back to docs for full text. CHANGELOG.md: Added: --no-policy / APM_POLICY_DISABLE escape hatch, --dry-run preview, install <pkg> rollback Changed: pipeline gains policy_gate + policy_target_check phases, shared chain discovery + atomic cache + MAX_STALE_TTL Security (headline): apm install enforces apm-policy.yml; transitive MCP checked before runtime config write Follow-up issue microsoft#829 filed: policy.fetch_failure: warn|block schema knob. Tests: 5049 pass (5032 unit + 17 integration). 1 pre-existing MCP test deselected. PR body drafted at session-state/files/pr-body-827.md. Growth strategy entry + asciinema script staged in WIP (gitignored). Co-authored-by: Copilot <[email protected]> * fix(policy): C3 fixes - direct MCP enforcement, malformed posture, warn-mode coverage, doc drift (microsoft#827) C3 final panel + rubber-duck found 5 issues. All fixed. #1 (CRITICAL) - Direct MCP deps in apm.yml bypassed enforcement: ctx.direct_mcp_deps now populated in pipeline.py from apm_package.get_mcp_dependencies() before policy_gate runs. policy_gate reads direct_mcp_deps (not the dead mcp_deps_to_install) and passes them to run_dependency_policy_checks. install.py:1496 second preflight guard drops 'and transitive_mcp' so direct-only MCP installs are also caught. #2 (CRITICAL) - Malformed policy handling inconsistent + broke rollback: policy_gate.py replaced sys.exit(1) on malformed with fail-open warn (matches install_preflight + cache_miss_fetch_fail/garbage_response posture). sys.exit was bypassing the rollback handler in install.py for apm install <pkg>. CEO mandate: malformed = warn, fail-closed knob is follow-up microsoft#829. microsoft#4 (IMPORTANT) - Warn-mode dropped violations: policy_gate now passes fail_fast=(enforcement=='block') so warn mode collects ALL violations, not just the first. Also emits warnings for passed=True checks with non-empty details (project-wins version-pin mismatches were silently dropped). #3 (IMPORTANT) - Chain inheritance is 1-level, not multi-level: discover_policy_with_chain only walks one parent. Toned down docs in policy-reference.md and governance.md with explicit caution callout. Filed follow-up microsoft#831 for proper recursive walk + cycle detection. microsoft#5 (BLOCKER per panel) - Doc drift on apm update --no-policy: apm update is the CLI self-updater (refreshes the apm binary), not a dep refresh. Removed all mentions from both docs. apm deps update is the dep-refresh surface (runs install pipeline, gate applies); --no-policy is NOT exposed there today. Tests: 5059 pass (5049 baseline + 10 new: 6 unit gate + 4 integration I18/I19/I20). New integration tests cover real direct-MCP block, real malformed fail-open, warn-mode multi-violation. I16 class renamed to TestI16GarbageResponsePolicy to fix mislabeling. Follow-ups: microsoft#829 (fetch_failure schema knob), microsoft#831 (multi-level chain). Co-authored-by: Copilot <[email protected]> * fix(policy): in-PR resolution of microsoft#834 (warn-mode rendering) and microsoft#831 (recursive extends chain) (microsoft#827) Originally filed as follow-ups during C3, moved in-PR per reviewer request so microsoft#832 ships a complete enforcement story. microsoft#834 - Warn-mode policy violations did not render in the install summary. Root cause: pipeline created a fresh DiagnosticCollector for install_result.diagnostics while InstallLogger.policy_violation() pushed warnings into logger.diagnostics. Two collectors, one rendered. Fix: when a logger is present, reuse logger.diagnostics so policy records flow through render_summary() (block mode unaffected - it aborts inline before summary). microsoft#831 - extends: chain only supported one level (parent). Inheritance machinery (resolve_policy_chain, detect_cycle, MAX_CHAIN_DEPTH=5) was already N-deep capable; discovery never wired it. Fix: rewrite _resolve_and_persist_chain as iterative depth-first walk, leaf-first; cycle detection via inheritance.detect_cycle; honor MAX_CHAIN_DEPTH=5 with explicit pre-append check; partial-chain warning when a mid-chain ref fails to fetch ('Policy chain incomplete: <ref> unreachable, using <N> of <M> policies'); single cache write at leaf with full chain fingerprint. Tests: +1 unit (warn-render), +5 unit (3-level full, cycle, depth limit, partial chain, single-level regression), +1 integration (TestI21ThreeLevelExtendsChain). 5044 unit pass. Docs: enterprise/policy-reference.md and apm-usage/governance.md chain-depth callouts updated. Co-authored-by: Copilot <[email protected]> * docs(changelog): record in-PR resolution of microsoft#834 and microsoft#831 under microsoft#827 Co-authored-by: Copilot <[email protected]> * fix(policy): address review-panel pre-merge findings (microsoft#827) - Security F1 (HIGH): pin extends: chain to leaf policy host; disable HTTP redirects in _fetch_from_url and _fetch_github_contents. Closes cross-host credential leak vector via git credential fill fallback and SSRF/Referer-leak vector via 30x redirects. raw.githubusercontent .com is treated as distinct from github.com (strict pin). - Logging C1+C2 + UX F1/F2/F4/F5/F9: extract InstallLogger.policy_ discovery_miss() canonical helper covering all 7 discovery outcomes; route both policy_gate and install_preflight through it. absent now verbose-only; no_git_remote downgraded to [i]; garbage_response gets distinct wording (no VPN/firewall noise); cached_stale and cache_ miss_fetch_fail messages now state enforcement posture explicitly; violation messages dedupe dep_ref prefix; wire _policy_reason_blocked into block-severity policy_violation as dim secondary line. - Docs: remove [Planned] banner from policy-reference; update enforcement tables (policy-reference + governance skill) to reflect install-time blocking; document --no-policy / APM_POLICY_DISABLE in cli-commands.md with deps-update asymmetry callout; add discovery-vs- extends clarifying note; add CHANGELOG migration note under microsoft#827. Tests: 5053 -> 5068 (+15 logging, +9 security host-pin). Co-authored-by: Copilot <[email protected]> * feat(policy): ship enterprise hardening pack on top of microsoft#827 Four enterprise hardening items shipped in-PR per CISO-arbitrated panel verdict + CTO threat-model deep dive (PR microsoft#832 comments 4294087760 + 4294115069). Closes microsoft#829. 1. policy.fetch_failure: warn|block schema knob (microsoft#829) -- org admins opt into fail-closed on fetch failure / malformed / garbage_response. Default 'warn' preserves backwards compat. 2. apm.yml policy.fetch_failure_default: warn|block -- project-side complement so a project can lock down behavior even when no policy is reachable to read the org-side knob from. 3. apm policy status diagnostic command -- show discovery outcome, source, enforcement, cache age, extends chain, effective rule counts, and hash-pin state. --json for SIEM ingestion. Trust-but- verify tool that makes fail-open acceptable. 4. apm.yml policy.hash: 'sha256:...' consumer-side pin -- closes the garbage_response compromised-intermediary vector by verifying raw policy bytes against a project-pinned digest. Equivalent of pip --require-hashes for the policy itself. ALWAYS fail-closed on mismatch, regardless of fetch_failure setting (a hash mismatch is an explicit pin violation, not a fetch failure). sha384/sha512 accepted; md5/sha1 rejected (collision-resistant only). 5. apm audit --ci auto-discovers org policy when --policy-source is not provided; --no-policy flag added to skip. Closes the audit/install asymmetry that left CI blind to sideloaded primitives. Tests: 5068 -> 5157 (+89: hash pin 31, fetch_failure knob, audit auto-discovery, policy status command, plus updates to existing discovery tests for the new expected_hash kwarg threading). Docs: policy-reference §9.5 (fetch_failure), §9.6 (hash pin), §9.7 (apm policy status), §9.8 (audit auto-discovery); governance.md skill mirrors all of the above; cli-commands.md gets policy status + audit --no-policy. CHANGELOG entries under [Unreleased] Added / Added (Security). Co-authored-by: Copilot <[email protected]> * docs(policy): address doc-writer review BLOCKERs (microsoft#827) - policy-reference.md: remove stale 'planned fetch_failure knob' paragraph that contradicted the §9.5 entry shipped in the same PR; add Linux hash-compute one-liner alongside the macOS shasum example. - cli-commands.md: add 'apm policy status' command section under a new 'apm policy' family (synopsis, --policy-source/--no-cache/--json, exit-code note, examples). Add --no-policy flag to 'apm audit' options list. Reword --policy SOURCE description to reflect that --ci now auto-discovers when --policy is omitted. Update audit examples to match (drop the now-redundant '--policy org' from auto-discovery example, add explicit --no-policy variant). Co-authored-by: Copilot <[email protected]> * docs(policy): address doc-writer HIGH+LOW findings (microsoft#827) - manifest-schema.md: add policy: block to schema diagram + new section 3.9 documenting fetch_failure_default, hash, hash_algorithm - policy-reference.md: add fetch_failure: warn to canonical schema YAML and a fetch_failure entry under Top-level fields; lift apm policy status and apm audit --ci auto-discovery into proper numbered subsections (9.7 / 9.8) so anchors match the skill mirror - governance.md: surface install-time enforcement with link to policy-reference#install-time-enforcement - ci-policy-setup.md: annotate Step 3 noting apm audit --ci auto-discovers and --policy org is now an explicit override - security.md: add Compromised policy intermediary row to attack surface comparison, linked to policy.hash consumer-side pin - cli-commands.md: split --no-policy into 2-line nested bullet separating behaviour from env-var equivalence - apm-guide skill mirror: add fetch_failure: warn to schema overview to keep skill aligned with policy-reference Co-authored-by: Copilot <[email protected]> * fix(policy): address PR review panel logging+arch findings (microsoft#827) BLOCKING: - command_logger.policy_discovery_miss: gate no_git_remote info message on verbose mode; previously emitted on every install in a non-git directory Architecture: - New install/errors.py with canonical PolicyViolationError; PolicyBlockError kept as re-exported alias to preserve test patches - New policy/outcome_routing.py::route_discovery_outcome consolidating the 9-outcome routing table; policy_gate.py and install_preflight.py now delegate instead of duplicating - pipeline.py: catch PolicyViolationError before bare Exception so policy block messages are not double-nested in RuntimeError - commands/install.py: isinstance(PolicyViolationError) branch in the legacy handler for the same reason Logging UX: - install_preflight: empty check.details now falls back to [check.name] so the block message is never blank - _extract_dep_ref helper replaces detail.split(":")[0] with defensive parsing that falls back to check.name Security: - discovery._get_cache_dir asserts containment vs project_root (resolves symlinks) instead of an unguarded join - Removed dead no_policy= kwarg from discover_policy_with_chain; env-var defence-in-depth retained on the call site Tests: +tests/unit/policy/test_pr_832_findings.py covering all 8 findings; install_logger split into silent/verbose cases. 5176 unit tests pass, 0 regressions. Co-authored-by: Copilot <[email protected]> * test(policy): use urllib.parse for host assertions to silence CodeQL (microsoft#827) CodeQL's py/incomplete-url-substring-sanitization rule fired 6 times on test_extends_host_pin.py because bare 'host' in msg substring checks could in theory match a host appearing at an arbitrary URL position (path, query, userinfo). The assertions are correct in practice -- they assert on production error messages of known format -- but the pattern is not safe in general. Replace each substring check with a precise extractor: - _assert_extends_host_in_message / _assert_leaf_host_in_message: regex-anchor on the production 'extends host: <h>' / 'leaf host: <h>' tokens, then exact-compare the captured group. - _assert_redirect_target_host: regex-extract the redirect target URL after 'to ', then urllib.parse.urlparse(...).hostname compare. No production-code changes; all 9 host-pin tests still pass. Co-authored-by: Copilot <[email protected]> * fix(policy,audit): address PR microsoft#832 DevX UX blockers - audit --no-policy help text rewritten to describe positive behaviour first ("Skip org policy discovery and enforcement" instead of the negative "Skip auto-discovery ... in --ci mode"), so apm audit --help no longer hides the primary effect behind a caveat. Aligns the code with the docs. - apm policy status --check flag added: exits 1 when outcome is not 'found' (i.e. policy unresolvable / absent / disabled / fetch-failed), 0 otherwise. Default behaviour unchanged (always exit 0) so the diagnostic remains safe for human and SIEM use, while CI authors get the npm audit / pip check style contract via a single flag. Updates cli-commands.md, policy-reference.md, and CHANGELOG.md to document the new flag and exit-code table. Adds TestStatusCheckFlag covering the found / unresolvable / discovery-exception / json combinations. Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: Copilot <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds marketplace-based version management for monorepo packages, allowing per-package semver versioning through
versions[]arrays inmarketplace.json. This solves the monorepo versioning problem (repo-wide tags vs. tag pollution vs. poly-repos) by leveraging the marketplace as a version registry.Users can now install specific versions (
apm install plugin@marketplace#^2.0.0), view available versions (apm view plugin@marketplace), check for updates with range awareness (apm outdated), and publish new versions (apm marketplace publish). Security hardening includes advisory immutability warnings when version refs change and multi-marketplace shadow detection.Fully backward compatible: plugins without
versions[]use the existing single-ref flow. Direct git dependencies are completely unaffected.Fixes microsoft#514
Type of change
Changes
Phase 1: Marketplace Version Schema + Semver Resolution
marketplace/models.py:VersionEntrydataclass,versionsfield onMarketplacePluginmarketplace/version_resolver.py(new): Lightweight semver engine supporting^,~,>=,>,<,<=,!=, exact, and compound ranges. No external dependencies.marketplace/resolver.py: Version-awareresolve_marketplace_plugin()returning 3-tuple(canonical, plugin, resolved_version). Falls back to single-ref flow when noversions[]present.commands/install.py: Marketplace intercept with version spec parsing,resolved_versionprovenance in lockfilecommands/view.py:_display_marketplace_versions()— Rich table showing all published versions with "latest" tagcommands/outdated.py:_check_marketplace_versions()— range-aware update checking with "(outside range)" annotationsdeps/lockfile.py: New fields:version_spec,resolved_version,discovered_via,marketplace_plugin_namePhase 2: Publishing Tooling
commands/marketplace.py:apm marketplace publish— publishes current package version to marketplace.json with SHA-pinned refs.apm marketplace validate— validates marketplace integrity (4 checks: refs, duplicates, semver format, source URLs).marketplace/validator.py(new): Marketplace validation enginePhase 3: Security Hardening
marketplace/version_pins.py(new): Advisory immutability — caches version-to-ref mappings, warns on ref changes (potential ref-swap attacks)marketplace/shadow_detector.py(new): Multi-marketplace shadow detection — warns when the same plugin name appears in multiple registered marketplacesCommandLoggerviawarning_handlercallback (visible at-v)Documentation
marketplace publishandmarketplace validateBackward Compatibility
versions[]versions[]version_specTesting
Unit Tests
4017 tests passing (+227 new tests across 12 test files covering version resolver, models, publish, validate, shadow detector, version pins, versioned resolver, outdated marketplace, view versions, and install integration).
Integration Tests (50 tests against a live marketplace)
End-to-end tested against a private marketplace with 68 versioned plugins (including multi-version packages with 5-6 versions each):
^,~,>=, exact, compound, error)