Skip to content

feat: add SCM servers support for private Git collection repositories#559

Merged
cidrblock merged 1 commit intoansible:mainfrom
cidrblock:scm_servers
Mar 25, 2026
Merged

feat: add SCM servers support for private Git collection repositories#559
cidrblock merged 1 commit intoansible:mainfrom
cidrblock:scm_servers

Conversation

@cidrblock
Copy link
Copy Markdown
Collaborator

Summary

  • Introduces ScmServer dataclass (id, hostname, token_env_var) for configuring private Git-hosted collection repositories, mirroring the existing galaxy_servers pattern
  • Replaces hardcoded ~/.git-credentials setup and AAP_EE_BUILDER_* / SCM_TOKEN env vars with a dynamic, data-driven approach — Portal passes scm_servers via --ee-config JSON
  • SCM tokens are resolved via envsubst post-processing of context/_build/requirements.yml after ansible-builder create, with multi-stage build isolation ensuring tokens never reach the final image
  • Converts README.md to a Jinja2 template with dynamic secrets table and adds a new NEXT_STEPS.md.j2 template that dynamically enumerates all required secrets (Galaxy, SCM, Registry)

Changes

Core (types.py)

  • ENV_VAR_RE regex for validating uppercase env var names
  • ScmServer frozen dataclass with from_dict(), as_dict(), to_schema()
  • EEConfig: new scm_servers field, _KNOWN_KEYS, from_dict(), to_schema()
  • TemplateData: new ee_scm_servers and ee_scm_token_vars fields

Workflow template (ee-build.yml.j2)

  • Dynamic SCM token validation checks (same pattern as galaxy tokens)
  • Removed hardcoded Configure Git credentials step (~/.git-credentials)
  • Post-create envsubst resolves tokens in context/_build/requirements.yml
  • Dynamic env vars from ee_scm_token_vars in build step

Scaffolded files

  • README.mdREADME.md.j2 with dynamic secrets table
  • New NEXT_STEPS.md.j2 with required/optional secrets checklist

Documentation

  • New "SCM Servers" section in docs/ee_scaffolding.md covering schema, token flow, and security

Tests

  • 10 new tests: ScmServer unit tests, EEConfig integration, end-to-end scaffolding with SCM servers
  • Regenerated fixtures for default scaffolding

Example CLI usage

ansible-creator init execution_env \
  --ee-config '{
    "collections": [
      {"name": "https://${GITHUB_ORG1_TOKEN}@github.com/org1/my-collection", "type": "git"},
      {"name": "cisco.ios"}
    ],
    "scm_servers": [
      {"id": "github_org1", "hostname": "github.com", "token_env_var": "GITHUB_ORG1_TOKEN"}
    ]
  }' \
  my-ee-project

Security

The multi-stage build generated by ansible-builder ensures tokens never reach the final image:

  1. ansible-builder create generates build context with unresolved ${TOKEN} references
  2. envsubst resolves tokens in the generated context/_build/requirements.yml
  3. buildah bud builds the image — resolved URLs exist only in the intermediate galaxy stage
  4. The final stage only copies installed collections (/usr/share/ansible), not source files

Depends on

Test plan

  • ScmServer.from_dict() validates id, hostname, token_env_var
  • ScmServer.from_dict() rejects invalid env var names (lowercase, dashes, leading digits)
  • EEConfig.from_dict() parses scm_servers list
  • End-to-end scaffolding generates correct workflow with envsubst post-processing
  • Generated workflow contains no git-credentials or hardcoded AAP_EE_BUILDER_* references
  • NEXT_STEPS.md is generated with correct SCM token names
  • All pre-commit hooks pass (ruff, mypy, cspell, pylint, etc.)
  • tox py, py314-devel, docs environments pass

Made with Cursor

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds first-class, data-driven support for authenticating against private Git-hosted collection repositories in the execution-environment (EE) scaffold, aligning it with the existing galaxy_servers pattern and updating the generated GitHub Actions workflow + docs accordingly.

Changes:

  • Introduces ScmServer and EEConfig.scm_servers, and threads SCM token metadata into template rendering.
  • Updates the EE build workflow template to use per-server SCM secrets and post-process context/_build/requirements.yml via envsubst (removing ~/.git-credentials handling).
  • Converts scaffold docs to be dynamic (templated README + new NEXT_STEPS.md) and updates ee_scaffolding.md + fixtures/tests.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/ansible_creator/types.py Adds ENV_VAR_RE, ScmServer, and EEConfig.scm_servers; extends schema + TemplateData.
src/ansible_creator/subcommands/init.py Passes scm_servers and derived token var names into templates.
src/ansible_creator/resources/common/ee-ci/.github/workflows/ee-build.yml.j2 Generates dynamic SCM secret checks and envsubst post-processing for requirements.
src/ansible_creator/resources/execution_env_project/README.md.j2 Templated README with dynamic secrets list (Galaxy/SCM) and file name.
src/ansible_creator/resources/execution_env_project/NEXT_STEPS.md.j2 New templated setup checklist for required/optional secrets and variables.
docs/ee_scaffolding.md Documents scm_servers, token flow, and security model.
tests/units/test_init_ee.py Adds unit + integration coverage for ScmServer/EEConfig and EE scaffold output.
tests/fixtures/common/ee-ci/.github/workflows/ee-build.yml Updates expected common workflow fixture output.
tests/fixtures/project/ee_project/.github/workflows/ee-build.yml Updates expected project workflow fixture output.
tests/fixtures/project/ee_project/README.md Updates expected scaffolded README fixture output.
tests/fixtures/project/ee_project/NEXT_STEPS.md Adds expected scaffolded NEXT_STEPS fixture output.
cspell.config.yaml Adds envsubst to ignore list.
Comments suppressed due to low confidence (1)

src/ansible_creator/resources/execution_env_project/README.md.j2:23

  • This README claims the workflow "fails fast if credentials are missing", but the generated workflow only emits ::notice:: messages and continues regardless of missing secrets. Either update the wording to reflect the current behavior, or change the workflow to actually stop the build when required secrets are absent.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Introduce ScmServer dataclass and dynamic workflow generation for
private Git-hosted collections. Tokens are resolved via envsubst
post-processing of the build context, with multi-stage build isolation
ensuring tokens never reach the final image.

- Add ScmServer dataclass with id, hostname, token_env_var fields
- Wire scm_servers through EEConfig, TemplateData, and init.py
- Replace hardcoded git-credentials step with dynamic envsubst
- Convert README.md to Jinja2 template with dynamic secrets table
- Add NEXT_STEPS.md.j2 for dynamic secrets setup checklist
- Update docs/ee_scaffolding.md with SCM servers section
- Add unit, integration, and end-to-end tests

Made-with: Cursor
@cidrblock cidrblock merged commit f9aa3e7 into ansible:main Mar 25, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

3 participants