Skip to content

Add testing philosophy documentation#4504

Merged
cidrblock merged 30 commits intoansible:mainfrom
cidrblock:testing_philosophy
Aug 15, 2025
Merged

Add testing philosophy documentation#4504
cidrblock merged 30 commits intoansible:mainfrom
cidrblock:testing_philosophy

Conversation

@cidrblock
Copy link
Copy Markdown

@cidrblock cidrblock commented Aug 7, 2025

Depends on #4505

Summary

This pull request adds comprehensive testing philosophy documentation to help users understand Molecule's approach to Ansible automation testing and its strategic direction.

Changes Made

New Documentation: docs/philosophy.md

  • Testing suite fundamentals: Universal testing framework capabilities and core phases applicable to any robust testing system
  • Molecule's Ansible-centric approach: How Molecule addresses testing requirements specifically for Ansible automation across all domains (infrastructure, applications, network, cloud, APIs, business processes)
  • Resource lifecycle management: Detailed explanation of shared state and shared inventory mechanisms for cost optimization and workflow efficiency
  • Configuration flexibility: Ansible as unified driver, provisioner, and verifier with enterprise customization capabilities
  • Evolution and strategic direction: Molecule's focused development on deeper Ansible integration rather than broader tool support

Key Technical Improvements

  • Corrected shared state implementation: Accurate documentation of automatic default scenario management for create/destroy operations
  • Shared inventory mechanism: Clear explanation of how centralized inventory enables resource sharing across scenarios
  • Enterprise customization: Documentation of how sequence configurations can be committed and shared between teams
  • Framework evolution: Nine-year optimization history and future roadmap focusing on Ansible ecosystem alignment

Documentation Quality

  • Professional technical writing suitable for seasoned Ansible practitioners
  • Framework-agnostic core concepts with Ansible-specific implementation details
  • Comprehensive workflow examples and real-world usage patterns
  • Accurate technical details verified through codebase analysis

Target Audience

This documentation serves automation practitioners, enterprise teams, and decision makers who need to understand Molecule's testing philosophy, capabilities, and strategic direction within the Ansible ecosystem.

Testing

  • Documentation builds successfully with ROOT: will run in automatically provisioned tox, host /home/bthornto/github/molecule311/.venv/bin/python3 is missing [requires (has)]: tox-uv>=1.20.2
    ROOT: provision> .tox/.tox/bin/python -m tox -e docs
    .pkg: _optional_hooks> python /home/bthornto/github/molecule311/.tox/.tox/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta
    .pkg: get_requires_for_build_editable> python /home/bthornto/github/molecule311/.tox/.tox/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta
    .pkg: build_editable> python /home/bthornto/github/molecule311/.tox/.tox/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta
    docs: install_package> .tox/.tox/bin/uv pip install --reinstall --no-deps molecule@/home/bthornto/github/molecule311/.tox/.tmp/package/49/molecule-25.7.1.dev20-0.editable-py3-none-any.whl
    docs: commands_pre[0]> sh -c 'rm -f /home/bthornto/github/molecule311/.tox/docs/.coverage* 2>/dev/null || true'
    docs: commands[0]> mkdocs build --strict --site-dir=_readthedocs/html/
    docs: OK (8.61=setup[1.41]+cmd[0.01,7.19] seconds)
    congratulations :) (8.71 seconds)
  • All technical details verified through codebase review
  • Examples tested for accuracy and clarity

- Add comprehensive philosophy documentation covering testing suite fundamentals
- Explain essential functionality, configuration flexibility, and lifecycle management
- Document how Molecule addresses testing suite requirements for Ansible automation
- Cover sequence configuration, shared state/inventory, and extensibility design
- Position philosophy as foundational understanding before installation guide

This documentation provides the conceptual foundation for understanding
Molecule's design decisions and architectural approach to Ansible testing.
- Broaden scope from infrastructure-only to all Ansible use cases
- Include application configuration, deployment, API integration, business processes
- Update testing requirements to cover infrastructure, apps, services, and integrations
- Expand Molecule capabilities explanation for multi-domain automation
- Cover diverse platforms: infrastructure, applications, network devices, cloud services, APIs
- Address variable management across all credential and configuration types
- Update examples to reflect broader automation scenarios and use cases

This ensures the philosophy documentation accurately represents Ansible's
full automation capabilities and how Molecule supports testing across
all domains, not just infrastructure provisioning.
- Add testing workflow fundamentals section covering all testing phases
- Document core phases: provisioning, dependency resolution, convergence,
  idempotence, verification, side effects, cleanup, and destruction
- Map testing phases to Molecule actions with detailed purpose explanations
- Provide comprehensive command-line subcommand reference with usage scenarios
- Add configurable sequence examples for different testing strategies
- Enhance shared state documentation with cost and speed optimization benefits
- Contrast shared state vs. isolation scenarios with specific recommendations
- Include practical workflow examples for both shared and isolated testing
- Document isolation vs. efficiency trade-offs for different use cases

This provides complete coverage of testing workflow concepts and their
implementation in Molecule, helping teams choose appropriate testing
strategies for their specific automation domains and requirements.
- Remove Ansible-specific references from core testing phases
- Change 'Ansible roles and collections' to 'automation modules'
- Change 'playbooks, roles, or tasks' to 'automation scripts, configurations, or procedures'
- Ensure testing workflow fundamentals apply to any robust testing framework
- Maintain comprehensive coverage while being technology-neutral

The core testing phases now describe universal testing framework capabilities
rather than Ansible-specific implementation details, making the philosophy
section more broadly applicable while still leading naturally into how
Molecule implements these concepts for Ansible automation.
- Remove 'automation' specificity from 'What a testing suite provides'
- Change from 'Automation testing frameworks' to 'Testing frameworks'
- Update from 'automation domains' to 'testing domains' throughout
- Change 'automation workflows' to 'development workflows'
- Update 'automation testing' to just 'testing' where appropriate
- Change 'automation logic' to 'system logic' and 'system under test'
- Update examples from automation-specific to universal testing concepts
- Make testing workflow fundamentals apply to all testing domains

The philosophy section now describes universal testing framework capabilities
that apply to unit testing, integration testing, system testing, and any
other testing domain, not just automation testing. This provides a broader
foundation while still leading naturally into Molecule's automation-specific
implementation.
- Fix shared state explanation to accurately reflect default scenario usage
- Document that non-default scenarios skip create/destroy when shared state is enabled
- Explain that default scenario manages infrastructure lifecycle for shared state
- Update workflow examples to show proper default scenario usage
- Correct shared inventory coordination to show default scenario role
- Fix state isolation examples to reflect actual implementation behavior

Based on code review of src/molecule/command/base.py:
- execute_scenario() skips create/destroy for shared_data scenarios (lines 357-359)
- execute_subcommand_default() runs create/destroy from default scenario (lines 291-321)
- Default scenario must exist and have shared_data=True for shared state to work

This ensures documentation accurately reflects how Molecule actually implements
shared state functionality rather than how it might conceptually work.
- Correct shared state documentation to reflect automatic behavior
- Document that Molecule automatically runs default create before scenarios
- Document that Molecule automatically runs default destroy after scenarios
- Remove manual default scenario management from examples
- Show proper single scenario and multi-scenario automatic workflows
- Document molecule test --all --shared-state automatic behavior
- Fix shared inventory examples to show automatic coordination

Based on code review of src/molecule/command/base.py _run_scenarios():
- Line 227: execute_subcommand_default(default_config, 'create') runs automatically
- Line 286: execute_subcommand_default(default_config, 'destroy') runs automatically
- Lines 162-165: Default config auto-detected from MOLECULE_DEFAULT_SCENARIO_NAME

Users never need to manually run default scenario - Molecule handles it
automatically when --shared-state is enabled. This provides the key benefit
of running create/destroy only once even with molecule test --all.
…aring

- Clarify that shared inventory centralizes inventory location for all scenarios
- Document that default scenario writes connection details to shared inventory
- Document that other scenarios read from shared inventory to access same resources
- Add detailed explanation of how shared inventory enables resource sharing
- Emphasize write-once-read-many pattern for resource connection details
- Show that variables, groups, and connection options are shared through inventory
- Update workflow examples to highlight inventory-based resource sharing
- Connect shared-state and shared-inventory as complementary mechanisms

Technical details:
- Default scenario provisions infrastructure and writes to shared inventory
- All other scenarios read from same shared inventory location
- This enables true resource sharing without duplicate provisioning
- Each scenario sees identical infrastructure through shared inventory
- Remove complex driver architecture discussion
- Focus on Ansible as the default driver and its benefits
- Emphasize consistency between test and production patterns
- Highlight reduced learning curves through familiar constructs
- Document reliability benefits of using same Ansible knowledge
- Show how Ansible's declarative syntax improves test scenarios
- Connect test environment management to production automation

Benefits highlighted:
- Same Ansible knowledge for test and production
- Familiar playbooks and patterns across domains
- Declarative syntax for all environment management
- Test scenarios mirror real-world automation usage
- Reduced complexity and learning requirements
- Remove complex plugin architecture discussion
- Focus on Ansible as both provisioner and verifier
- Emphasize unified automation execution and validation logic
- Highlight same skills, modules, and patterns for test and production
- Document benefits of Ansible's extensive module ecosystem
- Show consistent interfaces across all automation domains
- Emphasize reduced complexity through unified approach
- Connect test validation to existing team Ansible expertise

Benefits highlighted:
- Unified approach using Ansible for both execution and validation
- Leverages existing team Ansible skills and knowledge
- Consistent module ecosystem across all domains
- Easily understood and maintainable by Ansible practitioners
- Reduces learning curve and operational complexity
- Document Molecule's 9-year evolution since 2015 inception
- Highlight that defaults are optimized from real-world usage at scale
- Explain enterprise workflow customization capabilities
- Emphasize sequence configurations as committed, shareable code
- Connect team consistency through version-controlled test sequences
- Show how defaults satisfy most users while enabling enterprise flexibility

Key points added:
- Real-world optimization of defaults over 9 years
- Enterprise-specific workflow requirements recognition
- Configuration as code for team consistency
- Committed sequences shared between team members
- Balance between sensible defaults and customization flexibility
- Document Molecule's strategic focus on deeper Ansible integration
- Emphasize specialization over broader tool support
- Detail current and planned integration enhancements
- Cover enhanced collection testing with shared state/inventory
- Describe unified UX with Ansible output alignment
- Explain deeper inventory integration and validation
- Document ansible-navigator and execution environment support
- Detail automatic collection detection and dependency resolution
- Establish strategic direction as definitive Ansible testing solution
- Emphasize backward compatibility while advancing Ansible-native experience
- Position Molecule as specialized framework rather than generic testing tool

Key themes:
- Ansible as de facto automation DSL drives development priorities
- Molecule + Ansible experience optimization over tool breadth
- Seamless integration while maintaining compatibility
- Enterprise collection testing workflow improvements
- Natural transition between development, testing, and deployment
@cidrblock cidrblock changed the title Add comprehensive testing philosophy documentation Add testing philosophy documentation Aug 11, 2025
- Replace all 'provisioner:' sections with modern 'ansible:' configuration
- Convert 'ansible_args' to 'ansible.executor.args.ansible_playbook'
- Add ansible.cfg configuration example for enhanced integration
- Update descriptive text to reference modern configuration patterns
- Maintain all existing functionality and testing approaches
- Ensure schema compliance and consistent formatting

All configuration examples now use the supported ansible-native approach
while preserving the comprehensive testing philosophy and strategies.
@cidrblock cidrblock marked this pull request as ready for review August 15, 2025 22:46
@cidrblock cidrblock merged commit 304bf27 into ansible:main Aug 15, 2025
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

1 participant