Skip to content

Add canary testing system for end-to-end testing#1441

Merged
tomer-stripe merged 11 commits intomasterfrom
tomer/canary-testing-system
Mar 11, 2026
Merged

Add canary testing system for end-to-end testing#1441
tomer-stripe merged 11 commits intomasterfrom
tomer/canary-testing-system

Conversation

@tomer-stripe
Copy link
Copy Markdown
Collaborator

Reviewers

r? @
cc @stripe/developer-products

Summary

Note: I worked on this with Claude Code. This is something I've long wanted + was curious if it'd be useful to us. I basically want a version of compiling every PR against each target OS we support and running through some of the core capabilities to make sure it works. Curious if folks think this is useful. Claude's summary below 👇

Adds a comprehensive end-to-end testing system that builds CLI binaries and physically invokes commands to verify real world behavior across Linux, macOS, and Windows.

Key Gap Addressed: Current tests use Cobra's ExecuteC() for in-process testing. No tests actually invoke the compiled binary, meaning packaging/build issues would go undetected.

What's Included

  • Test Framework (canary/testutil/runner.go): Cross-platform binary execution helper with timeout, environment isolation, and config directory management
  • 32 Tests (canary/runner_test.go):
    • 19 offline tests (no API key required)
    • 13 API tests (require STRIPE_TEST_API_KEY secret)
  • CI Workflow (.github/workflows/canary-test.yml): Runs on all 3 platforms for every PR
  • Makefile targets: make canary and make canary-offline

Production-Like Build

Canary builds match production settings:

  • CGO_ENABLED=0
  • -ldflags="-s -w"
  • go generate ./...

Test Plan

  • All 32 tests pass locally on macOS
  • Verified with real Stripe test API key
  • CI runs on ubuntu-latest, macos-latest, windows-latest
  • Configure STRIPE_TEST_API_KEY secret for API tests

@tomer-stripe tomer-stripe requested a review from a team as a code owner February 11, 2026 02:11
@tomer-stripe tomer-stripe mentioned this pull request Mar 5, 2026
@tomer-stripe tomer-stripe changed the title Add canary testing system for end-to-end testomg Add canary testing system for end-to-end testing Mar 5, 2026
@tomer-stripe tomer-stripe force-pushed the tomer/canary-testing-system branch from 6138e4a to df27e1d Compare March 5, 2026 19:04
Copy link
Copy Markdown
Contributor

@vzhang-stripe vzhang-stripe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Claude recommends extending the timeout limits for some places. I'll let @vcheung-stripe make the final judgement

Copy link
Copy Markdown
Collaborator

@vcheung-stripe vcheung-stripe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but wondering if there's a way to check that it's passing first? I saw https://github.com/stripe/stripe-cli/actions/workflows/canary-test.yml but I'm not sure if these runs are related to the PR.

tomer-stripe and others added 11 commits March 11, 2026 09:21
Adds a new test framework that invokes the compiled CLI binary to catch
build/packaging issues that in-process tests miss. Includes:

- canary/testutil/runner.go: Cross-platform binary execution helper
- canary/runner_test.go: 16 offline tests + API tests (when key available)
- canary/README.md: Documentation
- .github/workflows/canary-test.yml: CI on Linux, macOS, Windows
- Makefile: canary and canary-offline targets

Offline tests run without API key. API tests require STRIPE_TEST_API_KEY
secret to be configured in GitHub.

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Committed-By-Agent: claude
Adds comprehensive tests for streaming commands and configuration:

- Listen: --print-secret to verify webhook connection, event filtering
- Logs tail: startup verification with timeout, filter support
- Config: set API key via config, multi-profile support
- Login: help verification (interactive login requires browser)

Build improvements:
- Use production-like build (CGO_ENABLED=0, -ldflags="-s -w", go generate)
- Add build-canary Makefile target

Fixes:
- Add --confirm flag for delete operations (skip interactive prompt)
- Create config directory structure in temp dirs for config tests

Total: 19 offline tests, 13 API tests (32 total)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Committed-By-Agent: claude
Adds background process support to the test runner and three new tests:
- TestAPIListenForwardTo: Verifies webhooks are forwarded to local server
  with correct payload, headers (Stripe-Signature), and event structure
- TestAPIListenOutputFormat: Verifies listen shows events in JSON format
- TestAPILogsTailCapture: Verifies logs tail captures API requests

The runner enhancement includes RunBackground(), WaitForOutput(),
GetOutput(), and Stop() methods for managing long-running CLI processes.

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Committed-By-Agent: claude
Split the monolithic runner_test.go into logical files:
- helpers_test.go: shared test utilities (getRunner, requireAPIKey)
- basic_test.go: version, help, completion, status, error handling
- api_test.go: API resource tests (balance, customers, products, events)
- listen_test.go: webhook listener tests
- logs_test.go: logs tail streaming tests
- config_test.go: config and authentication tests
- login_test.go: login command tests

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Committed-By-Agent: claude
Remove pull_request trigger from canary-test.yml (unit tests cover PRs).
Add workflow_call trigger so the release workflow can invoke canary tests
as a prerequisite — all three OS build jobs now depend on canary-tests
passing before GoReleaser runs. Update README to reflect new triggers.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
Committed-By-Agent: claude
Rename printf-like function logSanitized → logSanitizedf to satisfy
goprintffuncname linter. Fix whitespace alignment in sanitize.go for
gofmt compliance.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Committed-By-Agent: claude
…options, add login test

- Make STRIPE_TEST_API_KEY required for workflow_call (callers must pass it)
- Tighten API test if-condition to respect run_api_tests toggle per event type
- Add RunnerOption functional options to NewRunner for idiomatic Go construction
- Migrate test files to pass env at construction via getRunner(t, WithEnv(...))
- Add TestAPIGlobalAPIKeyFlag to verify non-interactive auth via --api-key flag

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Committed-By-Agent: claude
The secrets context can't be used in step-level if expressions in
GitHub Actions. Add a preceding step that checks secret availability
via an environment variable and exposes it as a step output.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Committed-By-Agent: claude
Enables testing the new workflow on a PR before merging to master.
Remove this trigger before merging.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Committed-By-Agent: claude
PR events don't have access to repo secrets, so API tests skip
gracefully while offline tests still run. Safe to keep enabled.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Committed-By-Agent: claude
Bump job timeout from 20 to 25 minutes and WaitForOutput timeouts from
30s to 60s for listen and logs tail tests. Webhook endpoint initialization
and API connections can take longer on slow runners.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
Committed-By-Agent: claude
@tomer-stripe tomer-stripe force-pushed the tomer/canary-testing-system branch from 68bd225 to aae0556 Compare March 11, 2026 16:27
Copy link
Copy Markdown
Collaborator

@vcheung-stripe vcheung-stripe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 as mentioned, it's passing on the branch but it doesn't have access to secrets until we merge

@tomer-stripe tomer-stripe merged commit 2e563b1 into master Mar 11, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants