Add canary testing system for end-to-end testing#1441
Merged
tomer-stripe merged 11 commits intomasterfrom Mar 11, 2026
Merged
Conversation
Merged
tomer-stripe
commented
Mar 5, 2026
6138e4a to
df27e1d
Compare
vzhang-stripe
approved these changes
Mar 5, 2026
Contributor
vzhang-stripe
left a comment
There was a problem hiding this comment.
Looks good to me! Claude recommends extending the timeout limits for some places. I'll let @vcheung-stripe make the final judgement
vcheung-stripe
approved these changes
Mar 10, 2026
Collaborator
vcheung-stripe
left a comment
There was a problem hiding this comment.
LGTM but wondering if there's a way to check that it's passing first? I saw https://github.com/stripe/stripe-cli/actions/workflows/canary-test.yml but I'm not sure if these runs are related to the PR.
Adds a new test framework that invokes the compiled CLI binary to catch build/packaging issues that in-process tests miss. Includes: - canary/testutil/runner.go: Cross-platform binary execution helper - canary/runner_test.go: 16 offline tests + API tests (when key available) - canary/README.md: Documentation - .github/workflows/canary-test.yml: CI on Linux, macOS, Windows - Makefile: canary and canary-offline targets Offline tests run without API key. API tests require STRIPE_TEST_API_KEY secret to be configured in GitHub. Co-Authored-By: Claude Opus 4.5 <[email protected]> Committed-By-Agent: claude
Adds comprehensive tests for streaming commands and configuration: - Listen: --print-secret to verify webhook connection, event filtering - Logs tail: startup verification with timeout, filter support - Config: set API key via config, multi-profile support - Login: help verification (interactive login requires browser) Build improvements: - Use production-like build (CGO_ENABLED=0, -ldflags="-s -w", go generate) - Add build-canary Makefile target Fixes: - Add --confirm flag for delete operations (skip interactive prompt) - Create config directory structure in temp dirs for config tests Total: 19 offline tests, 13 API tests (32 total) Co-Authored-By: Claude Opus 4.5 <[email protected]> Committed-By-Agent: claude
Adds background process support to the test runner and three new tests: - TestAPIListenForwardTo: Verifies webhooks are forwarded to local server with correct payload, headers (Stripe-Signature), and event structure - TestAPIListenOutputFormat: Verifies listen shows events in JSON format - TestAPILogsTailCapture: Verifies logs tail captures API requests The runner enhancement includes RunBackground(), WaitForOutput(), GetOutput(), and Stop() methods for managing long-running CLI processes. Co-Authored-By: Claude Opus 4.5 <[email protected]> Committed-By-Agent: claude
Split the monolithic runner_test.go into logical files: - helpers_test.go: shared test utilities (getRunner, requireAPIKey) - basic_test.go: version, help, completion, status, error handling - api_test.go: API resource tests (balance, customers, products, events) - listen_test.go: webhook listener tests - logs_test.go: logs tail streaming tests - config_test.go: config and authentication tests - login_test.go: login command tests Co-Authored-By: Claude Opus 4.5 <[email protected]> Committed-By-Agent: claude
Remove pull_request trigger from canary-test.yml (unit tests cover PRs). Add workflow_call trigger so the release workflow can invoke canary tests as a prerequisite — all three OS build jobs now depend on canary-tests passing before GoReleaser runs. Update README to reflect new triggers. Co-Authored-By: Claude Opus 4.6 <[email protected]> Committed-By-Agent: claude
Rename printf-like function logSanitized → logSanitizedf to satisfy goprintffuncname linter. Fix whitespace alignment in sanitize.go for gofmt compliance. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> Committed-By-Agent: claude
…options, add login test - Make STRIPE_TEST_API_KEY required for workflow_call (callers must pass it) - Tighten API test if-condition to respect run_api_tests toggle per event type - Add RunnerOption functional options to NewRunner for idiomatic Go construction - Migrate test files to pass env at construction via getRunner(t, WithEnv(...)) - Add TestAPIGlobalAPIKeyFlag to verify non-interactive auth via --api-key flag Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> Committed-By-Agent: claude
The secrets context can't be used in step-level if expressions in GitHub Actions. Add a preceding step that checks secret availability via an environment variable and exposes it as a step output. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> Committed-By-Agent: claude
Enables testing the new workflow on a PR before merging to master. Remove this trigger before merging. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> Committed-By-Agent: claude
PR events don't have access to repo secrets, so API tests skip gracefully while offline tests still run. Safe to keep enabled. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> Committed-By-Agent: claude
Bump job timeout from 20 to 25 minutes and WaitForOutput timeouts from 30s to 60s for listen and logs tail tests. Webhook endpoint initialization and API connections can take longer on slow runners. Co-Authored-By: Claude Opus 4.6 <[email protected]> Committed-By-Agent: claude
68bd225 to
aae0556
Compare
vcheung-stripe
approved these changes
Mar 11, 2026
Collaborator
vcheung-stripe
left a comment
There was a problem hiding this comment.
👍 as mentioned, it's passing on the branch but it doesn't have access to secrets until we merge
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reviewers
r? @
cc @stripe/developer-products
Summary
Note: I worked on this with Claude Code. This is something I've long wanted + was curious if it'd be useful to us. I basically want a version of compiling every PR against each target OS we support and running through some of the core capabilities to make sure it works. Curious if folks think this is useful. Claude's summary below 👇
Adds a comprehensive end-to-end testing system that builds CLI binaries and physically invokes commands to verify real world behavior across Linux, macOS, and Windows.
Key Gap Addressed: Current tests use Cobra's ExecuteC() for in-process testing. No tests actually invoke the compiled binary, meaning packaging/build issues would go undetected.
What's Included
Production-Like Build
Canary builds match production settings:
Test Plan