test(e2e): add genuine end-to-end tests against the compiled binary#581
Conversation
Introduce helpers for e2e/ tests that exercise the real ofelia binary: - buildBinary() compiles `go build -race -o <tmp>/ofelia .` once per `go test` invocation and caches the result. - startDaemon() spawns the binary with an INI config, captures stdout/ stderr via a concurrency-safe buffer, waits for the boot banner and isolates the child in its own process group so SIGTERM can't leak. - waitForLog / countLogOccurrences / waitExit / shutdown provide the minimum set of observations needed to assert on scheduler behavior from the outside. - runCommand() is a one-shot analogue for validate-style tests. - dockerAvailable / dockerLogs / dockerRemove thinly wrap the docker CLI so container-backed tests skip cleanly when docker is absent. These helpers unblock subsequent test_e2e categories (local-exec jobs, docker-run jobs, config validation, graceful shutdown) that previously relied on in-process mocks instead of real subprocess execution. Signed-off-by: Sebastian Mendel <[email protected]>
Three scenarios running the compiled daemon with a real INI config: - TestE2E_LocalJob_RunsOnSchedule: a @every 1s local job must write a distinct marker to disk. Asserts on the file-system side-effect rather than just stdout to prove the spawned process actually executed (not just that ofelia logged an "execution" event). - TestE2E_LocalJob_RunOnStartup: run-on-startup=true must fire at boot even when the cron (0 0 1 1 *) would never trigger within the test window — isolates the startup-trigger code path. - TestE2E_LocalJob_SurvivesMultipleExecutions: sanity-check that the scheduler keeps ticking across ~6 consecutive runs and still accepts SIGTERM afterwards. Regression guard against goroutine leaks or scheduling-drift hangs. All three use t.TempDir() + t.Parallel() and rely exclusively on the harness added in the prior commit. Signed-off-by: Sebastian Mendel <[email protected]>
Two scenarios that exercise the full Docker path (unlike the existing e2e tests which stub the provider): - TestE2E_DockerRunJob_SpawnsContainer: runs `echo OFELIA_E2E_DOCKER_MARKER` inside alpine:3.20 via job-run, forces `delete = false` so the container survives long enough to inspect, then uses `docker logs` to verify the marker reached the container and came back through ofelia's stdout forwarder. - TestE2E_DockerRunJob_FailingContainerMarkedFailed: a container that exits 42 must be surfaced as `failed: true` in ofelia's log. Regression guard so broken cron jobs never look healthy. Both tests skip cleanly via `dockerAvailable()` when the docker daemon is unreachable, pre-pull alpine:3.20 to avoid registry-latency races, and use timestamped container names so parallel runs can't collide. Signed-off-by: Sebastian Mendel <[email protected]>
Three scenarios for the `ofelia validate` subcommand: - TestE2E_Validate_MalformedINI: unterminated section ⇒ the user must see the `unclosed section` detail and the `INI syntax` hint. - TestE2E_Validate_MissingConfigFile: missing path ⇒ the error must include the path and the `no such file or directory` cause. - TestE2E_Validate_AcceptsValidConfig: a well-formed config containing both job-local and job-run must produce a JSON dump that lists both jobs and their image. The tests intentionally do not assert on the exit code because ofelia.go documents exiting gracefully (exit 0) on argument errors; they check the human-readable stderr/stdout instead — which is what an operator debugging a failing deploy actually reads. Signed-off-by: Sebastian Mendel <[email protected]>
…emon Two scenarios exercising the full shutdown path end-to-end: - TestE2E_GracefulShutdown_SIGTERM sends SIGTERM after at least one job has run, then asserts the daemon exits within 10s, emits the 'Received shutdown signal' and 'graceful shutdown' banners, and leaves exit code 0. Exercises ShutdownManager → GracefulScheduler → main-loop `<-c.done` rendezvous via signal, which cannot be tested in-process because Go's signal delivery is per-PID. - TestE2E_GracefulShutdown_SIGINT covers the Ctrl+C signal path independently, since ShutdownManager registers SIGINT and SIGTERM separately. These complement the existing in-process scheduler lifecycle tests in e2e/scheduler_lifecycle_test.go (kept untouched). Signed-off-by: Sebastian Mendel <[email protected]>
…ut-of-scope items Rewrite e2e/README.md so new contributors understand: - What distinguishes e2e tests from the existing integration tests (build real binary vs. in-process Docker stub). - How to run the suite locally (go test -tags=e2e -race ./e2e/...). - Which scenarios are covered per-file. - Which job types are deliberately out of scope (job-compose, Swarm job-service-run, web UI auth, SIGHUP reload) so reviewers don't conflate e2e with a fix-all target. - How the helpers in helpers_test.go fit together. Signed-off-by: Sebastian Mendel <[email protected]>
Flip `enable-e2e-tests: true` on the reusable go-check workflow caller and pin `e2e-test-packages: ./e2e/...` so the dedicated E2E Tests job runs on every PR, push to main, merge-group and weekly cron. Coverage is uploaded under the `e2e` Codecov flag. The e2e tests build the real ofelia binary and spawn it with Docker on ubuntu-latest runners; see e2e/README.md for the list of scenarios. They are independent of the existing integration tests (-tags=integration) which remain in-process and continue to run in the Integration Tests job. Update .github/template.yaml's intentional-drift entry so the check-template-drift.yml workflow keeps accepting ci.yml's deviation from the go-app template — now documented to cover both `enable-integration-tests: true` and `enable-e2e-tests: true`. Signed-off-by: Sebastian Mendel <[email protected]>
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
There was a problem hiding this comment.
Automated approval for maintainer PR
All automated quality gates passed. See SECURITY_CONTROLS.md for compensating controls.
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive end-to-end (E2E) testing suite for Ofelia, including a new test harness and tests for configuration validation, Docker and local job execution, and graceful shutdown. Documentation has been updated to detail the new E2E architecture. Review feedback suggests optimizing the test harness by implementing more efficient log searching and counting methods to avoid repeated buffer allocations, and replacing fixed sleep intervals with polling to improve test reliability in CI environments.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #581 +/- ##
==========================================
- Coverage 87.27% 87.19% -0.09%
==========================================
Files 88 88
Lines 10625 10625
==========================================
- Hits 9273 9264 -9
- Misses 1112 1120 +8
- Partials 240 241 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
…or fs effects Applies three suggestions from the gemini-code-assist review: 1. Add Contains/Count methods to syncBuffer that hold the mutex while running bytes.Contains / bytes.Count on the underlying buffer. The previous implementation copied the full buffer on every poll, which was O(N²) over the lifetime of a long-running test. 2. Rewrite waitForLog and countLogOccurrences on top of the new Contains/Count helpers — no more per-iteration allocations; the line-by-line bufio.Scanner in countLogOccurrences is replaced with a single bytes.Count call. 3. In TestE2E_LocalJob_RunsOnSchedule, replace the fixed 500ms sleep before reading the marker file with a 5s poll loop. Slow CI runners can exhibit multi-second fs-sync latency; polling keeps the test fast on fast machines and robust on slow ones. All ten new e2e tests and the three pre-existing lifecycle tests continue to pass under `go test -tags=e2e -race -count=1 ./e2e/...` locally. Signed-off-by: Sebastian Mendel <[email protected]>
There was a problem hiding this comment.
Automated approval for maintainer PR
All automated quality gates passed. See SECURITY_CONTROLS.md for compensating controls.
There was a problem hiding this comment.
Pull request overview
Adds true end-to-end coverage by running the compiled ofelia binary as a subprocess (instead of a previously skipped CI slot), validating behavior across config parsing, scheduling/execution, Docker interactions, and graceful shutdown.
Changes:
- Introduces an e2e test harness that builds
ofeliawith-race, spawnsofelia daemon, and provides log/exit/signal helpers plus thin docker CLI wrappers. - Adds new e2e test suites for local jobs, Docker run jobs, config validation, and signal-driven graceful shutdown.
- Enables e2e tests in CI and updates template drift documentation accordingly.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| e2e/helpers_test.go | Adds the subprocess-based harness (build binary once, spawn daemon, capture logs, signal/shutdown helpers, docker CLI helpers). |
| e2e/local_job_test.go | Adds local job e2e tests asserting filesystem side effects, run-on-startup, and repeated executions. |
| e2e/docker_job_test.go | Adds docker run-job e2e tests that validate container execution/log propagation and failure surfacing. |
| e2e/config_validation_test.go | Adds binary-level validate command tests for malformed config, missing files, and valid configs. |
| e2e/graceful_shutdown_test.go | Adds SIGTERM/SIGINT tests against the real daemon subprocess to validate clean shutdown. |
| e2e/README.md | Updates documentation to reflect the new binary-level e2e approach and what is covered/out of scope. |
| .github/workflows/ci.yml | Enables the e2e job in the reusable CI workflow and scopes it to ./e2e/.... |
| .github/template.yaml | Updates intentional drift rationale to include e2e enablement in CI. |
…t docker jobs Two fixes from the Copilot pull-request-reviewer pass: 1. killProcessGroup helper. Since startDaemon sets Setpgid=true, the daemon and any spawned job children share a process group. The SIGKILL fallback in shutdown() now signals the group (via negative PID) rather than just the daemon PID, so stray `sh` subprocesses from local-exec jobs that happened to be mid-execution when we had to SIGKILL can't leak. Graceful SIGTERM still goes only to the daemon so the shutdown manager can drain jobs itself. 2. Docker tests now schedule exactly one execution via `run-on-startup = true` + a far-future cron. The previous `@every 1s` + `delete = false` + fixed container-name combo caused tick #2+ to attempt re-creating the same container and collide on the docker name — producing spurious `name already in use` failure logs. A single-shot schedule keeps the state deterministic, removes log noise and shortens the docker tests from ~5s to ~5s (no change) while making the `failed: true` assertion in the failing-container test unambiguous (there is only ever one execution to fail). Both tests still pass locally under `go test -tags=e2e -race`. Signed-off-by: Sebastian Mendel <[email protected]>
There was a problem hiding this comment.
Automated approval for maintainer PR
All automated quality gates passed. See SECURITY_CONTROLS.md for compensating controls.
Summary
Replaces the previous
skippede2e slot in CI with real end-to-end coveragethat exercises the full pipeline: parse INI → schedule → fire → spawn job
→ collect output → graceful shutdown. All ten new tests build the
ofeliabinary with
-raceand spawn it as a child process, asserting on itsstdout, file-system side effects, and container state via the docker CLI.
Harness (
e2e/helpers_test.go):buildBinary()caches ago build -raceper test process;startDaemon()spawns it with a generated INIconfig, captures stdout/stderr via a concurrency-safe buffer, waits for
the boot banner, and isolates the child in its own process group so
SIGTERM cannot leak to the test runner. Provides
waitForLog,countLogOccurrences,waitExitandshutdownhelpers plus aone-shot
runCommandforvalidate-style tests, and thin docker-CLIwrappers.
Local-exec jobs (
e2e/local_job_test.go):TestE2E_LocalJob_RunsOnSchedule— @every 1s job writes a disk marker;asserts the file-system effect, not just a log line.
TestE2E_LocalJob_RunOnStartup—run-on-startup = truefires at booteven with a far-future cron.
TestE2E_LocalJob_SurvivesMultipleExecutions— scheduler stays healthyacross ~6 consecutive ticks and still accepts SIGTERM afterwards.
Real Docker jobs (
e2e/docker_job_test.go), skip cleanly if dockeris unavailable:
TestE2E_DockerRunJob_SpawnsContainer— runs alpine:3.20 via job-run,forces
delete = false, verifies the marker reached the containervia
docker logsand ofelia's stdout forwarder.TestE2E_DockerRunJob_FailingContainerMarkedFailed— containerexiting 42 must surface as
failed: truein ofelia's log.Config validation (
e2e/config_validation_test.go):TestE2E_Validate_MalformedINI,TestE2E_Validate_MissingConfigFile,TestE2E_Validate_AcceptsValidConfig— check user-visible errortext instead of exit codes, since
ofelia.gointentionally exits 0after go-flags reports an error.
Graceful shutdown (
e2e/graceful_shutdown_test.go):TestE2E_GracefulShutdown_SIGTERM/..._SIGINT— real signalsagainst the child process; asserts the shutdown banner appears and
the daemon exits cleanly within 10s.
CI wiring (
.github/workflows/ci.yml): flipenable-e2e-tests: trueand pin
e2e-test-packages: ./e2e/...so the dedicatedE2E Testsjobruns on every PR, push to main, merge-group and weekly cron, reporting
coverage under the
e2eCodecov flag..github/template.yaml'sintentional-drift entry for
ci.ymlis updated to cover bothenable-integration-testsandenable-e2e-tests.Deliberately out of scope
job-compose(needs a compose stack) and Swarmjob-service-run(needs a manager) — covered by their own integration tests.
cli/docker_handler_integration_test.go).web/tests).it out explicitly.
The existing in-process
e2e/scheduler_lifecycle_test.gois keptuntouched; the new tests are additive.
Test plan
Local (Go 1.26, Docker 29.4):
13/13 tests pass (3 pre-existing + 10 new),
-raceis clean,golangci-lint run --build-tags=e2e,integration --timeout=5m ./...is0 issues.
-racescheduler_lifecycle_test.gostill passes (no changes)golangci-lintclean withe2ebuild taggo build ./...still succeedsE2E Testsjob runs and uploads coverage under thee2eflagIntegration Testsjob continues to pass (no changes there)