Skip to content

test(e2e): add genuine end-to-end tests against the compiled binary#581

Merged
CybotTM merged 9 commits into
mainfrom
test/add-e2e
Apr 20, 2026
Merged

test(e2e): add genuine end-to-end tests against the compiled binary#581
CybotTM merged 9 commits into
mainfrom
test/add-e2e

Conversation

@CybotTM

@CybotTM CybotTM commented Apr 20, 2026

Copy link
Copy Markdown
Member

Summary

Replaces the previous skipped e2e slot in CI with real end-to-end coverage
that exercises the full pipeline: parse INI → schedule → fire → spawn job
→ collect output → graceful shutdown. All ten new tests build the ofelia
binary with -race and spawn it as a child process, asserting on its
stdout, file-system side effects, and container state via the docker CLI.

  • Harness (e2e/helpers_test.go): buildBinary() caches a go build -race per test process; startDaemon() spawns it with a generated INI
    config, captures stdout/stderr via a concurrency-safe buffer, waits for
    the boot banner, and isolates the child in its own process group so
    SIGTERM cannot leak to the test runner. Provides waitForLog,
    countLogOccurrences, waitExit and shutdown helpers plus a
    one-shot runCommand for validate-style tests, and thin docker-CLI
    wrappers.

  • Local-exec jobs (e2e/local_job_test.go):

    • TestE2E_LocalJob_RunsOnSchedule@every 1s job writes a disk marker;
      asserts the file-system effect, not just a log line.
    • TestE2E_LocalJob_RunOnStartuprun-on-startup = true fires at boot
      even with a far-future cron.
    • TestE2E_LocalJob_SurvivesMultipleExecutions — scheduler stays healthy
      across ~6 consecutive ticks and still accepts SIGTERM afterwards.
  • Real Docker jobs (e2e/docker_job_test.go), skip cleanly if docker
    is unavailable:

    • TestE2E_DockerRunJob_SpawnsContainer — runs alpine:3.20 via job-run,
      forces delete = false, verifies the marker reached the container
      via docker logs and ofelia's stdout forwarder.
    • TestE2E_DockerRunJob_FailingContainerMarkedFailed — container
      exiting 42 must surface as failed: true in ofelia's log.
  • Config validation (e2e/config_validation_test.go):

    • TestE2E_Validate_MalformedINI, TestE2E_Validate_MissingConfigFile,
      TestE2E_Validate_AcceptsValidConfig — check user-visible error
      text instead of exit codes, since ofelia.go intentionally exits 0
      after go-flags reports an error.
  • Graceful shutdown (e2e/graceful_shutdown_test.go):

    • TestE2E_GracefulShutdown_SIGTERM / ..._SIGINT — real signals
      against the child process; asserts the shutdown banner appears and
      the daemon exits cleanly within 10s.
  • CI wiring (.github/workflows/ci.yml): flip enable-e2e-tests: true
    and pin e2e-test-packages: ./e2e/... so the dedicated E2E Tests job
    runs on every PR, push to main, merge-group and weekly cron, reporting
    coverage under the e2e Codecov flag. .github/template.yaml's
    intentional-drift entry for ci.yml is updated to cover both
    enable-integration-tests and enable-e2e-tests.

Deliberately out of scope

  • job-compose (needs a compose stack) and Swarm job-service-run
    (needs a manager) — covered by their own integration tests.
  • Docker label discovery (cli/docker_handler_integration_test.go).
  • Web UI auth flows (web/ tests).
  • Full SIGHUP config-reload semantics — future work; the README calls
    it out explicitly.

The existing in-process e2e/scheduler_lifecycle_test.go is kept
untouched; the new tests are additive.

Test plan

Local (Go 1.26, Docker 29.4):

$ go test -tags=e2e -race -count=1 -timeout=5m ./e2e/...
ok github.com/netresearch/ofelia/e2e 9.095s

13/13 tests pass (3 pre-existing + 10 new), -race is clean,
golangci-lint run --build-tags=e2e,integration --timeout=5m ./... is
0 issues.

  • All ten new tests pass locally with -race
  • Existing scheduler_lifecycle_test.go still passes (no changes)
  • golangci-lint clean with e2e build tag
  • go build ./... still succeeds
  • CI E2E Tests job runs and uploads coverage under the e2e flag
  • CI Integration Tests job continues to pass (no changes there)
  • Template drift check is green (ci.yml listed under intentional-drift)

CybotTM added 7 commits April 20, 2026 14:47
Introduce helpers for e2e/ tests that exercise the real ofelia binary:
- buildBinary() compiles `go build -race -o <tmp>/ofelia .` once per
  `go test` invocation and caches the result.
- startDaemon() spawns the binary with an INI config, captures stdout/
  stderr via a concurrency-safe buffer, waits for the boot banner and
  isolates the child in its own process group so SIGTERM can't leak.
- waitForLog / countLogOccurrences / waitExit / shutdown provide the
  minimum set of observations needed to assert on scheduler behavior
  from the outside.
- runCommand() is a one-shot analogue for validate-style tests.
- dockerAvailable / dockerLogs / dockerRemove thinly wrap the docker
  CLI so container-backed tests skip cleanly when docker is absent.

These helpers unblock subsequent test_e2e categories (local-exec jobs,
docker-run jobs, config validation, graceful shutdown) that previously
relied on in-process mocks instead of real subprocess execution.

Signed-off-by: Sebastian Mendel <[email protected]>
Three scenarios running the compiled daemon with a real INI config:

- TestE2E_LocalJob_RunsOnSchedule: a @every 1s local job must write a
  distinct marker to disk. Asserts on the file-system side-effect rather
  than just stdout to prove the spawned process actually executed (not
  just that ofelia logged an "execution" event).

- TestE2E_LocalJob_RunOnStartup: run-on-startup=true must fire at boot
  even when the cron (0 0 1 1 *) would never trigger within the test
  window — isolates the startup-trigger code path.

- TestE2E_LocalJob_SurvivesMultipleExecutions: sanity-check that the
  scheduler keeps ticking across ~6 consecutive runs and still accepts
  SIGTERM afterwards. Regression guard against goroutine leaks or
  scheduling-drift hangs.

All three use t.TempDir() + t.Parallel() and rely exclusively on the
harness added in the prior commit.

Signed-off-by: Sebastian Mendel <[email protected]>
Two scenarios that exercise the full Docker path (unlike the existing
e2e tests which stub the provider):

- TestE2E_DockerRunJob_SpawnsContainer: runs `echo OFELIA_E2E_DOCKER_MARKER`
  inside alpine:3.20 via job-run, forces `delete = false` so the container
  survives long enough to inspect, then uses `docker logs` to verify the
  marker reached the container and came back through ofelia's stdout
  forwarder.

- TestE2E_DockerRunJob_FailingContainerMarkedFailed: a container that
  exits 42 must be surfaced as `failed: true` in ofelia's log. Regression
  guard so broken cron jobs never look healthy.

Both tests skip cleanly via `dockerAvailable()` when the docker daemon
is unreachable, pre-pull alpine:3.20 to avoid registry-latency races,
and use timestamped container names so parallel runs can't collide.

Signed-off-by: Sebastian Mendel <[email protected]>
Three scenarios for the `ofelia validate` subcommand:

- TestE2E_Validate_MalformedINI: unterminated section ⇒ the user must
  see the `unclosed section` detail and the `INI syntax` hint.
- TestE2E_Validate_MissingConfigFile: missing path ⇒ the error must
  include the path and the `no such file or directory` cause.
- TestE2E_Validate_AcceptsValidConfig: a well-formed config containing
  both job-local and job-run must produce a JSON dump that lists both
  jobs and their image.

The tests intentionally do not assert on the exit code because
ofelia.go documents exiting gracefully (exit 0) on argument errors;
they check the human-readable stderr/stdout instead — which is what
an operator debugging a failing deploy actually reads.

Signed-off-by: Sebastian Mendel <[email protected]>
…emon

Two scenarios exercising the full shutdown path end-to-end:

- TestE2E_GracefulShutdown_SIGTERM sends SIGTERM after at least one job
  has run, then asserts the daemon exits within 10s, emits the
  'Received shutdown signal' and 'graceful shutdown' banners, and
  leaves exit code 0. Exercises ShutdownManager → GracefulScheduler →
  main-loop `<-c.done` rendezvous via signal, which cannot be tested
  in-process because Go's signal delivery is per-PID.

- TestE2E_GracefulShutdown_SIGINT covers the Ctrl+C signal path
  independently, since ShutdownManager registers SIGINT and SIGTERM
  separately.

These complement the existing in-process scheduler lifecycle tests in
e2e/scheduler_lifecycle_test.go (kept untouched).

Signed-off-by: Sebastian Mendel <[email protected]>
…ut-of-scope items

Rewrite e2e/README.md so new contributors understand:
- What distinguishes e2e tests from the existing integration tests
  (build real binary vs. in-process Docker stub).
- How to run the suite locally (go test -tags=e2e -race ./e2e/...).
- Which scenarios are covered per-file.
- Which job types are deliberately out of scope (job-compose, Swarm
  job-service-run, web UI auth, SIGHUP reload) so reviewers don't
  conflate e2e with a fix-all target.
- How the helpers in helpers_test.go fit together.

Signed-off-by: Sebastian Mendel <[email protected]>
Flip `enable-e2e-tests: true` on the reusable go-check workflow caller
and pin `e2e-test-packages: ./e2e/...` so the dedicated E2E Tests job
runs on every PR, push to main, merge-group and weekly cron. Coverage
is uploaded under the `e2e` Codecov flag.

The e2e tests build the real ofelia binary and spawn it with Docker
on ubuntu-latest runners; see e2e/README.md for the list of scenarios.
They are independent of the existing integration tests (-tags=integration)
which remain in-process and continue to run in the Integration Tests job.

Update .github/template.yaml's intentional-drift entry so the
check-template-drift.yml workflow keeps accepting ci.yml's deviation
from the go-app template — now documented to cover both
`enable-integration-tests: true` and `enable-e2e-tests: true`.

Signed-off-by: Sebastian Mendel <[email protected]>
Copilot AI review requested due to automatic review settings April 20, 2026 12:49
@github-actions github-actions Bot added documentation Improvements or additions to documentation ci tests labels Apr 20, 2026
@github-actions

github-actions Bot commented Apr 20, 2026

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

github-actions[bot]
github-actions Bot previously approved these changes Apr 20, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated approval for maintainer PR

All automated quality gates passed. See SECURITY_CONTROLS.md for compensating controls.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a comprehensive end-to-end (E2E) testing suite for Ofelia, including a new test harness and tests for configuration validation, Docker and local job execution, and graceful shutdown. Documentation has been updated to detail the new E2E architecture. Review feedback suggests optimizing the test harness by implementing more efficient log searching and counting methods to avoid repeated buffer allocations, and replacing fixed sleep intervals with polling to improve test reliability in CI environments.

Comment thread e2e/helpers_test.go
Comment thread e2e/helpers_test.go
Comment thread e2e/helpers_test.go
Comment thread e2e/local_job_test.go Outdated
@codecov

codecov Bot commented Apr 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.19%. Comparing base (2eab2a7) to head (4e5fac8).
⚠️ Report is 10 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #581      +/-   ##
==========================================
- Coverage   87.27%   87.19%   -0.09%     
==========================================
  Files          88       88              
  Lines       10625    10625              
==========================================
- Hits         9273     9264       -9     
- Misses       1112     1120       +8     
- Partials      240      241       +1     
Flag Coverage Δ
integration 87.17% <ø> (-0.11%) ⬇️
unittests 83.77% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…or fs effects

Applies three suggestions from the gemini-code-assist review:

1. Add Contains/Count methods to syncBuffer that hold the mutex while
   running bytes.Contains / bytes.Count on the underlying buffer. The
   previous implementation copied the full buffer on every poll, which
   was O(N²) over the lifetime of a long-running test.

2. Rewrite waitForLog and countLogOccurrences on top of the new
   Contains/Count helpers — no more per-iteration allocations; the
   line-by-line bufio.Scanner in countLogOccurrences is replaced with
   a single bytes.Count call.

3. In TestE2E_LocalJob_RunsOnSchedule, replace the fixed 500ms sleep
   before reading the marker file with a 5s poll loop. Slow CI runners
   can exhibit multi-second fs-sync latency; polling keeps the test
   fast on fast machines and robust on slow ones.

All ten new e2e tests and the three pre-existing lifecycle tests continue
to pass under `go test -tags=e2e -race -count=1 ./e2e/...` locally.

Signed-off-by: Sebastian Mendel <[email protected]>
github-actions[bot]
github-actions Bot previously approved these changes Apr 20, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated approval for maintainer PR

All automated quality gates passed. See SECURITY_CONTROLS.md for compensating controls.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds true end-to-end coverage by running the compiled ofelia binary as a subprocess (instead of a previously skipped CI slot), validating behavior across config parsing, scheduling/execution, Docker interactions, and graceful shutdown.

Changes:

  • Introduces an e2e test harness that builds ofelia with -race, spawns ofelia daemon, and provides log/exit/signal helpers plus thin docker CLI wrappers.
  • Adds new e2e test suites for local jobs, Docker run jobs, config validation, and signal-driven graceful shutdown.
  • Enables e2e tests in CI and updates template drift documentation accordingly.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
e2e/helpers_test.go Adds the subprocess-based harness (build binary once, spawn daemon, capture logs, signal/shutdown helpers, docker CLI helpers).
e2e/local_job_test.go Adds local job e2e tests asserting filesystem side effects, run-on-startup, and repeated executions.
e2e/docker_job_test.go Adds docker run-job e2e tests that validate container execution/log propagation and failure surfacing.
e2e/config_validation_test.go Adds binary-level validate command tests for malformed config, missing files, and valid configs.
e2e/graceful_shutdown_test.go Adds SIGTERM/SIGINT tests against the real daemon subprocess to validate clean shutdown.
e2e/README.md Updates documentation to reflect the new binary-level e2e approach and what is covered/out of scope.
.github/workflows/ci.yml Enables the e2e job in the reusable CI workflow and scopes it to ./e2e/....
.github/template.yaml Updates intentional drift rationale to include e2e enablement in CI.

Comment thread e2e/helpers_test.go
Comment thread e2e/docker_job_test.go Outdated
…t docker jobs

Two fixes from the Copilot pull-request-reviewer pass:

1. killProcessGroup helper. Since startDaemon sets Setpgid=true, the
   daemon and any spawned job children share a process group. The
   SIGKILL fallback in shutdown() now signals the group (via negative
   PID) rather than just the daemon PID, so stray `sh` subprocesses
   from local-exec jobs that happened to be mid-execution when we had
   to SIGKILL can't leak. Graceful SIGTERM still goes only to the
   daemon so the shutdown manager can drain jobs itself.

2. Docker tests now schedule exactly one execution via
   `run-on-startup = true` + a far-future cron. The previous
   `@every 1s` + `delete = false` + fixed container-name combo caused
   tick #2+ to attempt re-creating the same container and collide on
   the docker name — producing spurious `name already in use` failure
   logs. A single-shot schedule keeps the state deterministic, removes
   log noise and shortens the docker tests from ~5s to ~5s (no change)
   while making the `failed: true` assertion in the failing-container
   test unambiguous (there is only ever one execution to fail).

Both tests still pass locally under `go test -tags=e2e -race`.

Signed-off-by: Sebastian Mendel <[email protected]>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated approval for maintainer PR

All automated quality gates passed. See SECURITY_CONTROLS.md for compensating controls.

@CybotTM CybotTM added this pull request to the merge queue Apr 20, 2026
Merged via the queue into main with commit a095beb Apr 20, 2026
22 checks passed
@CybotTM CybotTM deleted the test/add-e2e branch April 20, 2026 13:03
@CybotTM CybotTM mentioned this pull request May 10, 2026
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci documentation Improvements or additions to documentation tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants