Skip to content

feat(webhooks): add json-post default preset so url-only webhooks work (#676)#677

Merged
CybotTM merged 4 commits into
mainfrom
fix/676-generic-preset-and-default
May 15, 2026
Merged

feat(webhooks): add json-post default preset so url-only webhooks work (#676)#677
CybotTM merged 4 commits into
mainfrom
fix/676-generic-preset-and-default

Conversation

@CybotTM

@CybotTM CybotTM commented May 15, 2026

Copy link
Copy Markdown
Member

Summary

Closes #676. Adds the bundled json-post preset and a [global] webhook-default-preset selector. After this PR, a webhook configured with just url = ... actually works — Ofelia POSTs a JSON payload describing the execution to that URL, no custom preset YAML required.

The docs section "Custom Webhooks" already promised this behavior, but the code rejected it: loader.Load("") returned an error before any send code ran, and there was no bundled preset that turned a bare URL into a working webhook. The doc snippet has been false since the webhook system shipped.

Three-state semantics (no sentinel ambiguity)

DefaultPreset is *string so unset / explicit-empty / explicit-value are all distinguishable — mirrors the SaveConfig.RestoreHistory pattern in middlewares/save.go. Same conversation as #640 and #652 but resolved properly for this new field:

INI / label state Meaning EffectiveDefaultPreset() returns
key omitted entirely (nil) use bundled fallback "json-post"
webhook-default-preset = (non-nil "") explicit opt-out "" (NewWebhook then fails Validate)
webhook-default-preset = my-custom operator's choice "my-custom"

Verified end-to-end via gcfg INI parsing — see TestEffectiveDefaultPreset_ThreeIntents.

Resolution timing

The fallback is selected via WebhookGlobalConfig.EffectiveDefaultPreset() at webhook attach time in NewWebhook, not at daemon startup. Live INI reloads and label changes to webhook-default-preset take effect on the next attach without restart.

json-post payload shape

POST {url} with Content-Type: application/json; charset=utf-8, body:

{
  "job": {"name": "...", "command": "...", "schedule": "...", "type": "..."},
  "execution": {
    "id": "...", "status": "...",
    "failed": false, "skipped": false,
    "duration": "1.234s", "duration_ns": 1234000000,
    "start_time": "...", "end_time": "...",
    "error": "", "output": "...", "stderr": ""
  },
  "host": {"hostname": "...", "timestamp": "..."},
  "ofelia": {"version": "..."}
}

output and stderr are truncated to 4000 chars (test pins this). Optional link block is included when the webhook config has link = set. Every dynamic string field is wrapped through the template json helper so JSON-hostile chars (", \, \n, \r, \t) round-trip cleanly — exercised on the failure path by TestJSONPostPreset_FailedExecutionRendersValidJSON.

Tests (every documented use case has an e2e or unit pin)

Test What it pins
TestBundledPreset_JSONPostExists The YAML embeds, parses, registers under json-post, url is the only required variable.
TestEffectiveDefaultPreset_ThreeIntents nil / empty / explicit / nil-receiver — all four observable inputs resolve correctly.
TestNewWebhook_URLOnly_UsesJSONPostFallback End-to-end success path: url-only config attaches and posts valid JSON.
TestNewWebhook_DefaultPresetCustom Operator's-choice arm — fallback uses operator's preset, not json-post.
TestNewWebhook_ExplicitPresetWinsOverDefault Per-webhook preset = X overrides any global default.
TestNewWebhook_DefaultPresetOptOut DefaultPreset=&"" restores the pre-fallback "either preset or url" error.
TestNewWebhook_NilLoaderReturnsError New nil-loader guard (review feedback).
TestJSONPostPreset_FailedExecutionRendersValidJSON End-to-end failure path with quotes / newlines / backslashes in job name, command, error — body still parses as JSON, escaped fields round-trip.
TestJSONPostPreset_OutputTruncation 4000-char truncation of stdout/stderr.
TestApplyGlobalWebhookLabels_DefaultPreset 4 sub-tests: missing / empty (opt-out) / custom / non-string ignored.
TestMergeWebhookGlobals_DefaultPreset 5 sub-tests covering every nil/set combination on dst/src + explicit-INI-wins-over-label.

Coverage: mergeWebhookGlobals, applyGlobalWebhookLabels, EffectiveDefaultPreset, PresetLoader.DefaultPreset are all at 100% statement coverage.

Design notes worth flagging

  • Naming chose json-post over generic: preset: json-post self-documents (method + body format), matches the existing hyphenated convention (ntfy-token), and avoids generic tautology.
  • webhook-default-preset lives in the operator-tunable label allow-list. Not SSRF-sensitive: narrowing or widening the default cannot reach a new destination — the URL allow-list and preset loader still gate delivery.
  • This is the first *string field in WebhookGlobalConfig. Sibling string globals (Webhooks, AllowedHosts) still use plain string + sentinel-compare in mergeWebhookGlobals — inconsistent on purpose. *string is the right answer where unset / explicit-empty / explicit-value all matter; migrating siblings is a separate follow-up.

Test plan

  • go test ./... (incl. cli, middlewares)
  • golangci-lint run --timeout 3m
  • go test -coverprofile=... ./middlewares/... ./cli/... — all four new functions at 100%
  • End-to-end: url-only INI config attaches and posts valid JSON
  • End-to-end: failed execution with JSON-hostile chars produces valid JSON body
  • End-to-end: webhook-default-preset = (empty) restores the original validation error
  • gcfg INI parsing of *string verified for all three intents (omit / empty / explicit)

docs/webhooks.md "Custom Webhooks" section promised that a webhook with
just `url = ...` set would POST a JSON payload to that URL — no custom
preset YAML required. The code rejected this: `loader.Load("")` returned
an error before any send code ran, and there was no bundled preset that
turned a bare URL into a working webhook. The doc snippet has been false
since the webhook system shipped.

Fixes #676 by:

- Adding a bundled `json-post` preset (middlewares/presets/json-post.yaml,
  embedded into the binary). url_scheme `{url}`, method POST,
  Content-Type application/json, body shaped as
  {job:{...}, execution:{...}, host:{...}, ofelia:{...}}. Stdout / stderr
  are truncated to 4000 chars to keep payloads modest. The optional
  `link` field is included when set.

- Adding `webhook-default-preset` global, *string-typed so unset-vs-set
  is unambiguous (same pattern as SaveConfig.RestoreHistory):
    nil          → bundled DefaultPresetName ("json-post")
    non-nil ""   → explicit opt-out; webhooks missing `preset` fail
    non-nil "X"  → operator's chosen fallback preset name
  Resolution happens at webhook attach time via
  WebhookGlobalConfig.EffectiveDefaultPreset(), so live INI reloads and
  label changes take effect on the next attach without restart.

- Wiring the fallback into NewWebhook: when config.Preset == "", fill
  from loader.DefaultPreset() before Validate runs.

- Wiring the global through every existing seam: INI parse (gcfg),
  mapstructure, docker label allow-list (operator-tunable; not
  SSRF-sensitive — narrowing or widening the default cannot reach a new
  destination because the URL allow-list and preset loader still gate
  delivery), mergeWebhookGlobals (nil-check instead of sentinel-compare),
  applyGlobalWebhookLabels.

- Rewriting docs/webhooks.md "Custom Webhooks" to describe what now
  actually works, including the payload shape and the opt-out path.

Tests added:
- TestBundledPreset_JSONPostExists: preset embeds and validates.
- TestEffectiveDefaultPreset_ThreeIntents: nil / empty / explicit, plus
  nil-receiver safety.
- TestNewWebhook_URLOnly_UsesJSONPostFallback: end-to-end — url-only
  config attaches and the test server receives a valid JSON POST.
- TestNewWebhook_DefaultPresetOptOut: setting DefaultPreset=&"" restores
  the pre-fallback "either preset or url" error path.

Signed-off-by: Sebastian Mendel <[email protected]>
Copilot AI review requested due to automatic review settings May 15, 2026 17:00
@github-actions github-actions Bot added documentation Improvements or additions to documentation tests labels May 15, 2026
@github-actions

github-actions Bot commented May 15, 2026

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@gemini-code-assist

Copy link
Copy Markdown

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

github-actions[bot]
github-actions Bot previously approved these changes May 15, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated approval for maintainer PR

All automated quality gates passed. See SECURITY_CONTROLS.md for compensating controls.

@github-actions

github-actions Bot commented May 15, 2026

Copy link
Copy Markdown

✅ Mutation Testing Results

Mutation Score: 95.45% (threshold: 60%)

✨ Good job! Mutation score meets the threshold.

What is mutation testing?

Mutation testing measures test quality by introducing small changes (mutations) to the code and checking if tests detect them. A higher score means better test effectiveness.

  • Killed mutants: Tests caught the mutation (good!)
  • Survived mutants: Tests missed the mutation (needs improvement)

@codecov

codecov Bot commented May 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.89189% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.99%. Comparing base (7c1e2ec) to head (a42a47e).

Files with missing lines Patch % Lines
middlewares/preset.go 88.00% 6 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #677      +/-   ##
==========================================
+ Coverage   87.98%   87.99%   +0.01%     
==========================================
  Files          89       89              
  Lines       11216    11280      +64     
==========================================
+ Hits         9868     9926      +58     
- Misses       1099     1104       +5     
- Partials      249      250       +1     
Flag Coverage Δ
integration 87.99% <91.89%> (+0.01%) ⬆️
unittests 85.16% <91.89%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements the previously documented “URL-only webhook” behavior by introducing a bundled json-post preset and a new global selector ([global] webhook-default-preset) used as the fallback when a webhook omits preset.

Changes:

  • Add bundled json-post webhook preset and make it the default fallback for URL-only webhooks.
  • Add WebhookGlobalConfig.DefaultPreset (*string) with three-state semantics and wire it through preset loading and label merge paths.
  • Expand docs, tests, and changelog to cover the new default behavior and opt-out mechanism.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
middlewares/webhook.go Applies loader-provided default preset when per-webhook preset is omitted.
middlewares/webhook_test.go Adds tests for bundled preset existence, three-intent default semantics, URL-only attach/dispatch, and opt-out.
middlewares/webhook_config.go Introduces DefaultPresetName and EffectiveDefaultPreset() plus DefaultPreset *string in global config.
middlewares/presets/json-post.yaml Adds the bundled JSON POST preset used as the default fallback.
middlewares/preset.go Adds (*PresetLoader).DefaultPreset() helper to surface the effective global fallback preset.
docs/webhooks.md Documents webhook-default-preset, URL-only webhook usage, payload shape, and opt-out.
cli/docker-labels.go Allow-lists webhook-default-preset for Docker label ingestion.
cli/config_webhook.go Adds the new global key constant, merges the new global, and applies the label into live config.
CHANGELOG.md Adds an unreleased entry describing the new default preset and URL-only webhook behavior.

Comment thread cli/config_webhook.go Outdated
Comment thread docs/webhooks.md
Comment thread docs/webhooks.md Outdated
Comment thread docs/webhooks.md Outdated
Comment thread middlewares/webhook.go
Five review comments + codecov/patch failure (61% on diff, target 80%).

Code/comment fixes:
- middlewares/webhook.go: NewWebhook now returns an explicit error when
  loader is nil instead of dereferencing nil later. The "loader may be
  nil in tests" comment was wrong — the function unconditionally calls
  loader.Load() further down.
- cli/config_webhook.go: stale doc comment said the default was
  "generic"; now references middlewares.DefaultPresetName ("json-post")
  consistently with the rest of the code and the docs.

Doc fixes (docs/webhooks.md):
- INI-vs-labels note rewritten: webhook-default-preset is now also
  label-accepted; webhook-preset-cache-ttl already had label support
  (the "not yet implemented" claim was stale since #640).
- Content-Type aligned with the preset YAML: application/json; charset=utf-8.
- Opt-out wording clarified: with the fallback disabled, every webhook
  must declare preset explicitly — `url = ...` alone does NOT help,
  since `url` only overrides the preset's url_scheme.

Coverage (codecov/patch fail → fix):
- cli/TestApplyGlobalWebhookLabels_DefaultPreset: 4 sub-tests covering
  missing label / empty (opt-out) / custom value / non-string ignored.
- cli/TestMergeWebhookGlobals_DefaultPreset: 5 sub-tests covering all
  combinations of nil-vs-set on dst/src, plus explicit-INI-wins-over-label.
- middlewares/TestNewWebhook_DefaultPresetCustom: operator's-choice arm
  of EffectiveDefaultPreset (was untested in isolation from the unit
  test on the accessor).
- middlewares/TestNewWebhook_ExplicitPresetWinsOverDefault: per-webhook
  `preset = X` overrides any global default — the fallback is only a
  fallback.
- middlewares/TestNewWebhook_NilLoaderReturnsError: pins the new
  nil-guard contract added in this commit.
- middlewares/TestJSONPostPreset_FailedExecutionRendersValidJSON:
  end-to-end on the FAILURE path with JSON-hostile chars (quotes,
  newlines, backslashes) in job name, command, and error. Asserts the
  body parses as JSON and the escaped fields round-trip cleanly. This
  is the real regression value — any future refactor of the json-post
  template that forgets to wrap a string field through the `json`
  helper will fail here.
- middlewares/TestJSONPostPreset_OutputTruncation: pins the documented
  4000-char truncation of stdout/stderr.

mergeWebhookGlobals + applyGlobalWebhookLabels + EffectiveDefaultPreset
+ PresetLoader.DefaultPreset are now all at 100% statement coverage.

Signed-off-by: Sebastian Mendel <[email protected]>
github-actions[bot]
github-actions Bot previously approved these changes May 15, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated approval for maintainer PR

All automated quality gates passed. See SECURITY_CONTROLS.md for compensating controls.

…e, DX

Six parallel reviewers (security, anti-pattern, continuity, DX, doc-
accuracy, test-completeness) ran against the initial #677 commit; this
commit lands everything that fits in the same PR plus follow-up issues
for what doesn't.

Correctness (security LOW + anti-pattern Critical):

- middlewares/preset.go: the `json` template helper now escapes every
  RFC 8259 control byte (U+0000-U+001F), not just \, ", \n, \r, \t. The
  earlier hand-rolled escape would pass through NUL, BEL, FF, terminal
  ESC (0x1b, the start of every ANSI sequence), and the rest unchanged
  — strict JSON parsers on the receiving end then rejected the body.
  Real trigger: any job emitting progress bars, ls --color output,
  pytest, make. Implementation moved to a dedicated jsonStringEscape
  function with full round-trip regression test in
  TestWebhookTemplateFuncs_JSON_RFC8259EscapeCoverage.

- middlewares/preset.go: the `truncate` helper is now rune-aware
  (utf8.RuneCountInString + range over s for the boundary). The
  byte-slice version would split multi-byte UTF-8 mid-rune when
  truncating non-ASCII output, producing a replacement rune that would
  combine with the json helper to break JSON parsing downstream.
  TestWebhookTemplateFuncs_Truncate now pins multi-byte emoji + ASCII
  mixes.

- Added a new `jsonRaw` template helper backed by encoding/json.Marshal
  for future presets that want to emit a fully-formed JSON value rather
  than a string interior.

Test e2e completeness (test-engineer review HIGH gaps):

- cli/docker_labels_test.go: three new tests exercise the full Docker-
  label pipeline (buildFromDockerContainers → applyGlobalWebhookLabels
  → mergeWebhookGlobals → InitManager → NewWebhook) — previously only
  the programmatic path was end-to-end tested.
    - TestURLOnlyWebhookFromLabels_UsesJSONPostFallback: url-only label
      config resolves to json-post.
    - TestCustomDefaultPresetViaLabel_AppliesToURLOnlyWebhook:
      operator-chosen default via label flows through to NewWebhook.
    - TestDefaultPresetOptOutViaLabel_FailsURLOnlyWebhook: explicit
      empty label captured as non-nil "" (opt-out) and attach fails
      with the new actionable error.

- middlewares/webhook_test.go: TestNewWebhook_URLOnly_UsesJSONPostFallback
  now asserts actual field VALUES in the JSON body (not just key
  presence) so a future template refactor that hard-codes a field fails
  loudly. Added TestJSONPostPreset_OptionalLinkBlock_RendersValidJSON
  exercising BOTH branches of the {{if .Preset.Link}} conditional —
  catches the comma-after-ofelia / no-trailing-comma landmine.

DX (DX reviewer findings 1-6):

- README.md: line 74 webhook description now mentions the url-only flow
  ("just `url = ...` to POST a JSON payload via the bundled `json-post`
  fallback"). The global-keys reference list around line 291 now
  documents `webhook-default-preset` alongside its siblings.

- docs/webhooks.md: Quick Start now LEADS with "Simplest case: just
  POST to a URL" (json-post fallback example) before the preset-based
  examples. The Docker-label reference table now lists
  `ofelia.webhook-default-preset`. Opt-out section explains the WHY
  (audit policy, ban-implicit-fallback, BYO-default) before the WHAT.

- middlewares/webhook.go: NewWebhook's opt-out error now names
  `webhook-default-preset` and the bundled fallback so operators can
  grep their way from a log line to the docs. The previous error
  ("either preset or url") didn't mention either knob.

Continuity / upgrade impact (continuity reviewer Important):

- CHANGELOG.md: [Unreleased] Added entry now LEADS with the
  upgrade-impact lede ("webhooks configured with `url = ...` but no
  `preset` were previously rejected at startup … they now attach and
  fire"). Operators reading release notes during upgrade see the
  behavior change first, the API details second.

Follow-up issues filed for the four out-of-scope findings:

- #678 — [global] unknown-key warning lacks "did you mean?" suggestion
  (parity with job-section path; pre-existing asymmetry).
- #679 — Local preset file silently shadowed by bundled name; warn on
  AddLocalPresetDir collision (pre-existing precedence; PR #677 makes
  it more likely because `json-post` is a generic-enough name).
- #673 (already open) — sendWithRetry's time.Sleep ignores ctx.
- #674 (already open) — rebuildAllMiddlewares allocates fresh
  *http.Client per webhook per reconcile.

Signed-off-by: Sebastian Mendel <[email protected]>
github-actions[bot]
github-actions Bot previously approved these changes May 15, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated approval for maintainer PR

All automated quality gates passed. See SECURITY_CONTROLS.md for compensating controls.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Comment thread middlewares/preset.go Outdated
Comment thread middlewares/uncovered_ext_test.go Outdated
Two follow-up Copilot review comments on PR #677:

- middlewares/preset.go: DefaultPreset() godoc said the opt-out path
  fails attachment "when the per-webhook config also omits both
  `preset` and `url`". That was true before the actionable-error change
  in commit 57b4004 — now NewWebhook fails attachment whenever `preset`
  is empty after fallback, regardless of `url`, with the new error
  message that names `webhook-default-preset`. Comment now reflects the
  actual control-flow.

- middlewares/uncovered_ext_test.go: the regression-test comment
  pointed at "the parallel reviewer pass on #677" — accurate in
  isolation but #677 is the PR, not the canonical issue. Re-pointed at
  #676 (the issue tracking the json-post feature) so future readers can
  navigate from the test to the design discussion.

Signed-off-by: Sebastian Mendel <[email protected]>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated approval for maintainer PR

All automated quality gates passed. See SECURITY_CONTROLS.md for compensating controls.

@sonarqubecloud

Copy link
Copy Markdown

@CybotTM CybotTM added this pull request to the merge queue May 15, 2026
Merged via the queue into main with commit 37b598d May 15, 2026
27 checks passed
@CybotTM CybotTM deleted the fix/676-generic-preset-and-default branch May 15, 2026 18:01
CybotTM added a commit that referenced this pull request May 16, 2026
…u mean?"

Pre-fix, the unknown-key warning was asymmetric across INI section
types: job sections (job-exec, job-run, etc.) printed
'(did you mean "..."?)' via findClosestMatch, but [global] and [docker]
printed bare '(typo?)' with no suggestion — leaving the operator to
read the line and guess.

Surfaced in PR #677's parallel-reviewer pass: the new
webhook-default-preset global key is easy to mistype
(webhook-defauls-preset swaps t/s) and the typo logged a [global]
warning with no hint at the right spelling.

Fix extracts a logSectionUnknownKeyWarnings helper that mirrors
logJobUnknownKeyWarnings's filename/suggestion matrix and call it from
both [global] and [docker] paths in logUnknownKeyWarnings (file-based)
and BuildFromString (string-based). Known keys come from
extractMapstructureKeys(Config{}.Global) and
extractMapstructureKeys(DockerConfig{}) via two thin helpers
(globalKnownKeys / dockerKnownKeys) so the suggestion list cannot drift
from the actual decoded struct.

Three new tests:
- TestGlobalSectionUnknownKeyWarning_DidYouMean: the issue's exact
  example — webhook-defauls-preset suggests webhook-default-preset.
- TestGlobalSectionUnknownKeyWarning_NoSuggestion: locks the fallback
  to '(typo?)' when no close match exists.
- TestDockerSectionUnknownKeyWarning_DidYouMean: same parity assertion
  for the [docker] path so a future code split can't silently drop
  suggestions on one half.

Closes #678. Tracks against #621 (global-keys drift detection family).

Signed-off-by: Sebastian Mendel <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

webhook: url var are ignored for notifications

2 participants