feat: add support for monospace in pushoverConfig #8018

heliapb · 2025-10-10T21:25:29Z

Description

Add support for monospace in pushoverConfig in alertmanager config, from version 0.29.0 onwards prometheus/alertmanager#4362

Type of change

What type of changes does your code introduce to the Prometheus operator? Put an x in the box that apply.

CHANGE (fix or feature that would cause existing functionality to not work as expected)
FEATURE (non-breaking change which adds functionality)
BUGFIX (non-breaking change which fixes an issue)
ENHANCEMENT (non-breaking change which improves existing functionality)
NONE (if none of the other choices apply. Example, tooling, build system, CI, docs, etc.)

Verification

Please check the Prometheus-Operator testing guidelines for recommendations about automated tests.

Changelog entry

Please put a one-line changelog entry below. This will be copied to the changelog file during the release process.

feat: add support for monospace in pushoverConfig

slashpai

lgtm
probably can wait till Alertmanager v0.29 release

heliapb · 2025-11-11T22:46:07Z

lgtm probably can wait till Alertmanager v0.29 release

hi @slashpai I think this can be merged now. thanks

slashpai

just a couple of nits after checking again

pkg/alertmanager/amcfg.go

slashpai · 2025-11-12T05:36:53Z

pkg/apis/monitoring/v1beta1/alertmanager_config_types.go

+	// monospace optional HTML/monospace formatting for the message, see https://pushover.net/api#html
+	// html and monospace formatting are mutually exclusive.
+	// +optional
+	Monospace bool `json:"monospace,omitempty"`


since optional make this as *bool?

I see for HTML as well its bool may be we should update that

cc: @simonpasquier

changed both to pointers as suggested.

Signed-off-by: Hélia Barroso <[email protected]>

simonpasquier

Thanks for the PR! We need to drop invalid AlertmanagerConfigs (e.g. with both monospace and html being true) before we sanitize. It should happen in checkPushoverConfigs().

Signed-off-by: Hélia Barroso <[email protected]>

heliapb · 2025-11-19T15:37:01Z

Thanks for the PR! We need to drop invalid AlertmanagerConfigs (e.g. with both monospace and html being true) before we sanitize. It should happen in checkPushoverConfigs().

thanks I added that and also noticed that I've missed the checks in validations, added those as well.

pkg/apis/monitoring/v1beta1/conversion_from.go

Signed-off-by: Hélia Barroso <[email protected]>

nutmos · 2025-11-20T13:38:53Z

pkg/apis/monitoring/v1beta1/alertmanager_config_types.go

-	HTML bool `json:"html,omitempty"`
+	HTML *bool `json:"html,omitempty"`
+	// monospace optional HTML/monospace formatting for the message, see https://pushover.net/api#html
+	// html and monospace formatting are mutually exclusive.


html and monospace formatting are mutually exclusive.

Should this message be added to the HTML description as well?

I've merged to have the PR in the next release but the comment is valid and we can address in a follow-up PR.

Makes sense, will open as follow up

onedr0p · 2025-12-09T12:14:53Z

@heliapb This PR broke HTML rendering for me. I upgraded to 0.87.0 and changed none of my configs, now HTML will not render in my notifications.

kubectl get alertmanagerconfigs.monitoring.coreos.com -n observability alertmanager -oyaml

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  creationTimestamp: "2025-11-01T17:05:04Z"
  generation: 3
  labels:
    kustomize.toolkit.fluxcd.io/name: kube-prometheus-stack
    kustomize.toolkit.fluxcd.io/namespace: observability
  name: alertmanager
  namespace: observability
  resourceVersion: "37268901"
  uid: 430cb801-8300-466f-addc-b844c14fbfd2
spec:
  inhibitRules:
  - equal:
    - alertname
    - namespace
    sourceMatch:
    - matchType: =
      name: severity
      value: critical
    targetMatch:
    - matchType: =
      name: severity
      value: warning
  receivers:
  - name: blackhole
  - name: buddy-heartbeat
    webhookConfigs:
    - httpConfig:
        bearerTokenSecret:
          key: BUDDY_HEARTBEAT_TOKEN
          name: alertmanager-secret
      urlSecret:
        key: BUDDY_HEARTBEAT_URL
        name: alertmanager-secret
  - name: pushover
    pushoverConfigs:
    - html: true
      message: |-
        {{- range .Alerts }}
          {{- if ne .Annotations.description "" }}
            {{ .Annotations.description }}
          {{- else if ne .Annotations.summary "" }}
            {{ .Annotations.summary }}
          {{- else if ne .Annotations.message "" }}
            {{ .Annotations.message }}
          {{- else }}
            Alert description not available
          {{- end }}
          {{- if gt (len .Labels.SortedPairs) 0 }}
            <small>
              {{- range .Labels.SortedPairs }}
                <b>{{ .Name }}:</b> {{ .Value }}
              {{- end }}
            </small>
          {{- end }}
        {{- end }}
      monospace: false
      priority: '{{ if eq .Status "firing" }}1{{ else }}0{{ end }}'
      sendResolved: true
      sound: gamelan
      title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing
        | len }}{{ end }}] {{ .CommonLabels.alertname }}'
      token:
        key: ALERTMANAGER_PUSHOVER_TOKEN
        name: alertmanager-secret
      ttl: 86400s
      urlTitle: View in Alertmanager
      userKey:
        key: PUSHOVER_USER_KEY
        name: alertmanager-secret
  route:
    groupBy:
    - alertname
    - cluster
    - job
    groupInterval: 5m
    groupWait: 1m
    receiver: pushover
    repeatInterval: 12h
    routes:
    - matchers:
      - matchType: =
        name: alertname
        value: InfoInhibitor
      receiver: blackhole
    - groupInterval: 2m
      groupWait: 0s
      matchers:
      - matchType: =
        name: alertname
        value: Watchdog
      receiver: buddy-heartbeat
      repeatInterval: 2m30s
    - matchers:
      - matchType: =
        name: severity
        value: critical
      receiver: pushover

onedr0p · 2025-12-09T13:29:11Z

Actually it looks like the html field is dropped completely when the config is rendered...

onedr0p · 2025-12-09T13:43:25Z

Opened issue #8152 for further discussion

simonpasquier · 2025-12-09T13:56:37Z

pkg/alertmanager/amcfg.go

 		URL:           in.URL,
 		URLTitle:      in.URLTitle,
 		Priority:      in.Priority,
-		HTML:          in.HTML,


heliapb · 2025-12-09T14:27:39Z

@heliapb This PR broke HTML rendering for me. I upgraded to 0.87.0 and changed none of my configs, now HTML will not render in my notifications.

kubectl get alertmanagerconfigs.monitoring.coreos.com -n observability alertmanager -oyaml

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  creationTimestamp: "2025-11-01T17:05:04Z"
  generation: 3
  labels:
    kustomize.toolkit.fluxcd.io/name: kube-prometheus-stack
    kustomize.toolkit.fluxcd.io/namespace: observability
  name: alertmanager
  namespace: observability
  resourceVersion: "37268901"
  uid: 430cb801-8300-466f-addc-b844c14fbfd2
spec:
  inhibitRules:
  - equal:
    - alertname
    - namespace
    sourceMatch:
    - matchType: =
      name: severity
      value: critical
    targetMatch:
    - matchType: =
      name: severity
      value: warning
  receivers:
  - name: blackhole
  - name: buddy-heartbeat
    webhookConfigs:
    - httpConfig:
        bearerTokenSecret:
          key: BUDDY_HEARTBEAT_TOKEN
          name: alertmanager-secret
      urlSecret:
        key: BUDDY_HEARTBEAT_URL
        name: alertmanager-secret
  - name: pushover
    pushoverConfigs:
    - html: true
      message: |-
        {{- range .Alerts }}
          {{- if ne .Annotations.description "" }}
            {{ .Annotations.description }}
          {{- else if ne .Annotations.summary "" }}
            {{ .Annotations.summary }}
          {{- else if ne .Annotations.message "" }}
            {{ .Annotations.message }}
          {{- else }}
            Alert description not available
          {{- end }}
          {{- if gt (len .Labels.SortedPairs) 0 }}
            <small>
              {{- range .Labels.SortedPairs }}
                <b>{{ .Name }}:</b> {{ .Value }}
              {{- end }}
            </small>
          {{- end }}
        {{- end }}
      monospace: false
      priority: '{{ if eq .Status "firing" }}1{{ else }}0{{ end }}'
      sendResolved: true
      sound: gamelan
      title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing
        | len }}{{ end }}] {{ .CommonLabels.alertname }}'
      token:
        key: ALERTMANAGER_PUSHOVER_TOKEN
        name: alertmanager-secret
      ttl: 86400s
      urlTitle: View in Alertmanager
      userKey:
        key: PUSHOVER_USER_KEY
        name: alertmanager-secret
  route:
    groupBy:
    - alertname
    - cluster
    - job
    groupInterval: 5m
    groupWait: 1m
    receiver: pushover
    repeatInterval: 12h
    routes:
    - matchers:
      - matchType: =
        name: alertname
        value: InfoInhibitor
      receiver: blackhole
    - groupInterval: 2m
      groupWait: 0s
      matchers:
      - matchType: =
        name: alertname
        value: Watchdog
      receiver: buddy-heartbeat
      repeatInterval: 2m30s
    - matchers:
      - matchType: =
        name: severity
        value: critical
      receiver: pushover

@onedr0p sorry my bad, I guess I deleted by mistake instead of adding monospace, seen that @simonpasquier already opened a pr with the fix 🙏

heliapb requested a review from a team as a code owner October 10, 2025 21:25

pull-request-size bot added the size/L label Oct 10, 2025

heliapb force-pushed the feat/pushover_config branch 2 times, most recently from 8dab075 to e54836d Compare October 15, 2025 15:54

slashpai previously approved these changes Oct 16, 2025

View reviewed changes

heliapb force-pushed the feat/pushover_config branch 2 times, most recently from 1b46a23 to f6b74e3 Compare November 5, 2025 12:48

heliapb requested a review from slashpai November 5, 2025 12:48

slashpai reviewed Nov 12, 2025

View reviewed changes

heliapb force-pushed the feat/pushover_config branch from f6b74e3 to 54b4693 Compare November 12, 2025 09:39

heliapb dismissed slashpai’s stale review via 570e9d5 November 12, 2025 13:47

heliapb requested a review from slashpai November 12, 2025 13:49

heliapb added 2 commits November 17, 2025 22:34

feat: add support for monospace in pushoverConfig

2d9121d

Signed-off-by: Hélia Barroso <[email protected]>

fix: bool, tests, comment

a6f0f7c

Signed-off-by: Hélia Barroso <[email protected]>

heliapb force-pushed the feat/pushover_config branch from 570e9d5 to a6f0f7c Compare November 17, 2025 22:34

simonpasquier reviewed Nov 19, 2025

View reviewed changes

feat: add missing validations for pushover

6347bf6

Signed-off-by: Hélia Barroso <[email protected]>

heliapb requested a review from simonpasquier November 19, 2025 15:37

simonpasquier reviewed Nov 19, 2025

View reviewed changes

pkg/apis/monitoring/v1beta1/conversion_from.go Show resolved Hide resolved

fix: typo

fff9912

Signed-off-by: Hélia Barroso <[email protected]>

nutmos reviewed Nov 20, 2025

View reviewed changes

simonpasquier approved these changes Nov 20, 2025

View reviewed changes

simonpasquier merged commit d1ff82e into prometheus-operator:main Nov 20, 2025
22 checks passed

heliapb deleted the feat/pushover_config branch November 20, 2025 14:33

heliapb mentioned this pull request Nov 20, 2025

fix: pushover doc #8110

Merged

5 tasks

onedr0p mentioned this pull request Dec 9, 2025

Alertmanager Pushover HTML is not rendering #8152

Closed

simonpasquier reviewed Dec 9, 2025

View reviewed changes

feat: add support for monospace in pushoverConfig #8018

feat: add support for monospace in pushoverConfig #8018

Uh oh!

Conversation

heliapb commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Verification

Changelog entry

Uh oh!

slashpai left a comment

Choose a reason for hiding this comment

Uh oh!

heliapb commented Nov 11, 2025

Uh oh!

slashpai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

slashpai Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

heliapb Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

simonpasquier left a comment

Choose a reason for hiding this comment

Uh oh!

heliapb commented Nov 19, 2025

Uh oh!

Uh oh!

nutmos Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

simonpasquier Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

heliapb Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

onedr0p commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

onedr0p commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

onedr0p commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simonpasquier Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

heliapb commented Dec 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

heliapb commented Oct 10, 2025 •

edited

Loading

onedr0p commented Dec 9, 2025 •

edited

Loading

onedr0p commented Dec 9, 2025 •

edited

Loading

onedr0p commented Dec 9, 2025 •

edited

Loading