For more details on the architecture, see the official Alertmanager documentation<\/a>.<\/p>\n\n\n\n Create a dedicated user for the Alertmanager service:<\/p>\n\n\n\n Ubuntu\/Debian:<\/strong><\/p>\n\n\n\n Rocky\/AlmaLinux:<\/strong><\/p>\n\n\n\n Create the configuration and data directories:<\/p>\n\n\n\n Download and install the latest Alertmanager release:<\/p>\n\n\n\n At the time of writing, this installs Alertmanager 0.31.1<\/strong>. Verify the installation:<\/p>\n\n\n\n You should see the version and build info:<\/p>\n\n\n\n On Rocky\/AlmaLinux<\/strong>, set the SELinux context for the binaries:<\/p>\n\n\n\n Email is the most common notification channel. The Alertmanager configuration uses YAML and supports multiple notification integrations in a single file.<\/p>\n\n\n\n Start with a basic email configuration. Replace the SMTP settings with your actual mail server credentials:<\/p>\n\n\n\n Key settings to understand:<\/p>\n\n\n\n For Gmail, you’ll need to generate an App Password in your Google Account settings. Regular passwords won’t work with SMTP if 2FA is enabled.<\/p>\n\n\n\n Slack is ideal for team-visible alerts. Create an incoming webhook in your Slack workspace under Apps > Incoming Webhooks<\/strong>, then add a Slack receiver to the configuration.<\/p>\n\n\n\n Update the receivers section in Microsoft Teams supports incoming webhooks through Power Automate workflows (the old Office 365 Connector method is being deprecated). Create a workflow that accepts an HTTP POST and posts to a Teams channel, then use the webhook URL in Alertmanager.<\/p>\n\n\n\n Add a Teams receiver using the webhook_configs integration:<\/p>\n\n\n\n For richer formatting in Teams, you’ll need an intermediate webhook relay that transforms the Alertmanager JSON payload into an Adaptive Card format. The prometheus-msteams<\/a> project handles this conversion.<\/p>\n\n\n\n PagerDuty is the standard for on-call alerting. Create a Prometheus integration in PagerDuty to get a routing key, then add this receiver:<\/p>\n\n\n\n PagerDuty uses the Events API v2 with routing keys (not service keys from v1). Get the routing key from Services > Service Directory > Your Service > Integrations > Events API v2<\/strong>.<\/p>\n\n\n\n The real power of Alertmanager is its routing tree. Here’s a production-ready route configuration that sends critical alerts to PagerDuty and Slack, warnings to email, and informational alerts to Slack only:<\/p>\n\n\n\n Notice the The Inhibition rules suppress less important alerts when a related critical alert is firing. This prevents alert floods – when a server is down, you don’t need separate notifications for every service on that server.<\/p>\n\n\n\n Add this to your The first rule says: if a critical alert is firing, suppress any warning alerts with the same alertname and instance. The second rule suppresses all alerts for an instance when InstanceDown is firing – because if the machine is down, alerting on high CPU or low disk on that same machine is noise.<\/p>\n\n\n\n The default Alertmanager notification format is functional but verbose. Custom templates let you control exactly what information appears in each notification and how it’s formatted.<\/p>\n\n\n\n Create the templates directory and a custom email template:<\/p>\n\n\n\n Add a clean, production-friendly email template:<\/p>\n\n\n\n Reference the template in your email receiver config by adding Templates use Go’s Create the systemd unit file for Alertmanager:<\/p>\n\n\n\n Add the following service definition:<\/p>\n\n\n\n The Set ownership on the configuration file and start the service:<\/p>\n\n\n\n Verify the service is running:<\/p>\n\n\n\n The service should show active (running):<\/p>\n\n\n\n Alertmanager listens on port 9093. Open it in your firewall.<\/p>\n\n\n\n Ubuntu\/Debian (UFW):<\/strong><\/p>\n\n\n\n Rocky\/AlmaLinux (firewalld):<\/strong><\/p>\n\n\n\n Open your browser and navigate to Silences temporarily mute alerts during maintenance windows. The First, configure amtool to know where Alertmanager is running:<\/p>\n\n\n\n Create a silence for a specific alert during a maintenance window:<\/p>\n\n\n\n The command returns a silence ID that you can use to manage it:<\/p>\n\n\n\n List all active silences:<\/p>\n\n\n\n You’ll see a table of all active silences with their matchers and expiry times:<\/p>\n\n\n\n Expire a silence before its scheduled end time:<\/p>\n\n\n\n You can also query the current alert status:<\/p>\n\n\n\n Always validate the Alertmanager config before restarting the service. A bad config will prevent Alertmanager from starting, which means no alerts get delivered.<\/p>\n\n\n\n A successful validation returns:<\/p>\n\n\n\n When alerts aren’t reaching the expected receiver, use amtool to test which route an alert would match:<\/p>\n\n\n\n This shows the receiver that would handle an alert with those labels:<\/p>\n\n\n\n You can also view the full route tree:<\/p>\n\n\n\n This prints a visual representation of your entire routing hierarchy, making it easy to spot misconfigurations.<\/p>\n\n\n\n If you haven’t already configured Prometheus to send alerts to Alertmanager, add the alerting block to your Reload Prometheus after the change:<\/p>\n\n\n\n Verify the connection by checking the Prometheus UI under Status > Runtime & Build Information<\/strong>. The Alertmanager should appear as a discovered endpoint.<\/p>\n\n\n\n Don’t wait for a real incident to find out your alerts are broken. Use amtool to send a test alert through the entire pipeline:<\/p>\n\n\n\n This creates a real alert in Alertmanager that will be routed through your configured receivers. Check your email, Slack, or PagerDuty to confirm delivery. The test alert will auto-resolve after Alertmanager’s You can also trigger test alerts from Prometheus by temporarily lowering the threshold on an existing rule. For example, change For production environments where a single Alertmanager is a single point of failure, run multiple instances in a cluster. Alertmanager uses a gossip protocol to synchronize silences and notification state across instances. Each instance needs the In the Prometheus configuration, list all Alertmanager instances so that if one goes down, alerts still reach the others:<\/p>\n\n\n\n The cluster automatically deduplicates notifications, so you won’t get triple alerts even though all three instances receive the same firing alert from Prometheus. Port 9094 needs to be open between the Alertmanager nodes for gossip communication.<\/p>\n\n\n\n Grafana can also use Alertmanager as a notification target. In Grafana, go to Alerting > Contact points<\/strong> and add a new contact point with type “Alertmanager”. Set the URL to This is the most common issue on cloud VMs. Alertmanager’s cluster discovery tries to find a private RFC 1918 IP address for gossip communication. On cloud instances with only public interfaces or NAT-ed networking, it fails.<\/p>\n\n\n\n The fix for single-node deployments is to disable clustering entirely:<\/p>\n\n\n\n Add this to your systemd service ExecStart line. For multi-node clusters, explicitly set the bind address:<\/p>\n\n\n\n Check the Alertmanager logs for SMTP errors:<\/p>\n\n\n\n Common issues: wrong port (use 587 for TLS, 465 for SSL), authentication failure (Gmail needs App Password, not regular password), or firewall blocking outbound SMTP. Test SMTP connectivity from the server:<\/p>\n\n\n\n First verify alerts are reaching Alertmanager:<\/p>\n\n\n\n If alerts appear there but notifications aren’t sent, the issue is in the route configuration. Test your routing:<\/p>\n\n\n\n If the test shows the wrong receiver, adjust your route matchers. Also check if the alert might be inhibited by an active inhibition rule or silenced by an active silence.<\/p>\n\n\n\n Slack notification text uses Go template syntax. Common mistakes include missing Alertmanager stores silence data in its The Yes. Update the Prometheus Edit the Alertmanager config file and reload the service. Alertmanager supports hot reloading – either send SIGHUP to the process or POST to the Alertmanager is the critical link between Prometheus detection and human action. With the routing, inhibition, and multi-channel configuration covered here, you have a production-ready alerting pipeline that routes critical issues to PagerDuty, sends warnings to Slack, and uses email as the catch-all. Fine-tune the Alertmanager handles alert deduplication, grouping, silencing, and routing for Prometheus. It takes alerts from Prometheus and sends notifications to the right people through the right channels – email, Slack, PagerDuty, Microsoft Teams, and more. This guide covers a full Alertmanager setup with real-world routing configurations on both Ubuntu and Rocky Linux. How Does Alertmanager Work? … Read more<\/a><\/p>\n","protected":false},"author":3,"featured_media":164046,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[299,50,165],"tags":[],"class_list":["post-164045","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-how-to","category-linux-tutorials","category-monitoring"],"_links":{"self":[{"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/posts\/164045","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/comments?post=164045"}],"version-history":[{"count":0,"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/posts\/164045\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/media\/164046"}],"wp:attachment":[{"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/media?parent=164045"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/categories?post=164045"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/computingforgeeks.com\/wp-json\/wp\/v2\/tags?post=164045"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Prerequisites<\/h2>\n\n\n\n
\n
Step 1: Install Alertmanager<\/h2>\n\n\n\n
sudo useradd --no-create-home --shell \/bin\/false alertmanager<\/code><\/pre>\n\n\n\nsudo useradd --no-create-home --shell \/sbin\/nologin alertmanager<\/code><\/pre>\n\n\n\nsudo mkdir -p \/etc\/alertmanager \/var\/lib\/alertmanager\nsudo chown alertmanager:alertmanager \/var\/lib\/alertmanager<\/code><\/pre>\n\n\n\nVER=$(curl -sI https:\/\/github.com\/prometheus\/alertmanager\/releases\/latest | grep -i ^location | grep -o v[0-9.]* | sed s\/^v\/\/)\necho \"Installing Alertmanager $VER\"\nwget https:\/\/github.com\/prometheus\/alertmanager\/releases\/download\/v${VER}\/alertmanager-${VER}.linux-amd64.tar.gz\ntar xvf alertmanager-${VER}.linux-amd64.tar.gz\nsudo cp alertmanager-${VER}.linux-amd64\/{alertmanager,amtool} \/usr\/local\/bin\/\nsudo chown alertmanager:alertmanager \/usr\/local\/bin\/{alertmanager,amtool}<\/code><\/pre>\n\n\n\nalertmanager --version<\/code><\/pre>\n\n\n\nalertmanager, version 0.31.1 (branch: HEAD, revision: e8d9fba4)\n build user: root@71dc7c3e3a06\n build date: 20260228-14:18:37\n go version: go1.24.0\n platform: linux\/amd64<\/code><\/pre>\n\n\n\nsudo dnf install -y policycoreutils-python-utils\nsudo semanage fcontext -a -t bin_t \"\/usr\/local\/bin\/alertmanager\"\nsudo semanage fcontext -a -t bin_t \"\/usr\/local\/bin\/amtool\"\nsudo restorecon -v \/usr\/local\/bin\/{alertmanager,amtool}\nsudo semanage port -a -t http_port_t -p tcp 9093<\/code><\/pre>\n\n\n\nStep 2: Configure Email Alerts (SMTP)<\/h2>\n\n\n\n
sudo vi \/etc\/alertmanager\/alertmanager.yml<\/code><\/pre>\n\n\n\nglobal:\n resolve_timeout: 5m\n smtp_smarthost: 'smtp.gmail.com:587'\n smtp_from: 'alertmanager@example.com'\n smtp_auth_username: 'alertmanager@example.com'\n smtp_auth_password: 'your-app-password'\n smtp_require_tls: true\n\ntemplates:\n - '\/etc\/alertmanager\/templates\/*.tmpl'\n\nroute:\n receiver: 'email-notifications'\n group_by: ['alertname', 'instance']\n group_wait: 30s\n group_interval: 5m\n repeat_interval: 4h\n\n routes:\n - match:\n severity: critical\n receiver: 'critical-email'\n repeat_interval: 1h\n\n - match:\n severity: warning\n receiver: 'email-notifications'\n repeat_interval: 4h\n\nreceivers:\n - name: 'email-notifications'\n email_configs:\n - to: 'team@example.com'\n send_resolved: true\n\n - name: 'critical-email'\n email_configs:\n - to: 'oncall@example.com'\n send_resolved: true\n headers:\n Subject: '[CRITICAL] {{ .GroupLabels.alertname }} firing on {{ .GroupLabels.instance }}'<\/code><\/pre>\n\n\n\n\n
group_wait: 30s<\/code> – Wait 30 seconds for related alerts to arrive before sending the first notification<\/li>\ngroup_interval: 5m<\/code> – Wait 5 minutes before sending updates about new alerts in the same group<\/li>\nrepeat_interval: 4h<\/code> – Don’t re-send the same alert more often than every 4 hours<\/li>\nsend_resolved: true<\/code> – Send a notification when an alert resolves (goes back to normal)<\/li>\n<\/ul>\n\n\n\nStep 3: Add Slack Integration<\/h2>\n\n\n\n
\/etc\/alertmanager\/alertmanager.yml<\/code> to include Slack:<\/p>\n\n\n\nreceivers:\n - name: 'email-notifications'\n email_configs:\n - to: 'team@example.com'\n send_resolved: true\n\n - name: 'critical-email'\n email_configs:\n - to: 'oncall@example.com'\n send_resolved: true\n\n - name: 'slack-notifications'\n slack_configs:\n - api_url: 'https:\/\/hooks.slack.com\/services\/T00000000\/B00000000\/XXXXXXXXXXXXXXXXXXXXXXXX'\n channel: '#alerts'\n title: '{{ .GroupLabels.alertname }}'\n text: >-\n {{ range .Alerts }}\n *Alert:* {{ .Annotations.summary }}\n *Description:* {{ .Annotations.description }}\n *Severity:* {{ .Labels.severity }}\n {{ end }}\n send_resolved: true\n color: '{{ if eq .Status \"firing\" }}danger{{ else }}good{{ end }}'\n\n - name: 'slack-critical'\n slack_configs:\n - api_url: 'https:\/\/hooks.slack.com\/services\/T00000000\/B00000000\/XXXXXXXXXXXXXXXXXXXXXXXX'\n channel: '#critical-alerts'\n title: 'CRITICAL: {{ .GroupLabels.alertname }}'\n text: >-\n {{ range .Alerts }}\n *Alert:* {{ .Annotations.summary }}\n *Instance:* {{ .Labels.instance }}\n *Description:* {{ .Annotations.description }}\n {{ end }}\n send_resolved: true<\/code><\/pre>\n\n\n\nStep 4: Add Microsoft Teams Webhook<\/h2>\n\n\n\n
- name: 'teams-notifications'\n webhook_configs:\n - url: 'https:\/\/prod-XX.westus.logic.azure.com:443\/workflows\/XXXXX\/triggers\/manual\/paths\/invoke?api-version=2016-06-01'\n send_resolved: true\n http_config:\n follow_redirects: true<\/code><\/pre>\n\n\n\nStep 5: Add PagerDuty Integration<\/h2>\n\n\n\n
- name: 'pagerduty-critical'\n pagerduty_configs:\n - routing_key: 'your-pagerduty-events-v2-routing-key'\n severity: '{{ .GroupLabels.severity }}'\n description: '{{ .GroupLabels.alertname }}: {{ .CommonAnnotations.summary }}'\n details:\n firing: '{{ .Alerts.Firing | len }}'\n resolved: '{{ .Alerts.Resolved | len }}'\n instance: '{{ .GroupLabels.instance }}'<\/code><\/pre>\n\n\n\nStep 6: Configure Severity-Based Route Tree<\/h2>\n\n\n\n
route:\n receiver: 'email-notifications'\n group_by: ['alertname', 'cluster', 'service']\n group_wait: 30s\n group_interval: 5m\n repeat_interval: 4h\n\n routes:\n - match:\n severity: critical\n receiver: 'pagerduty-critical'\n continue: true\n repeat_interval: 30m\n\n - match:\n severity: critical\n receiver: 'slack-critical'\n repeat_interval: 1h\n\n - match:\n severity: warning\n receiver: 'slack-notifications'\n repeat_interval: 4h\n\n - match_re:\n alertname: '^(Watchdog|InfoInhibitor)$'\n receiver: 'null'\n\nreceivers:\n - name: 'null'\n\n - name: 'email-notifications'\n email_configs:\n - to: 'team@example.com'\n send_resolved: true\n\n - name: 'pagerduty-critical'\n pagerduty_configs:\n - routing_key: 'your-pagerduty-routing-key'\n\n - name: 'slack-critical'\n slack_configs:\n - api_url: 'https:\/\/hooks.slack.com\/services\/XXXXX'\n channel: '#critical-alerts'\n send_resolved: true\n\n - name: 'slack-notifications'\n slack_configs:\n - api_url: 'https:\/\/hooks.slack.com\/services\/XXXXX'\n channel: '#alerts'\n send_resolved: true<\/code><\/pre>\n\n\n\ncontinue: true<\/code> on the first critical route. This tells Alertmanager to keep matching after this route, so critical alerts go to both PagerDuty AND Slack. Without continue<\/code>, routing stops at the first match.<\/p>\n\n\n\nnull<\/code> receiver is a common pattern for dropping alerts you don’t want notifications for, like the Watchdog heartbeat alert from kube-prometheus-stack.<\/p>\n\n\n\nStep 7: Configure Inhibition Rules<\/h2>\n\n\n\n
alertmanager.yml<\/code>:<\/p>\n\n\n\ninhibit_rules:\n - source_match:\n severity: 'critical'\n target_match:\n severity: 'warning'\n equal: ['alertname', 'instance']\n\n - source_match:\n alertname: 'InstanceDown'\n target_match_re:\n alertname: '.+'\n equal: ['instance']<\/code><\/pre>\n\n\n\nStep 8: Create Custom Notification Templates<\/h2>\n\n\n\n
sudo mkdir -p \/etc\/alertmanager\/templates\nsudo vi \/etc\/alertmanager\/templates\/email.tmpl<\/code><\/pre>\n\n\n\n{{ define \"email.custom.subject\" }}\n[{{ .Status | toUpper }}] {{ .GroupLabels.alertname }} ({{ .Alerts | len }} alerts)\n{{ end }}\n\n{{ define \"email.custom.html\" }}\n<h2>{{ .GroupLabels.alertname }}<\/h2>\n<p>Status: <strong>{{ .Status | toUpper }}<\/strong><\/p>\n<table border=\"1\" cellpadding=\"5\">\n <tr><th>Instance<\/th><th>Severity<\/th><th>Summary<\/th><\/tr>\n {{ range .Alerts }}\n <tr>\n <td>{{ .Labels.instance }}<\/td>\n <td>{{ .Labels.severity }}<\/td>\n <td>{{ .Annotations.summary }}<\/td>\n <\/tr>\n {{ end }}\n<\/table>\n{{ end }}<\/code><\/pre>\n\n\n\nhtml<\/code> and headers<\/code> fields:<\/p>\n\n\n\n - name: 'email-notifications'\n email_configs:\n - to: 'team@example.com'\n send_resolved: true\n html: '{{ template \"email.custom.html\" . }}'\n headers:\n Subject: '{{ template \"email.custom.subject\" . }}'<\/code><\/pre>\n\n\n\ntext\/template<\/code> syntax. The most common variables are .Status<\/code>, .GroupLabels<\/code>, .CommonLabels<\/code>, .CommonAnnotations<\/code>, and .Alerts<\/code> (which you iterate with range<\/code>). Each alert inside .Alerts<\/code> has .Labels<\/code>, .Annotations<\/code>, .StartsAt<\/code>, and .EndsAt<\/code>.<\/p>\n\n\n\nStep 9: Create the Systemd Service<\/h2>\n\n\n\n
sudo vi \/etc\/systemd\/system\/alertmanager.service<\/code><\/pre>\n\n\n\n[Unit]\nDescription=Prometheus Alertmanager\nDocumentation=https:\/\/prometheus.io\/docs\/alerting\/latest\/alertmanager\/\nWants=network-online.target\nAfter=network-online.target\n\n[Service]\nType=simple\nUser=alertmanager\nGroup=alertmanager\nExecStart=\/usr\/local\/bin\/alertmanager \\\n --config.file=\/etc\/alertmanager\/alertmanager.yml \\\n --storage.path=\/var\/lib\/alertmanager \\\n --web.listen-address=0.0.0.0:9093 \\\n --cluster.listen-address=\"\"\nRestart=always\nRestartSec=5\nSyslogIdentifier=alertmanager\n\n[Install]\nWantedBy=multi-user.target<\/code><\/pre>\n\n\n\n--cluster.listen-address=\"\"<\/code> flag is critical for single-node deployments, especially on cloud VMs. Without it, Alertmanager tries to find a private IP for its gossip protocol and fails with “no private IP found”<\/strong> on many cloud providers. If you’re running a multi-node Alertmanager cluster, remove this flag and use --cluster.peer<\/code> instead.<\/p>\n\n\n\nsudo chown alertmanager:alertmanager \/etc\/alertmanager\/alertmanager.yml\nsudo systemctl daemon-reload\nsudo systemctl enable --now alertmanager<\/code><\/pre>\n\n\n\nsudo systemctl status alertmanager<\/code><\/pre>\n\n\n\n\u25cf alertmanager.service - Prometheus Alertmanager\n Loaded: loaded (\/etc\/systemd\/system\/alertmanager.service; enabled; preset: enabled)\n Active: active (running) since Mon 2026-03-24 11:05:32 UTC; 4s ago\n Docs: https:\/\/prometheus.io\/docs\/alerting\/latest\/alertmanager\/\n Main PID: 4521 (alertmanager)\n Tasks: 7 (limit: 4557)\n Memory: 18.4M\n CPU: 0.234s\n CGroup: \/system.slice\/alertmanager.service\n \u2514\u25004521 \/usr\/local\/bin\/alertmanager --config.file=\/etc\/alertmanager\/alertmanager.yml ...<\/code><\/pre>\n\n\n\nStep 9: Open Firewall Ports<\/h2>\n\n\n\n
sudo ufw allow 9093\/tcp comment \"Alertmanager\"\nsudo ufw reload<\/code><\/pre>\n\n\n\nsudo firewall-cmd --permanent --add-port=9093\/tcp\nsudo firewall-cmd --reload<\/code><\/pre>\n\n\n\nStep 10: Access the Alertmanager Web UI<\/h2>\n\n\n\n
http:\/\/your-server-ip:9093<\/code>. The Alertmanager UI shows active alerts, silences, and the current routing configuration:<\/p>\n\n\n\n![\"Alertmanager](\"https:\/\/computingforgeeks.com\/wp-content\/uploads\/2026\/03\/alertmanager-ui.png\" "\\"\\"")
How to Manage Silences with amtool<\/h2>\n\n\n\n
amtool<\/code> CLI makes managing silences easy without touching the web UI.<\/p>\n\n\n\nsudo mkdir -p \/etc\/amtool\necho \"alertmanager.url: http:\/\/localhost:9093\" | sudo tee \/etc\/amtool\/config.yml<\/code><\/pre>\n\n\n\namtool silence add alertname=InstanceDown instance=\"10.0.1.20:9100\" --duration=2h --comment=\"Scheduled maintenance on web-02\"<\/code><\/pre>\n\n\n\n4b5e8f2a-1c3d-4e5f-9a8b-7c6d5e4f3a2b<\/code><\/pre>\n\n\n\namtool silence query<\/code><\/pre>\n\n\n\nID Matchers Ends Created By Comment\n4b5e8f2a-1c3d-4e5f-9a8b-7c6d5e4f3a2b alertname=InstanceDown instance=10.0.1.20:9100 2026-03-24 13:05:32 admin Scheduled maintenance on web-02<\/code><\/pre>\n\n\n\namtool silence expire 4b5e8f2a-1c3d-4e5f-9a8b-7c6d5e4f3a2b<\/code><\/pre>\n\n\n\namtool alert query<\/code><\/pre>\n\n\n\nHow to Validate Your Configuration<\/h2>\n\n\n\n
amtool check-config \/etc\/alertmanager\/alertmanager.yml<\/code><\/pre>\n\n\n\nChecking '\/etc\/alertmanager\/alertmanager.yml' SUCCESS\nFound:\n - global config\n - route\n - 1 inhibit rules\n - 5 receivers\n - 0 templates<\/code><\/pre>\n\n\n\nHow to Debug Alert Routing<\/h2>\n\n\n\n
amtool config routes test --config.file=\/etc\/alertmanager\/alertmanager.yml severity=critical alertname=InstanceDown<\/code><\/pre>\n\n\n\npagerduty-critical<\/code><\/pre>\n\n\n\namtool config routes show --config.file=\/etc\/alertmanager\/alertmanager.yml<\/code><\/pre>\n\n\n\nConnect Alertmanager to Prometheus<\/h2>\n\n\n\n
\/etc\/prometheus\/prometheus.yml<\/code>:<\/p>\n\n\n\nalerting:\n alertmanagers:\n - static_configs:\n - targets:\n - localhost:9093<\/code><\/pre>\n\n\n\nsudo systemctl reload prometheus<\/code><\/pre>\n\n\n\nHow to Test Alert Notifications<\/h2>\n\n\n\n
amtool alert add test_alert severity=critical instance=\"test-server:9090\" --annotation=summary=\"This is a test alert\" --annotation=description=\"Testing the alerting pipeline\"<\/code><\/pre>\n\n\n\nresolve_timeout<\/code> (5 minutes with the default config).<\/p>\n\n\n\nHighCPUUsage<\/code> threshold from 85 to 1 temporarily – this will fire on any system with more than 1% CPU usage, which is essentially every running system. Remember to revert the change after testing.<\/p>\n\n\n\nHigh Availability with Alertmanager Clustering<\/h2>\n\n\n\n
--cluster.peer<\/code> flag pointing to the other instances:<\/p>\n\n\n\nExecStart=\/usr\/local\/bin\/alertmanager \\\n --config.file=\/etc\/alertmanager\/alertmanager.yml \\\n --storage.path=\/var\/lib\/alertmanager \\\n --web.listen-address=0.0.0.0:9093 \\\n --cluster.listen-address=0.0.0.0:9094 \\\n --cluster.peer=10.0.1.11:9094 \\\n --cluster.peer=10.0.1.12:9094<\/code><\/pre>\n\n\n\nalerting:\n alertmanagers:\n - static_configs:\n - targets:\n - 10.0.1.10:9093\n - 10.0.1.11:9093\n - 10.0.1.12:9093<\/code><\/pre>\n\n\n\nSet Up Alerting in Grafana<\/h2>\n\n\n\n
http:\/\/localhost:9093<\/code>. This lets Grafana-managed alerts use the same routing and grouping logic as Prometheus alerts.<\/p>\n\n\n\n![\"Grafana](\"https:\/\/computingforgeeks.com\/wp-content\/uploads\/2026\/03\/grafana-alerting.png\" "\\"\\"")
Troubleshooting Common Issues<\/h2>\n\n\n\n
Alertmanager fails with “no private IP found”<\/h3>\n\n\n\n
--cluster.listen-address=\"\"<\/code><\/pre>\n\n\n\n--cluster.listen-address=\"10.0.1.10:9094\"<\/code><\/pre>\n\n\n\nEmails not being delivered<\/h3>\n\n\n\n
sudo journalctl -u alertmanager -n 50 --no-pager | grep -i smtp<\/code><\/pre>\n\n\n\ncurl -v telnet:\/\/smtp.gmail.com:587<\/code><\/pre>\n\n\n\nAlerts are firing but no notifications arrive<\/h3>\n\n\n\n
amtool alert query<\/code><\/pre>\n\n\n\namtool config routes test --config.file=\/etc\/alertmanager\/alertmanager.yml severity=warning alertname=HighCPUUsage<\/code><\/pre>\n\n\n\nSlack messages not formatting correctly<\/h3>\n\n\n\n
range<\/code> blocks for iterating over alerts and incorrect template variable names. Test your templates with a simple config first, then add complexity. The Alertmanager logs will show template rendering errors if the syntax is wrong.<\/p>\n\n\n\nAlertmanager restarts lose silence state<\/h3>\n\n\n\n
--storage.path<\/code> directory. If you’re running Alertmanager with a temporary storage path or the directory gets wiped, silences will be lost on restart. Make sure \/var\/lib\/alertmanager<\/code> is persistent and backed up. The nflog (notification log) that tracks which notifications were already sent is also stored there – without it, Alertmanager may re-send alerts that were already delivered.<\/p>\n\n\n\nAlerts are grouping incorrectly<\/h3>\n\n\n\n
group_by<\/code> setting determines which alerts are bundled into a single notification. If you group by ['alertname']<\/code> only, all InstanceDown alerts for different servers will be sent as one message. If you want separate notifications per server, add 'instance'<\/code> to the group_by list. Be careful not to group by too many labels – this can lead to one notification per alert, which defeats the purpose of grouping and causes alert flood during outages.<\/p>\n\n\n\nFrequently Asked Questions<\/h2>\n\n\n\n
Can I run Alertmanager on a different server than Prometheus?<\/h3>\n\n\n\n
alertmanagers<\/code> target to point to the remote server IP and port. Make sure port 9093 is open in the firewall between the two servers. Running Alertmanager separately is common in larger deployments where you want the alerting system to survive even if the Prometheus server has issues.<\/p>\n\n\n\nHow do I add a new notification channel without downtime?<\/h3>\n\n\n\n
\/-\/reload<\/code> endpoint if --web.enable-lifecycle<\/code> is enabled. Validate the config first with amtool check-config<\/code> to avoid breaking a running instance.<\/p>\n\n\n\nConclusion<\/h2>\n\n\n\n
group_wait<\/code>, group_interval<\/code>, and repeat_interval<\/code> values based on your team’s response patterns – the defaults here are a solid starting point that prevent alert fatigue without missing real incidents.<\/p>\n","protected":false},"excerpt":{"rendered":"