mgr/prometheus: Add BlueStore fragmentation alert by AnuragRaut08 · Pull Request #72889 · ceph/ceph

AnuragRaut08 · 2026-10-21T05:03:27Z

Fixes: https://tracker.ceph.com/issues/81948
Signed-off-by: Anurag Raut [email protected]

PR Description:

The Prometheus alert rules for BlueStore free-space fragmentation need to identify affected OSDs and provide actionable troubleshooting information
This PR addresses this by:

Defining an alert rule to detect high BlueStore free-space fragmentation.
Including OSD-specific labels to identify affected OSDs.
Providing descriptive alert annotations with the fragmentation score and troubleshooting command.
Setting appropriate alert severity and type labels.

Validation covers rule syntax, metric scraping, and alert evaluation using controlled fragmentation metrics for multiple OSDs.

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins test classic perf Jenkins Job | Jenkins Job Definition
jenkins test crimson perf Jenkins Job | Jenkins Job Definition
jenkins test signed Jenkins Job | Jenkins Job Definition
jenkins test make check Jenkins Job | Jenkins Job Definition
jenkins test make check arm64 Jenkins Job | Jenkins Job Definition
jenkins test submodules Jenkins Job | Jenkins Job Definition
jenkins test dashboard Jenkins Job | Jenkins Job Definition
jenkins test dashboard cephadm Jenkins Job | Jenkins Job Definition
jenkins test api Jenkins Job | Jenkins Job Definition
jenkins test docs ReadTheDocs | Github Workflow Definition
jenkins test ceph-volume all Jenkins Jobs | Jenkins Jobs Definition
jenkins test windows Jenkins Job | Jenkins Job Definition
jenkins test rook e2e Jenkins Job | Jenkins Job Definition

You must only issue one Jenkins command per-comment. Jenkins does not understand
comments with more than one command.

abhidesai6

Add screenshot / screencast of the alert raised. Show UI alert screen, and prometheus alert condition.

github-actions · 2026-10-26T19:00:57Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

AnuragRaut08 · 2026-10-27T05:58:24Z

Added screenshots showing CephOSDFreeSpaceFragmentation firing in Prometheus and appearing as an active alert in the Ceph Dashboard.

cloudbehl

Feels like we could simplify this a bit. We’re handling cert health checks separately here and skipping them again in the generic health-history loop. Could we define a shared constant for these checks and reuse it in both places? That way, we don’t have to maintain the same alert names in multiple places, and it’s easier to keep things in sync.

abhidesai6 · 2026-10-27T15:30:55Z

Currently with your changes , we are only showing summary of message. We also want detail of each warned OSD in message.
Can you take care of this ?

[ceph: root@ceph-node-00 /]# ceph health detail --format json

{"status":"HEALTH_WARN","checks":{"BLUESTORE_FREE_FRAGMENTATION":{"severity":"HEALTH_WARN","summary":{"message":"6 OSD(s) experiencing high free space fragmentation of BlueStore","count":6},"detail":[{"message":" osd.0 0.000089"},{"message":" osd.1 0.000089"},{"message":" osd.2 0.000069"},{"message":" osd.3 0.000037"},{"message":" osd.4 0.000032"},{"message":" osd.5 0.000035"}],"muted":false}},"mutes":[]}
[ceph: root@ceph-node-00 /]# ceph health detail --format json | jq
{
  "status": "HEALTH_WARN",
  "checks": {
    "BLUESTORE_FREE_FRAGMENTATION": {
      "severity": "HEALTH_WARN",
      "summary": {
        "message": "6 OSD(s) experiencing high free space fragmentation of BlueStore",
        "count": 6
      },
      "detail": [
        {
          "message": " osd.0 0.000089"
        },
        {
          "message": " osd.1 0.000089"
        },
        {
          "message": " osd.2 0.000069"
        },
        {
          "message": " osd.3 0.000037"
        },
        {
          "message": " osd.4 0.000032"
        },
        {
          "message": " osd.5 0.000035"
        }
      ],
      "muted": false
    }
  },
  "mutes": []
}
[ceph: root@ceph-node-00 /]#

AnuragRaut08 · 2026-10-27T16:23:19Z

Verified: `CephOSDFreeSpaceFragmentation` is firing as a warning, with the summary and details of all 6 affected OSDs included in the `message` label.

ceph_health_detail{cluster="8c9bcf82-d13e-11f1-8fb8-52540034b236", instance="ceph_cluster", job="ceph", message="6 OSD(s) experiencing high free space fragmentation of BlueStore. Affected OSDs: osd.0 0.000090, osd.1 0.000089, osd.2 0.000070, osd.3 0.000041, osd.4 0.000035, osd.5 0.000040", name="BLUESTORE_FREE_FRAGMENTATION", severity="HEALTH_WARN"}

AnuragRaut08 · 2026-10-27T16:52:02Z

Feels like we could simplify this a bit. We’re handling cert health checks separately here and skipping them again in the generic health-history loop. Could we define a shared constant for these checks and reuse it in both places? That way, we don’t have to maintain the same alert names in multiple places, and it’s easier to keep things in sync.

I’ve defined a shared HEALTH_CHECKS_WITH_MESSAGE constant and reused it in both places to keep the check names in sync.

AnuragRaut08 · 2026-10-28T03:58:10Z

jenkins test dashboard

AnuragRaut08 · 2026-10-28T04:08:17Z

jenkins test dashboard cephadm

abhidesai6

LGTM

Populate the health detail message label and include affected OSD details for BlueStore fragmentation. Add a dedicated Prometheus alert with warning severity. Fixes: https://tracker.ceph.com/issues/81948 Signed-off-by: Anurag Raut <[email protected]>

AnuragRaut08 · 2026-10-29T06:29:05Z

Thanks @afreen23 Addressed all three suggestions in the latest commit:

Renamed alert_id to health_check.
Added and reused the BLUESTORE_FREE_FRAGMENTATION constant.
Renamed the helper to _process_health_checks_with_message and updated the corresponding tests.

github-actions · 2026-10-29T07:26:26Z

This is an automated message by src/script/redmine-upkeep.py.

I found one or more Fixes: tags in the commit messages in

git log 2c7159f1c5e73fe4a98548abcbd5a00af8e130ce^..2c7159f1c5e73fe4a98548abcbd5a00af8e130ce

The referenced tickets are:

https://tracker.ceph.com/issues/81948

Those tickets do not reference this merged Pull Request. If this Pull Request merge resolves any of those tickets, please update the "Pull Request ID" field on each ticket. A future run of this script will appropriately update them.

Update Log: https://github.com/ceph/ceph/actions/runs/40835290468

github-actions Bot added dashboard monitoring labels Oct 21, 2026

github-project-automation Bot added this to Ceph-Dashboard Oct 21, 2026

github-project-automation Bot moved this to New in Ceph-Dashboard Oct 21, 2026

AnuragRaut08 marked this pull request as ready for review October 21, 2026 05:59

AnuragRaut08 requested a review from a team as a code owner October 21, 2026 05:59

AnuragRaut08 requested review from aaSharma14, abhidesai6, cloudbehl and nizamial09 October 21, 2026 05:59

AnuragRaut08 force-pushed the wip-bluestore-fragmentation-alert branch from 6fcef77 to d1f2186 Compare October 26, 2026 07:58

AnuragRaut08 requested a review from a team as a code owner October 26, 2026 07:58

github-actions Bot added documentation pybind labels Oct 26, 2026

cloudbehl reviewed Oct 26, 2026

View reviewed changes

Comment thread src/pybind/mgr/prometheus/module.py Resolved

Comment thread monitoring/ceph-mixin/prometheus_alerts.libsonnet Resolved

abhidesai6 requested changes Oct 26, 2026

View reviewed changes

Comment thread src/pybind/mgr/prometheus/module.py Resolved

github-project-automation Bot moved this from New to Review in progress in Ceph-Dashboard Oct 26, 2026

AnuragRaut08 force-pushed the wip-bluestore-fragmentation-alert branch from bfaf84d to a089e8c Compare October 26, 2026 18:06

github-actions Bot added the needs-rebase label Oct 26, 2026

abhidesai6 requested changes Oct 27, 2026

View reviewed changes

Comment thread src/pybind/mgr/prometheus/module.py Resolved

Comment thread src/pybind/mgr/prometheus/module.py Resolved

Comment thread src/pybind/mgr/prometheus/module.py Resolved

AnuragRaut08 force-pushed the wip-bluestore-fragmentation-alert branch from a089e8c to 299272a Compare October 27, 2026 11:05

github-actions Bot removed the needs-rebase label Oct 27, 2026

AnuragRaut08 force-pushed the wip-bluestore-fragmentation-alert branch from 299272a to 77e04eb Compare October 27, 2026 12:55

cloudbehl reviewed Oct 27, 2026

View reviewed changes

AnuragRaut08 requested a review from cloudbehl October 27, 2026 16:57

AnuragRaut08 requested a review from abhidesai6 October 27, 2026 16:57

abhidesai6 approved these changes Oct 28, 2026

View reviewed changes

Comment thread src/pybind/mgr/prometheus/module.py Resolved

github-project-automation Bot moved this from Review in progress to Reviewer approved in Ceph-Dashboard Oct 28, 2026

AnuragRaut08 force-pushed the wip-bluestore-fragmentation-alert branch from 95c59f7 to 6418a54 Compare October 28, 2026 10:45

afreen23 approved these changes Oct 28, 2026

View reviewed changes

Comment thread src/pybind/mgr/prometheus/module.py Resolved

Comment thread src/pybind/mgr/prometheus/module.py Resolved

Comment thread src/pybind/mgr/prometheus/module.py Resolved

AnuragRaut08 force-pushed the wip-bluestore-fragmentation-alert branch from 6418a54 to 0b00c7b Compare October 29, 2026 05:06

AnuragRaut08 force-pushed the wip-bluestore-fragmentation-alert branch from 0b00c7b to 6a49ee5 Compare October 29, 2026 05:10

afreen23 merged commit 2c7159f into ceph:main Oct 29, 2026
15 of 16 checks passed

github-project-automation Bot moved this from Reviewer approved to Done in Ceph-Dashboard Oct 29, 2026

AnuragRaut08 mentioned this pull request Oct 29, 2026

umbrella: mgr/dashboard: Batch backport #71344

Open

4 of 14 tasks

Conversation

AnuragRaut08 commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution Guidelines

Checklist

Uh oh!

Uh oh!

Uh oh!

abhidesai6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

AnuragRaut08 commented Oct 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cloudbehl left a comment

Choose a reason for hiding this comment

Uh oh!

abhidesai6 commented Oct 27, 2026

Uh oh!

AnuragRaut08 commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AnuragRaut08 commented Oct 27, 2026

Uh oh!

AnuragRaut08 commented Oct 28, 2026

Uh oh!

AnuragRaut08 commented Oct 28, 2026

Uh oh!

abhidesai6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AnuragRaut08 commented Oct 29, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AnuragRaut08 commented Oct 21, 2026 •

edited

Loading

AnuragRaut08 commented Oct 27, 2026 •

edited

Loading