Repository navigation
HBASE-30231 LoadIncrementalHFiles failed for MetricsException: Metrics source RegionServer,Sub=IO already exists - #8834
Merged
Conversation
wchevreuil
approved these changes
Nov 4, 2026
liuxiaocs7
approved these changes
Nov 4, 2026
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the IO metrics initialization fallback and server-only enablement changes.
No open issues 🎉
What changed in this PR
This PR prevents IO metrics initialization failures from breaking HFile reads and writes. It also limits IO metrics reporting to server processes while preserving lazy initialization.
Changes:
- Fall back to a no-op metrics source when initialization throws a runtime exception or linkage error.
- Enable IO metrics through
HBaseServerBase, covering both RegionServers and Masters. - Add tests for disabled defaults, enablement, failure fallback, and Master-only startup.
| File | Description |
|---|---|
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java |
Verifies that Master startup enables IO metrics. |
hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestMetricsIO.java |
Tests enablement, disabled behavior, and initialization fallback. |
hbase-server/src/main/java/org/apache/hadoop/hbase/io/MetricsIO.java |
Adds server opt-in and a no-op fallback for initialization failures. |
hbase-server/src/main/java/org/apache/hadoop/hbase/HBaseServerBase.java |
Enables IO metrics during shared server construction. |
Review effort: Balanced
PDavid
approved these changes
Nov 5, 2026
anmolnar
added a commit
that referenced
this pull request
Nov 5, 2026
…s source RegionServer,Sub=IO already exists (#8834) MetricsIO.getInstance() lazily registers the "RegionServer,sub=IO" metrics source on the first HFile read or write. Hadoop's MetricsSystemImpl.register() reserves the source name before registration has fully completed, so if anything fails after that point the singleton is never assigned and every later call fails with "Metrics source RegionServer,sub=IO already exists!". This breaks all HFile reads and writes for the rest of the JVM's life, as seen in a bulk load client splitting HFiles in LoadIncrementalHFiles. - If creating the source fails, log a warning once and use a MetricsIO backed by a no-op MetricsIOSource, so a metrics problem can never fail HFile I/O. - Make IO metrics opt-in: MetricsIO.getInstance() returns a shared no-op instance until MetricsIO.enable() is called, which HBaseServerBase does in its constructor, covering both HRegionServer and HMaster (the master reads and writes HFiles through its local MasterRegion). Clients and tools that read or write HFiles (bulk load, MapReduce and Spark tasks, HFilePrettyPrinter, ...) no longer start the Hadoop metrics system or register a RegionServer source. The real source is still created lazily on first use (HBASE-27966), so server behavior is unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Wellington Chevreuil <[email protected]> Signed-off-by: Xiao Liu <[email protected]> Signed-off-by: David Paksy <[email protected]>
anmolnar
added a commit
that referenced
this pull request
Nov 5, 2026
…s source RegionServer,Sub=IO already exists (#8834) MetricsIO.getInstance() lazily registers the "RegionServer,sub=IO" metrics source on the first HFile read or write. Hadoop's MetricsSystemImpl.register() reserves the source name before registration has fully completed, so if anything fails after that point the singleton is never assigned and every later call fails with "Metrics source RegionServer,sub=IO already exists!". This breaks all HFile reads and writes for the rest of the JVM's life, as seen in a bulk load client splitting HFiles in LoadIncrementalHFiles. - If creating the source fails, log a warning once and use a MetricsIO backed by a no-op MetricsIOSource, so a metrics problem can never fail HFile I/O. - Make IO metrics opt-in: MetricsIO.getInstance() returns a shared no-op instance until MetricsIO.enable() is called, which HBaseServerBase does in its constructor, covering both HRegionServer and HMaster (the master reads and writes HFiles through its local MasterRegion). Clients and tools that read or write HFiles (bulk load, MapReduce and Spark tasks, HFilePrettyPrinter, ...) no longer start the Hadoop metrics system or register a RegionServer source. The real source is still created lazily on first use (HBASE-27966), so server behavior is unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Wellington Chevreuil <[email protected]> Signed-off-by: Xiao Liu <[email protected]> Signed-off-by: David Paksy <[email protected]> (cherry picked from commit 242e81e)
anmolnar
added a commit
that referenced
this pull request
Nov 5, 2026
…s source RegionServer,Sub=IO already exists (#8834) MetricsIO.getInstance() lazily registers the "RegionServer,sub=IO" metrics source on the first HFile read or write. Hadoop's MetricsSystemImpl.register() reserves the source name before registration has fully completed, so if anything fails after that point the singleton is never assigned and every later call fails with "Metrics source RegionServer,sub=IO already exists!". This breaks all HFile reads and writes for the rest of the JVM's life, as seen in a bulk load client splitting HFiles in LoadIncrementalHFiles. - If creating the source fails, log a warning once and use a MetricsIO backed by a no-op MetricsIOSource, so a metrics problem can never fail HFile I/O. - Make IO metrics opt-in: MetricsIO.getInstance() returns a shared no-op instance until MetricsIO.enable() is called, which HBaseServerBase does in its constructor, covering both HRegionServer and HMaster (the master reads and writes HFiles through its local MasterRegion). Clients and tools that read or write HFiles (bulk load, MapReduce and Spark tasks, HFilePrettyPrinter, ...) no longer start the Hadoop metrics system or register a RegionServer source. The real source is still created lazily on first use (HBASE-27966), so server behavior is unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Wellington Chevreuil <[email protected]> Signed-off-by: Xiao Liu <[email protected]> Signed-off-by: David Paksy <[email protected]> (cherry picked from commit 242e81e)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's the problem
HFile reads and writes record latency metrics through
MetricsIO. The first time this runs in a process, it registers aRegionServer,sub=IOmetrics source. If that registration fails partway through, the source name stays reserved but theMetricsIOsingleton is never set. Every later HFile read or write then fails for the rest of the JVM's life with:The reporter hit this in a bulk-load client. Full analysis is in HBASE-30231.
What this PR changes
MetricsIOSourcewhen initialization fails.MetricsIO.getInstance()catches the failure once, logs a warning and uses a source that ignores updates. A metrics problem can no longer fail HFile I/O, on clients or servers.MetricsIOnow does nothing untilMetricsIO.enable()is called, whichHRegionServer(and soHMaster) does in its constructor. Bulk-load clients, MapReduce tasks and tools likeHFilePrettyPrinterno longer start the Hadoop metrics system or register a "RegionServer" source. Server behaviour is unchanged: the source is still created on first use, as HBASE-27966 requires.Note
The first change is the bug fix. The second is a small behaviour change: anything writing HFiles without being an
HRegionServerstops reportingRegionServer,sub=IOmetrics. HBase's own code doesn't depend on that, but it's worth a release note.Tests
MetricsExceptionand on aLinkageErrorMetricsIO.enable()switching to the real sourceTestMasterNoCluster.testMasterEnablesMetricsIOstarts an HMaster with no RegionServer in the same JVM and checks that IO metrics are on.Splitting
The two changes are independent, so I'm happy to split them into separate commits and Jira tickets if reviewers prefer. The fallback would stay on HBASE-30231, and the server-only change would go to a new sub-task.
🤖 Generated with Claude Code