Skip to content

Don't treat configured FinalTotalDifficulty as TTD reached - #12555

Merged
kamilchodola merged 12 commits into
masterfrom
fix/final-total-difficulty-ttd-reached
Sep 10, 2026
Merged

kamilchodola merged 12 commits into
masterfrom
fix/final-total-difficulty-ttd-reached

Conversation

@kamilchodola

Copy link
Copy Markdown
Contributor

Changes

  • PoSSwitcher no longer treats a config-declared Merge.FinalTotalDifficulty as proof that the terminal block was reached. All *_archive configs ship this value (for post-merge TD bookkeeping and gossip policy), which made a fresh archive node start with TTD reached: True, disabling the pre-merge PoW sync path (PosForwardHeaderProvider.ShouldUsePreMerge, ImprovementRequirementSatisfied) at genesis — so an archive node could not sync a single block without a consensus client attached.
  • _hasEverReachedTerminalDifficulty is now derived from local chain evidence only: a terminal block persisted in the DB, a post-TTD sync pivot, a merged-at-genesis chain (e.g. Hoodi), or a local best header at/above TTD. The last clause also keeps DBs that crossed TTD under previous versions (which never persisted the terminal block when FinalTotalDifficulty was configured) reporting correctly after restart.
  • TryUpdateTerminalBlock now skips on a non-zero finalized hash (EIP-3675 step 3) instead of TransitionFinished, so a live TTD crossing is recorded, persisted, and raises TerminalBlockReached even with FinalTotalDifficulty in config, while terminal-block reorg updates still freeze once the CL finalizes.
  • FinalTotalDifficulty value semantics are unchanged: gossip disconnect policy, newPayload TD fill, and beacon/fast header TD bookkeeping keep working as before.

With this fix, nethermind -c mainnet_archive with no CL attached full-syncs the pre-merge range (verified live: ~500 Blk/s from devp2p peers, startup line reports TTD reached: False), and parks at the merge boundary until a CL sends forkchoiceUpdated. Merged-at-genesis networks (Hoodi) still require the CL from block 1, and fast-sync pivot behavior is unchanged.

Types of changes

What types of changes does your code introduce?

  • Bugfix (a non-breaking change that fixes an issue)
  • New feature (a non-breaking change that adds functionality)
  • Breaking change (a change that causes existing functionality not to work as expected)
  • Optimization
  • Refactoring
  • Documentation update
  • Build-related changes
  • Other: Description

Testing

Requires testing

  • Yes
  • No

If yes, did you write tests?

  • Yes
  • No

Notes on testing

  • New PoSSwitcherTests cover: config FinalTotalDifficulty alone does not mark TTD reached (while FinalTotalDifficulty/TransitionFinished keep their values); a live crossing with FinalTotalDifficulty configured records + persists the terminal block and raises TerminalBlockReached exactly once, surviving restart; a local chain above TTD without terminal-block metadata still reports reached; merge-at-genesis and post-TTD-pivot init paths are preserved.
  • Full Nethermind.Merge.Plugin.Test and Nethermind.Merge.AuRa.Test suites pass.
  • Live validation on mainnet: stock mainnet_archive config, no CL — node full-syncs from genesis; unfixed binary on the same setup stays parked at genesis forever.

Documentation

Requires documentation update

  • Yes
  • No

Requires explanation in Release Notes

  • Yes
  • No

Archive nodes can now sync the pre-merge block range without a consensus client attached; a CL is only required from the merge block onward.

Remarks

Groundwork for CL-free pre-merge archive sync. Known follow-ups (separate PRs): a clear "waiting for CL" sync state at the merge boundary instead of Disconnected, running beacon-headers sync in parallel with pre-merge full sync for archive nodes, and number-based peer desirability for eth/69 peers (which advertise no total difficulty and are currently invisible to the pre-merge peer selection).

Archive configs ship Merge.FinalTotalDifficulty for post-merge TD
bookkeeping and gossip policy, but PoSSwitcher treated its presence as
"this node has reached the terminal block", disabling the pre-merge
full-sync path at genesis so archive nodes could not sync without a CL.
Derive the flag from local chain evidence instead (persisted terminal
block, post-TTD pivot, merge-at-genesis, or local best header above
TTD), and let TryUpdateTerminalBlock record a live crossing until the
CL finalizes a block.
@claude

claude Bot commented Jul 23, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @kamilchodola's task in 4m 53s —— View job


Review: Don't treat configured FinalTotalDifficulty as TTD reached

  • Read the diff and surrounding code
  • Analyze correctness / edge cases / regressions
  • Review tests
  • Post review

I traced the change through PoSSwitcher, its callers, and the network configs. The fix is correct and well-reasoned, and I verified its central premise against the shipped configs. No blocking issues found — findings are all Low.


What I verified

  • The fix's premise holds. mainnet_archive.json / sepolia_archive.json ship Merge.FinalTotalDifficulty but set no Sync.PivotTotalDifficulty, so HasLocalChainCrossedTerminalTotalDifficulty() correctly returns false at a fresh genesis (genesis TD ≪ TTD, pivot parsed = 0). The base mainnet.json (fast-sync) sets PivotTotalDifficulty ≥ TTD, so it still reports reached — behavior unchanged from before (where _finalTotalDifficulty was derived from the pivot). ✅
  • TerminalBlockReached still fires exactly once. The one-shot guard is the field _hasEverReachedTerminalDifficulty (set only in Initialize/TryUpdateTerminalBlock), while the new dynamic re-evaluation lives only in the method HasEverReachedTerminalBlock(). The early dynamic true never suppresses the event. ✅
  • Reorg-freeze survives restart. The TransitionFinished → _finalizedBlockHash != Keccak.Zero guard change is safe: MetadataDbKeys.FinalizedBlockHash is persisted by BlockTree (BlockTree.cs:1877) and re-loaded in LoadFinalizedBlockHash(), so a finalized post-merge node re-freezes on restart. For non-FTD nodes the guard is functionally identical to before (TransitionFinished reduced to the finalized-hash check); for FTD nodes it now correctly permits terminal-block recording/updates until the CL finalizes. ✅
  • No spurious re-recording. After crossing, subsequent post-merge head blocks are not terminal (IsTerminalBlock is false), so TryUpdateTerminalBlock no-ops until finalization. ✅
  • Tests are meaningful and internally consistent (the TTD=5,000,000 case expects false while TTD=3,000,000 expects true on the same chain-length-4 tree, pinning the crossing between those bounds). ✅

Findings (all Low)

1. Low — Stale comments in related files the fix supersedes. These files (not touched by this PR, correctly keeping the diff minimal) contain comments/workarounds premised on the old buggy behavior, which are now misleading:

  • Nethermind.Merge.AuRa/PoSSwitcherExtensions.cs:12-13 — "that flag is true on a fresh archive DB with Merge.FinalTotalDifficulty in config, even at genesis" — no longer true after this fix.
  • Nethermind.Merge.AuRa.Test/AuRaTerminalBlockDisposerTests.cs:98 — the regression test's premise (HasEverReachedTerminalBlock() true on a fresh archive DB from FTD alone) can no longer arise via that mechanism. The test still passes (it mocks the switcher) and remains valid as defense-in-depth, but the justification comment is outdated.

Not blocking, but a follow-up to refresh those comments (and possibly simplify IsHeadPostMerge back toward HasEverReachedTerminalBlock now that the root cause is gone) would prevent future confusion.

2. Low — Minor logic duplication. The pivot and genesis-difficulty checks in HasLocalChainCrossedTerminalTotalDifficulty() (PoSSwitcher.cs:132-138) mirror the same two conditions in LoadFinalTotalDifficulty() (PoSSwitcher.cs:99-112). They compute different things (a value vs. a bool), so extraction is optional, but the shared predicate could live in one helper.

3. Low — HasEverReachedTerminalBlock() now does more work per call. When _hasEverReachedTerminalDifficulty is false (the entire pre-merge archive sync), each call re-reads BestSuggestedHeader/Head. This is on the sync path (PosForwardHeaderProvider.ShouldUsePreMerge), but the reads are cheap field/property accesses — negligible. Noting only for completeness.


Note (not a regression of this PR)

With FTD configured, TransitionFinished is true, so MergeP2PCapabilityResolver advertises eth/69–71 even while an archive node full-syncs the pre-merge range. This predates the PR (TransitionFinished unchanged) and is explicitly listed in the PR's known follow-ups (eth/69 peer desirability). Flagging only so it isn't lost.

Verdict: correct, focused, tested. Mergeable.

@kamilchodola
kamilchodola marked this pull request as ready for review August 4, 2026 14:28
@claude

claude Bot commented Aug 4, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @kamilchodola's task in 9m 43s —— View job


Review: Don't treat configured FinalTotalDifficulty as TTD reached

  • Read the diff and surrounding code
  • Verify the premise against shipped network configs and chain specs
  • Analyze correctness / edge cases / regressions
  • Review tests
  • Post review

The diagnosis and the core fix are right, and I re-verified the premise from the shipped configs rather than taking it on trust. I found one Medium issue that the previous review pass missed: the TransitionFinished → _finalizedBlockHash guard swap in TryUpdateTerminalBlock was also the only thing preventing genesis from being recorded as the terminal PoW block on merge-at-genesis chains.


What I verified

  • The premise holds. mainnet_archive.json / sepolia_archive.json / gnosis_archive.json / chiado_archive.json are the only four configs that set Merge.FinalTotalDifficulty, and none of them sets Sync.PivotTotalDifficulty — so HasLocalChainCrossedTerminalTotalDifficulty() correctly returns false on a fresh archive genesis. Base mainnet.json (fast sync) sets PivotTotalDifficulty ≥ TTD, so it still reports reached, unchanged. ✅
  • No chain spec sets terminalPoWBlockNumber (checked foundation/sepolia/gnosis/chiado/hoodi), so _terminalBlockExplicitSpecified is false and the fix is actually live on mainnet — this was worth confirming, since a chain-spec terminal block would have made _terminalBlockNumber is not null and left TTD reached: True regardless. ✅
  • No regression for existing post-merge archive DBs. BestSuggestedHeader?.TotalDifficulty ?? Head?.Header.TotalDifficulty cannot silently fall through to null on a synced DB: BlockTree throws InvalidOperationException("Head block with null total difficulty") (BlockTree.cs:1095) rather than setting a head without TD, so the fallback always resolves. An existing mainnet_archive/gnosis_archive DB still reports reached. ✅
  • Reorg-freeze survives restart. MetadataDbKeys.FinalizedBlockHash is written unconditionally by BlockTree.cs:1877 and re-read by LoadFinalizedBlockHash(), so a finalized node re-freezes terminal-block updates after restart. For non-FTD nodes the new guard is functionally identical to the old one. ✅
  • TerminalBlockReached still fires exactly once, and no spurious re-recording: post-merge heads have Difficulty == 0, so IsTerminalBlock is false and the now-unblocked TryUpdateTerminalBlock no-ops on them. ✅
  • Tests exercise the intended clauses. In Local_chain_above_ttd_… (TTD 3,000,000, genesis difficulty 1,000,000, chain length 4 → TD 4,000,000) the genesis clause is false and the assertion genuinely comes from the BestSuggestedHeader clause — not an accidental pass. ✅

Findings

1. Medium — genesis is now recorded and persisted as the terminal PoW block on merge-at-genesis chains (PoSSwitcher.cs:152)

For a chain spec with terminalTotalDifficulty: 0 (Chains/hoodi.json:16, Hive Engine networks, post-merge devnets), LoadFinalTotalDifficulty() always takes the genesis branch (genesisDifficulty >= 0 is trivially true), so TransitionFinished was true from construction and TryUpdateTerminalBlock was a permanent no-op. The new _finalizedBlockHash != Keccak.Zero guard removes that on a fresh DB, and the path is reachable:

MergeBlockNumber == 0 → _terminalBlockNumber stays null → NewHeadBlock is subscribed; PoSSwitcher is constructed in InitializeBlockchain, a declared dependency of LoadGenesisBlock, so the subscription is live before genesis loads; genesis is forced as new head (BlockTree.cs:1089); and IsTerminalBlock(genesis) is true because header.IsGenesis short-circuits the Difficulty != 0 && ParentBeforeTTD() clause.

Result: TerminalPoWNumber = 0 / TerminalPoWHash = <genesis> persisted, and ISpecProvider.MergeBlockNumber becomes 1 instead of 0 — permanently for that DB, silently (no event, just an Updated terminal block log). Current consumers survive by accident only: GetBlockConsensusInfo catches genesis in the earlier IsPostMergeGenesis branch, and BlocksRootContext.GetAccumulatorType is short-circuited by IsEip4895Enabled before it reads MergeBlockNumber. Suggested fix (a one-token guard) and a test suggestion are in the inline comment. Fix this →

2. Low — the predicate and the TerminalBlockReached event are no longer in lockstep (PoSSwitcher.cs:260). On exactly the path the new dynamic clause exists to cover (a crossing where TryUpdateTerminalBlock never runs), HasEverReachedTerminalBlock() goes true but MergeBlockProducerRunner.OnSwitchHappened never stops the pre-merge producer and MergeP2PCapabilityResolver never rebuilds its cache. Both narrow; the capability one is covered by TransitionFinished. The related suggestion-time-vs-processing-time flip I traced and it's harmless.

3. Low — duplication with LoadFinalTotalDifficulty() (PoSSwitcher.cs:126-142). Same two predicates, composed as if/else in one place and || in the other. Plus a nit: _chainSpec?.Genesis is a redundant null check on a non-nullable field (matches existing line 106, so optional).

4. Low — two comments elsewhere in the tree now contradict the code this PR fixes. Keeping the diff minimal is right, but per .agents/rules/coding-style.md ("a comment that contradicts the code is worse than no comment"), these two are now false statements:

  • Nethermind.Merge.AuRa/PoSSwitcherExtensions.cs:12-13 — "that flag is true on a fresh archive DB with Merge.FinalTotalDifficulty in config, even at genesis".
  • Nethermind.Merge.AuRa.Test/AuRaTerminalBlockDisposerTests.cs:109-110 — same premise. The test still passes (it mocks the switcher) and is fine as defence-in-depth; only the justification is stale.

Two comment lines, worth folding into this PR since it is the change that invalidates them. Fix this →


Not blocking / context

  • With FinalTotalDifficulty configured, TransitionFinished is true, so MergeP2PCapabilityResolver advertises eth/69–71 while an archive node full-syncs the pre-merge range. Predates this PR (TransitionFinished is unchanged) and is already listed in the PR's known follow-ups.
  • I could not run the test suite in this job — dotnet test was outside the allowed tool set, so my review of the new tests is by inspection only. The PR reports both Nethermind.Merge.Plugin.Test and Nethermind.Merge.AuRa.Test green.

Verdict: the fix is sound and the live validation is convincing. I'd address finding 1 before merge — it silently writes incorrect merge-transition metadata on a shipped network config (hoodi, hoodi_archive), and the guard is a single added term.
• branch fix/final-total-difficulty-ttd-reached

@flcl42 flcl42 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found 1 medium-severity issue in src/Nethermind/Nethermind.Merge.Plugin/PoSSwitcher.cs.

@AnkushinDaniil AnkushinDaniil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On a merge-at-genesis chain (terminalTotalDifficulty: 0, e.g. hoodi.json) the guard swap from TransitionFinished to _finalizedBlockHash != Keccak.Zero in TryUpdateTerminalBlock drops the old no-op that kept these chains from recording a terminal block. On a fresh DB NewHeadBlock(genesis) then records genesis as the terminal PoW block and MergeBlockNumber moves 0 to 1, persisted. I could not trace a consensus divergence from it (GetBlockConsensusInfo short-circuits on IsPostMergeGenesis, BlocksRootContext on IsEip4895Enabled), so it reads as wrong-persisted metadata rather than a fork. Is skipping IsPostMergeGenesis(header) in that guard the intended shape, and is a regression test driving NewHeadBlock(genesis) on a TTD=0 chain worth adding so MergeBlockNumber stays 0?

@kamilchodola

Copy link
Copy Markdown
Contributor Author

@claude re-review

@claude

claude Bot commented Aug 24, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @kamilchodola's task in 6m 58s —— View job


Re-review: Don't treat configured FinalTotalDifficulty as TTD reached

  • Read the new commit (21dc039) and the full PR diff
  • Check whether the prior Medium findings were addressed
  • Analyze the new locking / latching code for correctness and perf
  • Re-verify the premise against shipped configs and chain specs
  • Post review

21dc039 addresses every open finding from the previous rounds. I re-traced each one rather than taking the commit message on trust. What's left is three Low items — nothing blocking.


Prior findings — all resolved

Finding Raised by Status
Genesis recorded as terminal PoW block on merge-at-genesis chains (MergeBlockNumber 0→1) claude, AnkushinDaniil ✅ IsPostMergeGenesis(header) added to the guard (PoSSwitcher.cs:153) + regression test Post_merge_genesis_is_not_persisted_as_terminal_pow asserting no metadata write and MergeBlockNumber == 0
Init-time local evidence suppresses TerminalBlockReached when the terminal block is recorded later flcl42 ✅ The flag was split: _hasTerminalBlockMetadata (event one-shot) vs _hasLocalChainCrossedTerminalTotalDifficulty (predicate). The event now keys off metadata only, so the local-evidence path no longer swallows it. Covered by Local_ttd_evidence_at_initialization_does_not_suppress_terminal_block_notification
Predicate un-latches when DeleteInvalidBlock resets BestSuggestedHeader wurdum ✅ Latched via _hasLocalChainCrossedTerminalTotalDifficulty |= + Best_suggested_header_regression_does_not_clear_local_ttd_evidence
TryUpdateTerminalBlock unsynchronised once it goes live (two non-atomic Sets, test-then-set on the event) wurdum ✅ _transitionLock around the critical section, event raised outside the lock, NewHeadBlock -= moved outside the lock. Two concurrency tests pin it
Duplicated pivot/genesis predicates claude ✅ Extracted to HasPostTerminalTotalDifficultyPivot / HasPostTerminalTotalDifficultyGenesis, now shared with LoadFinalTotalDifficulty

What I verified this round

  • Premise still holds. mainnet_archive, sepolia_archive, gnosis_archive, chiado_archive are the only configs setting Merge.FinalTotalDifficulty, and none sets Sync.PivotTotalDifficulty → HasLocalChainCrossedTerminalTotalDifficulty() is false at a fresh genesis. No chain spec sets terminalPoWBlockNumber, so the fix is live rather than masked by _terminalBlockExplicitSpecified. ✅
  • The genesis guard covers the real chains. hoodi.json is terminalTotalDifficulty: 0x0 with genesis difficulty 0x01 → IsPostMergeGenesis is true and genesis is rejected. gnosis/chiado genesis difficulty 0x20000 is far below their TTD, so they never took that path. ✅
  • BestSuggestedHeader really does exclude beacon headers (BlockTree.cs:227-238, 300-313 gate on !beaconInsert), so the comment on line 132 is accurate — and DeleteInvalidBlock really does reset it to Head?.Header (BlockTree.cs:784), which is why the latch is needed. ✅
  • No lock inversion or re-entrancy. Everything called under _transitionLock is pure or a DB write — IsTerminalBlock (BlockExtensions.cs:27) and UpdateMergeTransitionInfo (ChainSpecBasedSpecProvider.cs:411) never call back into the switcher. TerminalBlockReached?.Invoke and NewHeadBlock -= are both outside the lock, so subscriber callbacks (MergeP2PCapabilityResolver, MergeBlockProducerRunner, AuRaTerminalBlockDisposer) can't deadlock against it. lock is re-entrant anyway, so the TransitionFinished getter nested inside would be safe regardless. ✅
  • The ForkchoiceUpdatedHandler reorder is safe and load-bearing. Moving _poSSwitcher.ForkchoiceUpdated out of if (shouldUpdateHead) only widens it to the repeated-FCU-with-same-head case; every earlier return path (unknown head, unprocessed head, invalid finalized/safe) already bails before line 283, so a syncing node can't reach it and freeze terminal-block recording prematurely. It matches the already-unconditional _blockTree.ForkChoiceUpdated on the next line and is what the new same_head_finalization_notifies_pos_switcher_and_blocks_terminal_replacement test pins. ✅
  • No AuRa regression. TerminalBlockReached can now fire on gnosis_archive/chiado_archive where it previously could not, so I checked AuRaTerminalBlockDisposer: disposal is idempotent (Interlocked.Exchange), and on the archive full-sync path the event arrives via NewHeadBlock — i.e. after the terminal block is processed and AuRa has finalized it — so it fires no earlier than the old OnBlockProcessing fallback. PosForwardHeaderProvider.OnSuggestBlock only calls TryUpdateTerminalBlock when ShouldProcess is unset, which is the beacon-sync path where the terminal block isn't in range. InitializeBlockchainAuRaMerge keys off IsHeadPostMerge, untouched. ✅
  • No spurious event on restart: a DB with terminal metadata starts with _hasTerminalBlockMetadata == true, so a later reorg update logs Updated terminal block without re-raising — asserted at the end of Reaches_and_persists_terminal_block_with_final_total_difficulty_in_config. ✅

Findings (all Low)

1. Low — HasEverReachedTerminalBlock() recomputes and locks on every call, even once latched (PoSSwitcher.cs:281-289). The latch makes the answer monotonic, but the fast path never reads it: each call does the full recompute and then takes _transitionLock — the same monitor held across two _metadataDb.Set calls. A volatile bool early return keeps the semantics and makes the steady state lock-free. Side effect: it would give the Initialize() assignment on line 85 a purpose (today it's dead, since the first call re-latches the same value).

2. Low — TransitionFinished locks for a single reference read (PoSSwitcher.cs:203-213). It sits on GetBlockConsensusInfo (per header) and MergeGossipPolicy (per gossiped block, network threads). Chains that set FinalTotalDifficulty short-circuit before the lock; chains that don't (devnets, Hive, CLI TTD overrides) take it every time. Volatile.Read(ref _finalizedBlockHash) gives the same ordering guarantee — a reference read is already atomic. The mutual exclusion the tests actually pin (finalization vs. terminal-metadata write) stays in the lock.

3. Low — three comments in the tree now contradict the code. Per .agents/rules/coding-style.md ("a comment that contradicts the code is worse than no comment"), and this PR is the change that invalidates them:

  • Nethermind.Merge.AuRa/PoSSwitcherExtensions.cs:12-13 — "that flag is true on a fresh archive DB with Merge.FinalTotalDifficulty in config, even at genesis". No longer true; IsHeadPostMerge is still the right call, just for a different reason.
  • Nethermind.Merge.AuRa.Test/AuRaTerminalBlockDisposerTests.cs:109-110 — same premise. The test itself stays valid as defence-in-depth (it mocks the switcher); only the justification is stale.
  • Nethermind.Merge.Plugin/MergeP2PCapabilityResolver.cs:33-34 — "HasEverReachedTerminalBlock() (set when TerminalBlockReached fires)". It is now also set from local chain evidence with no event, which is exactly the decoupling this PR introduces.

Fix all three →


Note (not a finding)

TransitionFinished is true whenever FinalTotalDifficulty is configured, so MergeP2PCapabilityResolver still advertises eth/69–71 while an archive node full-syncs the pre-merge range. Unchanged by this PR and already listed in the PR's known follow-ups (eth/69 peer desirability) — flagging only so it isn't lost.

Testing caveat

I could not run the suites in this job — dotnet test is outside the allowed tool set here, so my read of the new tests is by inspection. They look sound: the two concurrency tests use a MemDb subclass that parks inside the first TerminalPoWNumber write and then assert the competing thread is in WaitSleepJoin, which genuinely exercises the lock rather than just passing by timing. The PR reports Nethermind.Merge.Plugin.Test and Nethermind.Merge.AuRa.Test green.

Verdict: mergeable. Correctness findings from the earlier rounds are all closed with tests; the three remaining items are a perf micro-optimisation, a lock-free read, and three comment lines.
• branch fix/final-total-difficulty-ttd-reached

@kamilchodola

Copy link
Copy Markdown
Contributor Author

@AnkushinDaniil @flcl42 Can you re-review?

@kamilchodola

Copy link
Copy Markdown
Contributor Author

@claude re-review

@claude

claude Bot commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @kamilchodola's task in 9m 0s —— View job


Re-review: Don't treat configured FinalTotalDifficulty as TTD reached

  • Read the new commit (3bde008) and the full PR diff
  • Check @flcl42's latest findings against the current code
  • Verify the new provisional-metadata / terminal-selection logic
  • Review tests
  • Post review

3bde008 answers all five of @flcl42's items, and three of the answers are clean wins. Two are not: one narrows IsPostMergeGenesis in a way that regresses Hoodi and weakens a pre-existing master test, and one reverses the latching @wurdum approved. Everything else I re-traced holds up.


@flcl42's findings — status

Finding Status
High — provisional merge height rejects later terminal candidates ✅ Fixed. GetBlockConsensusInfo now tests terminalSelectionOpen && isTerminalBlock before the TransitionFinished && theMergeEnabled branch (PoSSwitcher.cs:263-282). I checked this can't misclassify a PoS block: IsTerminalBlock requires Difficulty != 0 for non-genesis (BlockExtensions.cs:36). Matches EIP-3675 "jump between terminal PoW blocks"
Medium — restart stops observing legal terminal replacements ✅ Fixed. Subscription condition is now !_terminalBlockExplicitSpecified && _finalizedBlockHash == Keccak.Zero (line 88) instead of _terminalBlockNumber is null. ForkchoiceUpdated still unsubscribes exactly once on the zero→non-zero edge, so no leak. Covered end-to-end by restart_with_provisional_terminal_metadata_keeps_observing_new_head_candidates
Low — terminal metadata not crash-atomic ✅ Fixed with a single IWriteBatch (lines 167-171). I verified the key encoding is unchanged: IWriteBatch.Set(long) and IDb.KeyExists(int)/Get(int) both bind to the same long extension in KeyValueStoreExtensions.cs:49-53,145-147 and both go through ToBigEndianSpanWithoutLeadingZeros — (long)MetadataDbKeys.TerminalPoWNumber writes the identical physical key the read path looks up. MemDb.StartWriteBatch() returns a FakeWriteBatch over this, so the concurrency tests' Set override still intercepts mid-batch
Convention — System.Threading.Lock ✅ Applied (line 47)
Medium — TTD zero alone does not make genesis PoS ⚠️ Applied, but see Finding 1 — on the one shipped network it affects this is a regression
High — invalid suggestions become irreversible TTD evidence ⚠️ Applied by removing the latch, which reverses @wurdum's requirement — see Finding 2

Also good: TransitionFinished is now a lock-free Volatile.Read (my prior Low #2), and HasEverReachedTerminalBlock has a lock-free fast path (my prior Low #1).


Findings

1. High — narrowing IsPostMergeGenesis with header.Difficulty == 0 regresses Hoodi (PoSSwitcher.cs:298)

hoodi.json is terminalTotalDifficulty: "0x0" (line 16) with genesis difficulty: "0x01" (line 47). Every other chain-spec TTD-zero network in the tree (surge-hoodi, taiko-hoodi, taiko-alethia) has genesis difficulty 0x0, so Hoodi is the only chain this moves — and it moves in three ways:

  • A TD-less Hoodi genesis header is now classified isPostMerge: false. The master test GetBlockSwitchInfo_returns_post_merge_for_ttd_zero_genesis_without_total_difficulty — which loads the real Hoodi chain spec and the real Hoodi genesis timestamp — asserts it must be true. It exists on master (verified at b1b6fa5) with WithDifficulty(BlockHeaderBuilder.DefaultDifficulty); this commit changed that input to WithDifficulty(0), so the regression test no longer models the network it was written for.
  • A Hoodi genesis header with TD now falls into the new terminalSelectionOpen branch and comes back (isTerminal: true, isPostMerge: false).
  • TryUpdateTerminalBlock records genesis again — this is the Medium finding from the 2026-08-04 round (also raised by @AnkushinDaniil) that 21dc039 closed. On a fresh Hoodi DB, TerminalPoWNumber = 0 is persisted and MergeBlockNumber moves 0 → 1. An already-synced Hoodi DB keeps 0 (its PoS heads have Difficulty == 0, so IsTerminalBlock is false), so two nodes on the same network diverge on persisted merge metadata purely by DB age.

The codebase already treats chain-spec TTD-zero alone as post-merge without consulting genesis difficulty — ChainSpecBasedSpecProvider.IsPostMergeRelease (line 406) and MergeBlockNumber = 0 (line 132). Hoodi has no PoW block; the 0x01 is a Holesky-inherited artifact. The cross-client alignment @flcl42 raised may well be worth doing, but it deserves its own PR where the Hoodi test is changed deliberately rather than adjusted to keep a bundled change green. Fix this →

2. Medium — HasEverReachedTerminalBlock() is non-monotonic again (PoSSwitcher.cs:316)

The un-latched HasCurrentSuggestedTerminalTotalDifficultyEvidence() tail lets the predicate go true → false, and the renamed test now pins that (Best_suggested_header_regression_does_not_latch_local_ttd_evidence, Is.False). @wurdum's approving review asked for the latch because MergeSealer.Address, MergeBlockProducerRunner.StartAsync/StopAsync and MergeP2PCapabilityResolver.Resolve all treat it as one-way — and AlwaysPoS/NoPoS implement it as a constant.

@flcl42's objection is fair (an execution-invalid branch must not permanently kill ShouldUsePreMerge), but both constraints hold if the suggested-header clause is simply dropped and only durable evidence counts. Head crosses TTD one processed batch after the suggestion, which the archive full-sync path tolerates. Otherwise, please state the reversible reading as intentional and rename the member — a HasEver… that un-says itself will be mis-cached.

3. Low — three comments still contradict the code (unchanged from the last round; this PR is what invalidates them):

  • Nethermind.Merge.AuRa/PoSSwitcherExtensions.cs:12-13 — "that flag is true on a fresh archive DB with Merge.FinalTotalDifficulty in config, even at genesis".
  • Nethermind.Merge.AuRa.Test/AuRaTerminalBlockDisposerTests.cs:109-110 — same premise (test itself is fine, it mocks the switcher).
  • Nethermind.Merge.Plugin/MergeP2PCapabilityResolver.cs:33-34 — "HasEverReachedTerminalBlock() (set when TerminalBlockReached fires)"; it is now also derived from local chain evidence with no event.

4. Low — the metadata write batch runs under _transitionLock, which ForkchoiceUpdated also takes on every FCU carrying a finalized hash. Two keys, so the window is tiny, but the DB write could be hoisted out of the lock (compute under the lock, persist after) if you want the engine-API path fully off the storage path.

5. Low — header.IsTerminalBlock(_specProvider) is now computed eagerly at PoSSwitcher.cs:264 even when the TransitionFinished && theMergeEnabled branch is taken. It's a couple of UInt256 compares on a per-header path — negligible, noting only because the old code deferred it.


Testing caveat

dotnet build / dotnet test are outside the allowed tool set in this job, so my read of the new tests is by inspection. Finding 1 in particular is a claim about PoSSwitcherTests.GetBlockSwitchInfo_returns_post_merge_for_ttd_zero_genesis_without_total_difficulty — I verified it by diffing against b1b6fa5 rather than by running it. Worth confirming locally that reverting IsPostMergeGenesis keeps the whole Nethermind.Merge.Plugin.Test suite green.

Verdict

Not mergeable as-is. Finding 1 is a silent, persisted behaviour change on hoodi/hoodi_archive that also weakens an existing master regression test, and it reintroduces a defect two reviewers had already signed off as closed. Finding 2 is a reviewer conflict that needs an explicit decision rather than an implicit one. The rest of 3bde008 — batch atomicity, the Lock type, the volatile reads, the replaceable-terminal fix and the restart resubscription — is solid.
• branch fix/final-total-difficulty-ttd-reached

@kamilchodola

Copy link
Copy Markdown
Contributor Author

@claude re-review

@claude

claude Bot commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @kamilchodola's task in 7m 5s —— View job


Re-review: Don't treat configured FinalTotalDifficulty as TTD reached

  • Read the new commit (ecee9ee) and the full PR diff
  • Check whether the prior High/Medium findings were addressed
  • Verify the premise against shipped configs and chain specs
  • Review tests
  • Post review

ecee9ee closes both findings from the last round, and it picked the resolution I'd argued for on the reviewer conflict rather than papering over it. I re-traced each one against the code instead of trusting the commit message. What's left is four Low items, none of which change behaviour.


Prior findings — status

Finding Raised by Status
High — header.Difficulty == 0 in IsPostMergeGenesis regresses Hoodi (genesis persisted as terminal PoW, MergeBlockNumber 0→1, master test weakened) claude ✅ Reverted to header.IsGenesis && chainSpec TTD.IsZero (line 290). PoSSwitcherTests.cs:67 restored to WithDifficulty(BlockHeaderBuilder.DefaultDifficulty) and Positive_difficulty_genesis_is_terminal_pow_when_chain_spec_ttd_is_zero removed. I re-confirmed hoodi.json is terminalTotalDifficulty: "0x0" / genesis difficulty: "0x01", so IsPostMergeGenesis(genesis) is true again and the TryUpdateTerminalBlock guard (line 154) rejects genesis — the defect @AnkushinDaniil and I flagged on 08-04 stays closed
Medium — predicate non-monotonic again claude ✅ HasCurrentSuggestedTerminalTotalDifficultyEvidence() deleted; HasEverReachedTerminalBlock() is now durable-only + latched, so it can no longer go true → false. This satisfies both @wurdum's one-way-latch requirement (MergeSealer.Address, MergeBlockProducerRunner, MergeP2PCapabilityResolver, and the AlwaysPoS/NoPoS constants all read it as "ever") and @flcl42's objection — an execution-invalid branch can no longer poison the latch, because BestSuggestedHeader is not consulted at all. Pinned by Best_suggested_header_is_not_durable_ttd_evidence
Low — three stale comments claude ✅ All three reworded (PoSSwitcherExtensions.cs:12-13, AuRaTerminalBlockDisposerTests.cs:109, MergeP2PCapabilityResolver.cs:32-34)
High/Medium/Low items from the 08-24 round (provisional-height rejection, restart resubscription, batch atomicity, Lock, volatile reads) flcl42, claude ✅ Still in place and unchanged by ecee9ee

What I verified this round

  • Dropping BestSuggestedHeader does not regress the case the PR exists for. The archive full-sync crossing is recorded through NewHeadBlock → TryUpdateTerminalBlock, which sets _hasTerminalBlockMetadata at the same moment Head crosses TTD — so the durable clause and the metadata clause flip together. The suggested-header clause only ever bought one batch of latency for PosForwardHeaderProvider.ShouldUsePreMerge(), and the terminal block is already in the in-flight batch at that point. ✅
  • No regression for existing post-merge DBs. BlockTree sets Head inside its own constructor (LoadStartBlock() at BlockTree.cs:166, Head ??= FindBlock(genesis) at 167), and PoSSwitcher is constructed later in InitializeBlockchain — so Head is populated when Initialize() reads it, and the advertised TTD reached: startup line is accurate rather than accidentally False. On a fresh DB Head is null → False, which is the intent. ✅
  • Config premise re-confirmed at HEAD. mainnet_archive, sepolia_archive, gnosis_archive, chiado_archive are still the only four configs setting Merge.FinalTotalDifficulty, and none sets Sync.PivotTotalDifficulty; mainnet.json/sepolia.json/gnosis.json set PivotTotalDifficulty ≥ TTD so fast-sync nodes still report reached. Hoodi has neither and is covered by the genesis clause (1 >= 0). ✅
  • The GetBlockConsensusInfo reorder is behaviour-preserving except where intended. (TransitionFinished || _terminalBlockExplicitSpecified) && theMergeEnabled is a faithful rewrite of the base branch, terminalSelectionOpen is false whenever _terminalBlockExplicitSpecified is set, and the new precedence branch only fires for headers where IsTerminalBlock is true — which requires Difficulty != 0 for non-genesis (BlockExtensions.cs:37), so no valid PoS block can reach it. FilterPosHeader's merge-boundary cut is unaffected (post-merge headers hit the TotalDifficulty is null / !isTerminal paths exactly as before). ✅
  • The write batch really is intercepted by the concurrency tests. MemDb.StartWriteBatch() → this.LikeABatch() → FakeWriteBatch, whose Set forwards to the same MemDb instance, so BlockingTerminalMetadataDb's virtual Set(ReadOnlySpan<byte>, …) override still parks inside the batch. Key encoding matches the read path — Set(long) and KeyExists(int)/Get(int) both go through ToBigEndianSpanWithoutLeadingZeros (KeyValueStoreExtensions.cs:49,53,145). ✅
  • ForkchoiceUpdatedHandler widening is still safe. Every syncing/invalid path returns before line 283 (!blockInfo.WasProcessed → 228, finalized/safe errors → 236/242, terminal-block check → 251, ancestry → 259/260/264, failed TryUpdateMainChain → 275), so a syncing node cannot reach _poSSwitcher.ForkchoiceUpdated and freeze terminal selection early. ✅
  • Concurrency tests are deterministic where they claim to be: the second candidate always writes last (it blocks on _transitionLock while the first is parked inside the batch), so MatchesTerminalBlock(…, secondTerminalBlock) isn't timing-dependent. ✅

Findings (all Low)

1. Low — _terminalBlockHash is now write-only (PoSSwitcher.cs:166). The batch refactor swapped Rlp.Encode(_terminalBlockHash) for Rlp.Encode(header.Hash), removing the field's only read; it's now assigned twice and read nowhere, which also makes the LoadHashFromDb(MetadataDbKeys.TerminalPoWHash) call at startup dead work. Per AGENTS.md ("if a change makes existing code unused, remove it") either encode from the field again or drop the field and its load — the persisted key itself must keep being written.

2. Low — the metadata write batch runs under _transitionLock (lines 159-163), which ForkchoiceUpdated also takes on every FCU carrying a finalized hash. Two keys, so the window is tiny, but the RocksDB write could be hoisted out (compute under the lock, persist after) to keep the engine-API path off the storage path entirely. Unchanged from the last round.

3. Low — Volatile in the constructor is noise. Lines 85-88 and LoadFinalizedBlockHash (145) run only from the constructor, where no other thread can observe the instance. The Volatile.Writes there buy nothing and read as if concurrent publication were possible. The ones in TryUpdateTerminalBlock/ForkchoiceUpdated/HasEverReachedTerminalBlock/TransitionFinished are load-bearing and correct.

4. Low — header.IsTerminalBlock(_specProvider) is computed eagerly at line 256 even when the TransitionFinished && theMergeEnabled branch is taken. A couple of UInt256 compares on a per-header path — negligible; noting only because the old code deferred it.


Notes (not findings)

  • Can_load_parameters_after_the_restart was deleted and its assertions folded into Reaches_and_persists_terminal_block_with_final_total_difficulty_in_config (which now covers the same restart + MergeBlockNumber recovery, plus persistence and the no-duplicate-event check). Coverage looks preserved — flagging only so the deletion is a conscious call.
  • On a fresh archive DB HasEverReachedTerminalBlock() is now false, so MergeBlockProducerRunner.StartAsync / MergeSealer.Address restore genuine pre-merge behaviour instead of the post-merge shortcut. That's the intended consequence of the fix and only reachable with mining/validating enabled (HasPreMergeProducerRunner), so mainnet_archive is unaffected by default.
  • TransitionFinished is still true whenever FinalTotalDifficulty is configured, so MergeP2PCapabilityResolver advertises eth/69–71 while an archive node full-syncs the pre-merge range. Unchanged by this PR and already listed in the known follow-ups.

Testing caveat

dotnet build / dotnet test are outside the allowed tool set in this job, so my read of the tests is by inspection — including that reverting IsPostMergeGenesis leaves GetBlockSwitchInfo_returns_post_merge_for_ttd_zero_genesis_without_total_difficulty and Post_merge_genesis_is_not_persisted_as_terminal_pow both green (they should: the former's genesis now has DefaultDifficulty and hits the IsPostMergeGenesis branch, the latter's has Difficulty 0 and is rejected by the guard either way). I did verify the new test helpers exist — CreateBlockchain(…, configurer:) at BaseEngineModuleTests.cs:66-71, chain.PoSSwitcher at TestBlockchain.cs:87, AddBlockThroughPoW. Worth a local run of Nethermind.Merge.Plugin.Test + Nethermind.Merge.AuRa.Test before merge.

Verdict

Mergeable. Every Critical/High/Medium item raised across the four review rounds — mine, @flcl42's, @AnkushinDaniil's and @wurdum's — is now closed with a test pinning it, and the two conflicting requirements on HasEverReachedTerminalBlock() are reconciled rather than traded off. The four remaining items are a dead field, a lock-scope tightening, constructor-only volatiles, and an eager two-compare predicate.
• branch fix/final-total-difficulty-ttd-reached

}

_terminalBlockNumber = header.Number;
_terminalBlockHash = header.Hash;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low — _terminalBlockHash is now write-only.

The write-batch refactor replaced Rlp.Encode(_terminalBlockHash) with Rlp.Encode(header.Hash) (line 162), which removed the field's only read. It is now assigned in exactly two places (here and LoadTerminalBlock line 333) and read nowhere — ConfiguredTerminalBlockHash returns _mergeConfig.TerminalBlockHashParsed directly, not this field. That also makes the LoadHashFromDb(MetadataDbKeys.TerminalPoWHash) call at startup dead work.

AGENTS.md: "If a change makes existing code unused, remove it." Two options, both fine:

  • Minimal: encode from the field again so the read exists, matching the _terminalBlockNumber handling right below.
    batch.Set((long)MetadataDbKeys.TerminalPoWHash, Rlp.Encode(_terminalBlockHash).Bytes);
    (with the assignment hoisted above the batch, as _terminalBlockNumber used to be)
  • Or drop the field entirely along with the TerminalPoWHash branch of LoadTerminalBlock. Note the persisted key must stay written either way — the tests read it, and it is the only record of which candidate was selected.

Fix this →

@kamilchodola

Copy link
Copy Markdown
Contributor Author

@claude re-review

@claude

claude Bot commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @kamilchodola's task in 5m 9s —— View job


Re-review: Don't treat configured FinalTotalDifficulty as TTD reached

  • Read the new commit (d5466ef) and the full PR diff at HEAD
  • Check whether the prior findings were addressed
  • Re-verify correctness / edge cases against configs, chain specs and BlockTree
  • Review tests
  • Post review

The only commit since my last review is d5466ef, which is whitespace-only (git diff ecee9ee HEAD is a single brace/newline reflow in PoSSwitcherTests.StartOperation). So the previous verdict stands: every Critical/High/Medium item raised across the five review rounds is closed with a test pinning it. I re-traced the code at HEAD rather than relying on that, and I closed one gap I had only reasoned about before.


Prior findings — status at HEAD

Finding Raised by Status
High — header.Difficulty == 0 in IsPostMergeGenesis regressed Hoodi claude ✅ Reverted (PoSSwitcher.cs:289-290), test input restored
High — provisional merge height rejects later terminal candidates flcl42 ✅ terminalSelectionOpen && isTerminalBlock takes precedence (PoSSwitcher.cs:257-265)
High — invalid suggestions become irreversible TTD evidence flcl42 ✅ BestSuggestedHeader dropped from the predicate entirely
Medium — predicate non-monotonic / one-way-latch contract claude, wurdum ✅ Durable-only + latched (PoSSwitcher.cs:295-309)
Medium — genesis recorded as terminal PoW on merge-at-genesis chains claude, AnkushinDaniil ✅ IsPostMergeGenesis(header) in the guard (PoSSwitcher.cs:154)
Medium — restart stops observing legal terminal replacements flcl42 ✅ Subscription keyed on !explicit && finalizedHash == Zero (PoSSwitcher.cs:88)
Medium — init-time evidence suppresses TerminalBlockReached flcl42 ✅ Flag split into _hasTerminalBlockMetadata / _hasLocalChainCrossedTerminalTotalDifficulty
Medium — unsynchronised TryUpdateTerminalBlock wurdum ✅ _transitionLock, event raised outside the lock
Low — batch atomicity, System.Threading.Lock, volatile reads, dedup, stale comments flcl42, claude ✅ All applied

New verification this round

  • The one gap I had only argued by reasoning is now confirmed closed. My concern with the durable-only predicate was a head that jumps past the terminal block, leaving HasEverReachedTerminalBlock() true via Head while no terminal metadata is ever written. BlockTree.TryUpdateMainChain evaluates head improvement progressively, per header inside the batch (BlockTree.cs:1088-1101) and raises NewHeadBlock for every header flagged IsNewHead (BlockTree.cs:1124-1128) — not just the batch tail. So during archive full sync the terminal block itself always raises the event and TryUpdateTerminalBlock always records it. The Head clause and the metadata clause genuinely flip together. ✅
  • The GetBlockConsensusInfo rewrite is behaviour-preserving off the new branch. Master's TransitionFinished && theMergeEnabled || _terminalBlockExplicitSpecified && theMergeEnabled (verified at b1b6fa5) is logically identical to the new (TransitionFinished || _terminalBlockExplicitSpecified) && theMergeEnabled. And the new branch only diverges from master when MergeBlockNumber is already set from a dynamically recorded terminal block and the header is a genuine competing terminal candidate — exactly EIP-3675's jump-between-terminal-PoW-blocks window. On a fresh fast-sync mainnet.json DB (TransitionFinished true via pivot, MergeBlockNumber null → theMergeEnabled false) both old and new code reach the same result. ✅
  • No leak into TTD-zero / non-PoW chains. IsTerminalBlock requires Difficulty != 0 for non-genesis (BlockExtensions.cs:37), and genesis is caught by the earlier IsPostMergeGenesis branch, so Taiko/Surge/Hoodi headers can never reach the new precedence branch. ✅
  • Write-batch key encoding matches the read path. batch.Set((long)MetadataDbKeys.TerminalPoWNumber, …) binds to Set(long, byte[]) on IWriteOnlyKeyValueStore (KeyValueStoreExtensions.cs:94,145) — IWriteBatch : IWriteOnlyKeyValueStore (IWriteBatch.cs:8) — and KeyExists(int)/Get(int) widen to the long overloads (lines 49, 53). Both go through ToBigEndianSpanWithoutLeadingZeros, and TerminalPoWNumber = 2 (MetadataDbKeys.cs:9) encodes to the single byte the test's IsTerminalNumberKey matches. No silent key divergence. ✅
  • Widened NewHeadBlock subscription is safe. It now stays live until first finalization even when terminal metadata exists. Extra cost is one _transitionLock acquire + two UInt256 compares per new head; post-merge heads have Difficulty == 0 so TryUpdateTerminalBlock no-ops. Unsubscribe still happens exactly once, on the zero→non-zero _finalizedBlockHash edge (PoSSwitcher.cs:195-205), so no double-unsubscribe and no dangling handler beyond the pre-existing never-gets-an-FCU case. ✅
  • ForkchoiceUpdatedHandler widening still safe. Every syncing/invalid path returns before line 285 (Syncing at 227, finalized/safe errors at 236/242, terminal check at 253, ancestry at 261/262/265, failed TryUpdateMainChain at 275), so a syncing node cannot freeze terminal selection early. ✅
  • Field-level thread safety. _terminalBlockNumber / _firstPoSBlockNumber / _terminalBlockHash are written only under _transitionLock or in the constructor, and read only there; _terminalBlockExplicitSpecified and _finalTotalDifficulty are constructor-only. No unguarded shared mutable state. ✅

Findings (all Low, all carried over unchanged — d5466ef is whitespace)

1. Low — _terminalBlockHash is write-only. Confirmed at HEAD by grep: the only occurrences are the declaration (PoSSwitcher.cs:48) and two assignments (:166, :333). The IWriteBatch refactor encodes header.Hash directly (:162), which removed the field's last read, and that also makes the LoadHashFromDb(MetadataDbKeys.TerminalPoWHash) call at startup dead work. Per AGENTS.md ("if a change makes existing code unused, remove it"): either encode from the field again or drop the field plus that load. The persisted TerminalPoWHash key must keep being written either way. Already raised inline (r3864062937). Fix this →

2. Low — the metadata write batch runs under _transitionLock (PoSSwitcher.cs:159-163), the same monitor ForkchoiceUpdated takes on every FCU carrying a finalized hash. Two keys, so the window is tiny, but the RocksDB write could be hoisted out (decide under the lock, persist after) to keep the engine-API path off the storage path.

3. Low — constructor-only Volatile.Writes are noise. Lines 85-86 and LoadFinalizedBlockHash (145) run only from the constructor, where no other thread can observe the instance. Related nit: line 86's latch is redundant now that HasEverReachedTerminalBlock() self-latches — the first call would compute the same value. Harmless (it makes the startup log line deterministic), but it reads as if concurrent publication mattered. The volatiles in TryUpdateTerminalBlock / ForkchoiceUpdated / HasEverReachedTerminalBlock / TransitionFinished are load-bearing and correct.

4. Low — header.IsTerminalBlock(_specProvider) is computed eagerly at PoSSwitcher.cs:256 even when the TransitionFinished && theMergeEnabled branch is taken. A couple of UInt256 compares on a per-header path — negligible; noting only because the old code deferred it.

Notes (not findings)

  • TransitionFinished is still true whenever FinalTotalDifficulty is configured, so MergeP2PCapabilityResolver advertises eth/69–71 while an archive node full-syncs the pre-merge range. Unchanged by this PR and already listed in the known follow-ups.
  • On a fresh archive DB HasEverReachedTerminalBlock() is now false, so MergeBlockProducerRunner.StartAsync / MergeSealer.Address restore genuine pre-merge behaviour. Intended, and only reachable with mining/validating enabled.

Testing caveat

dotnet build and dotnet test are outside the allowed tool set in this job (both were denied), so my read of the tests is by inspection only — including that d5466ef's reflow leaves PoSSwitcherTests compiling. The PR reports Nethermind.Merge.Plugin.Test and Nethermind.Merge.AuRa.Test green; worth one local run before merge since the last two commits have not been re-run here.

Verdict

Mergeable — no Critical/High/Medium findings. The four remaining items are a dead field, a lock-scope tightening, constructor-only volatiles, and an eager two-compare predicate; none changes behaviour. The dead field (finding 1) is the only one that touches a repo rule, and it's a two-line fix if you want it in this PR.
• branch fix/final-total-difficulty-ttd-reached

@flcl42 flcl42 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current-state re-review across Ethereum, Gnosis/AuRa, and Taiko. Taiko production wiring bypasses PoSSwitcher via AlwaysPoS; the Ethereum transition logic is otherwise improved, but three production restart/AuRa defects and several regression gaps remain. I did not duplicate the existing open _terminalBlockHash thread.

return true;

// Head is processed and durable, so it remains valid evidence after restart.
return _blockTree.Head?.Header.TotalDifficulty >= terminalTotalDifficulty;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium — a committed terminal head can be latched without ever reconstructing its metadata. BlockTree.UpdateMainChainCore commits Head before raising NewHeadBlock, so a crash in that window leaves a terminal head with no TerminalPoW* keys. On restart this returns true, but initialization only subscribes to future heads; the next PoS head is not terminal, leaving MergeBlockNumber absent permanently and causing consumers such as BlockForRpc to keep treating post-merge blocks as pre-merge. Startup reconciliation of an existing terminal head, covered by a commit-before-event restart test, would close this window.

Volatile.Write(ref _hasLocalChainCrossedTerminalTotalDifficulty, HasDurableTerminalTotalDifficultyEvidence());

if (_terminalBlockNumber is null)
if (!_terminalBlockExplicitSpecified && Volatile.Read(ref _finalizedBlockHash) == Keccak.Zero)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High — this persisted hash does not necessarily represent an applied forkchoice. The unknown-head FCU path calls BlockTree.ForkChoiceUpdated and returns SYNCING without notifying PoSSwitcher; that writes the same FinalizedBlockHash key. After a crash, this condition treats the sync hint as a validated POS_FORKCHOICE_UPDATED, suppressing terminal tracking and freezing a provisional candidate although the identical in-process state remains open. With candidate A already setting MergeBlockNumber, a later canonical terminal candidate can then be classified as PoS and rejected. A separate persisted validated-FCU marker, or validation of the stored hash before closing selection, keeps restart behavior aligned with the Engine API result.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mechanism is exactly right, and I traced it independently: ForkchoiceUpdatedHandler.cs:155 (the unresolvable-head SYNCING path) calls _blockTree.ForkChoiceUpdated, which persists MetadataDbKeys.FinalizedBlockHash unconditionally at BlockTree.cs:1877, while the only _poSSwitcher.ForkchoiceUpdated call site is line 280 on the applied path. So a sync hint and a validated FCU are indistinguishable after restart.

Where I land differently is on whether this PR introduces it. On the merge base b1b6fa58a7, TryUpdateTerminalBlock was already guarded on TransitionFinished, defined as FinalTotalDifficulty is not null || _finalizedBlockHash != Keccak.Zero, with _finalizedBlockHash loaded from that same key — so a SYNCING-persisted hash already froze terminal recording, and base line 200 (TransitionFinished && theMergeEnabled) already classified a later terminal candidate as PoS in exactly this state. This PR narrows the freeze (metadata alone no longer closes selection) and leaves the finalized-hash case byte-for-byte as master.

On the suggested remedy: "validate the stored hash" can't work by provenance, because the key is not Engine-API-exclusive. AuRaBlockFinalizationManager.cs:133 and both EraImporters (Nethermind.Era1, Nethermind.EraE) call BlockTree.ForkChoiceUpdated with no CL involved, so a stored hash carries no evidence of who wrote it. A separate persisted validated-FCU marker is the workable option, but that is a new metadata key plus a new invariant shared with StartingSyncPivotUpdater, which reads the same state — I'd rather do that as its own PR than bolt it onto this one.

Happy to open that follow-up if you agree it's worth it; I don't think it should block this change, since the state it describes is unchanged from master.

_specProvider.UpdateMergeTransitionInfo(_firstPoSBlockNumber.Value);

Volatile.Write(ref _hasLocalChainCrossedTerminalTotalDifficulty, true);
shouldNotifyTerminalBlockReached = !Volatile.Read(ref _hasTerminalBlockMetadata);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High — the first provisional candidate now irreversibly disables AuRa finalization on archive Gnosis. AuRaTerminalBlockDisposer handles this first emission by permanently detaching AuRaBlockFinalizationManager, while EIP-3675 still permits a competing terminal candidate until FIRST_FINALIZED_BLOCK. A replacement Gnosis branch containing a finalized validator-contract change then never updates ValidatorStore, so a subsequent block from the valid new sealer set is rejected against stale validators. AuRa finalization must remain available for pre-merge replacement processing and retire only once terminal selection is actually frozen or post-merge processing begins.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on the hazard, but "now" is the part I can't reproduce — the disposal on the first provisional candidate predates this PR.

On the merge base b1b6fa58a7, PoSSwitcher already raised TerminalBlockReached on the first recorded candidate (if (!_hasEverReachedTerminalDifficulty)) and never unsubscribed from NewHeadBlock, so on any AuRa/Gnosis node without Merge.FinalTotalDifficulty configured, master disposes AuRaBlockFinalizationManager at the first provisional candidate exactly as this branch does. AuRaTerminalBlockDisposer is one-shot by construction (Interlocked.Exchange + Dispose(), lines 66-73) and this PR does not touch it — git diff b1b6fa58a7..HEAD over Nethermind.Merge.AuRa/ is two comments.

The one behavioural delta is on FTD-configured archives: master set the reached-flag from _finalTotalDifficulty at init and then returned false from TryUpdateTerminalBlock forever, so the event never fired and AuRa was instead disposed one block later at the first post-merge block via OnBlockProcessing. That suppression is the bug this PR fixes, and the terminal block is the disposer's designed trigger — so the change moves the disposal one block earlier on that config, from "first PoS block" to "terminal block".

Making AuRa finality survive a pre-merge reorg past a terminal candidate means making the disposer re-armable, which re-opens the double-writer race into BlockTree.ForkChoiceUpdated that the disposer's own remarks (lines 15-24) say it exists to prevent. That's a real design question for archive Gnosis and I'd rather not answer it inside this PR.

So: not disputing the scenario, disputing that it's new here. If you'd still like the retirement point moved to first-finalization, say so and I'll open it separately with the double-writer question addressed properly.


if (terminalSelectionOpen && isTerminalBlock)
{
// Until the first finalized PoS block, EIP-3675 permits a later qualifying terminal PoW block.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spec wording — the freezing point is not necessarily a PoS block. EIP-3675 closes terminal selection at FIRST_FINALIZED_BLOCK, the first non-zero finalized hash; this PR's same-head test even finalizes the terminal PoW hash. Using the spec term here keeps the comment aligned with the condition and its coverage.

Assert.That(poSSwitcher.GetBlockConsensusInfo(alternativeTerminalBlock.Header), Is.EqualTo((true, false)));

Block laterTerminalBlock = Build.A.Block
.WithTotalDifficulty(5000000L)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test gap — this later-height candidate is internally invalid. Its parent is already the TD=5,000,000 terminal block, so a difficulty-1,000,000 child must have TD=6,000,000. Production HeaderValidator rejects this fixture before the switcher result matters. A valid competing branch with its own pre-TTD parent, imported through the Merge/BlockTree validation path, would exercise the regression.

public void Best_suggested_header_is_not_durable_ttd_evidence()
{
TestSpecProvider specProvider = new(London.Instance) { TerminalTotalDifficulty = 5000000 };
IBlockTree blockTree = Substitute.For<IBlockTree>();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test-infrastructure rule — this lifecycle regression bypasses the production block tree. Mutating an IBlockTree substitute only proves that the current method ignores a property; it does not exercise the real suggestion-to-invalidation/reorg lifecycle, and the repository rule excludes mocked components supplied by TestBlockchain. A DI-backed MergeTestBlockchain scenario would retain constructor event wiring and invalidation behavior in the regression path.

}
}

private sealed class BlockingTerminalMetadataDb : MemDb

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test gap — the new write-batch guarantee is not exercised. MemDb.StartWriteBatch() returns a write-through FakeWriteBatch, so this fixture pauses inside the first Set and proves only outer-lock serialization; reverting production to two direct puts while retaining the lock would still pass. A buffered batch spy or abort-and-restart test would prove that neither terminal key becomes visible after an inter-key failure.

@@ -1,16 +1,21 @@
// SPDX-FileCopyrightText: 2022 Demerzel Solutions Limited

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Project convention — the significant changes retain outdated SPDX years. This substantially expanded test file and the production PoSSwitcher.cs change still carry 2022; the repository convention calls for 2026 on files changed at this scale.

kamilchodola and others added 4 commits August 28, 2026 13:05
BlockTree commits the head hash before raising NewHeadBlock, so a crash in that
window leaves a terminal head whose TerminalPoW* keys were never written. Nothing
announces that head again and its successors are not terminal, so the merge block
number stayed unknown for the lifetime of the database. Reconcile it from the head
at startup while terminal selection is still open.
The write batch encodes header.Hash directly, which left _terminalBlockHash (and the
TerminalPoWHash load behind it) assigned but never read. The persisted key is still
written. Also names the EIP-3675 freezing point FIRST_FINALIZED_BLOCK, which need not
be a PoS block.
The later-height terminal candidate now sits on a competing lower-difficulty branch,
so its total difficulty matches its own parent instead of being unreachable behind the
first terminal block. The best-suggested-header regression runs over the real block tree
(suggest, then TryUpdateMainChain) rather than a mutated IBlockTree substitute. Adds a
batch-tracking metadata double so both terminal keys are proven to be written inside one
write batch - MemDb's write batch is write-through, so the concurrency tests could not
show that.
One UpdateMainChain commits a whole processing branch and raises its
NewHeadBlock events afterwards, so a crash in that window can leave a
durable head several blocks past the terminal block - which is not itself
terminal, so probing only the head recovered nothing. Walk back from the
head while the header is at or above the terminal total difficulty, bounded
so a misconfigured TTD cannot scan the chain.

Also makes two tests mean what they claim: the write-batch tracker now
attributes a write by the object it arrives through rather than by 'a batch
is open' (two direct puts inside an open batch used to pass), and the
best-suggested-header test records which branch actually answers it.
@kamilchodola

Copy link
Copy Markdown
Contributor Author

@claude re-review

@claude

claude Bot commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @kamilchodola's task in 9m 35s —— View job


Re-review: Don't treat configured FinalTotalDifficulty as TTD reached

  • Read the four new commits since the last review (56aee55, 74d1a84, 8a65550, e88a4d6)
  • Check @flcl42's latest findings against the current code
  • Verify the new startup metadata-reconstruction logic
  • Review tests
  • Post review

The four new commits close the remaining open threads, and the walk-back in e88a4d6 is the right shape — I verified its premise in BlockTree rather than taking the commit message on trust. I found one Medium issue that no round has covered: the IsPostMergeGenesis guard only closes the chain-spec half of the genesis-as-terminal defect, and the MergeConfig half is reachable (and is what every MergeTestBlockchain runs on).


Prior findings — status

Finding Raised by Status
Medium — a committed terminal head can be latched without its metadata ever being reconstructed flcl42 ✅ ReconcileTerminalBlockFromHead() at startup (PoSSwitcher.cs:164), + Committed_terminal_head_is_recorded_when_terminal_metadata_is_missing and Terminal_block_behind_a_committed_head_is_recovered_by_walking_back
Low — _terminalBlockHash write-only claude ✅ Field and its LoadHashFromDb(TerminalPoWHash) load removed; the persisted key is still written (:205)
Spec wording — freezing point is FIRST_FINALIZED_BLOCK, not "first finalized PoS block" flcl42 ✅ :304
Test gap — later-height terminal candidate was internally invalid flcl42 ✅ It now sits on a competing branch with its own pre-TTD parent, and the test asserts TD == parentTD + difficulty so the fixture can't silently drift again
Test gap — write-batch guarantee not exercised (FakeWriteBatch is write-through) flcl42 ✅ BatchTrackingTerminalMetadataDb attributes writes by the object they arrive through, so two direct puts inside an open batch would now fail
High — persisted FinalizedBlockHash may be a SYNCING hint flcl42 🗣 Answered by @kamilchodola: unchanged from b1b6fa58a7, and the key is not Engine-API-exclusive (AuRaBlockFinalizationManager, both EraImporters write it), so provenance validation can't work — deferred to a follow-up. Reads as a fair call to me
High — first provisional candidate disables AuRa finalization flcl42 🗣 Answered by @kamilchodola: git diff b1b6fa58a7..HEAD -- Nethermind.Merge.AuRa/ is two comments, and master already disposes on the first candidate for non-FTD AuRa nodes. I re-checked AuRaTerminalBlockDisposer — one-shot Interlocked.Exchange, untouched. Deferred

What I verified this round

  • The walk-back's premise is exactly right. SetHeadBlock is called inside the ChainLevelInfoRepository batch (BlockTree.cs:1101) and its own comment says it "mutates Head and writes the head hash without raising NewHeadBlock" (:1441-1443); the deferred events run after the batch is disposed (:1113-1134). So a crash in that window does leave a committed terminal head with no TerminalPoW* keys, and since isNewHead is evaluated progressively the head can be several blocks past it. ✅
  • The walk terminates and can't misfire. Post-merge successors keep TD constant and have Difficulty == 0, so IsTerminalBlock rejects them (BlockExtensions.cs:37) and the walk continues to the real terminal block; it follows Head's ancestry so the block it finds is canonical by construction. FindParentHeader on genesis passes Keccak.Zero, which FindHeader short-circuits (BlockTree.cs:531), so the ulong Number - 1 underflow is unreachable. ✅
  • Terminal_block_behind_a_committed_head_is_recovered_by_walking_back is not accidentally green. Its Assert.That(blockTree.Head!.Number, Is.EqualTo(secondPoS.Number)) genuinely holds: the two PoS blocks share the terminal block's TD, so preMergeImprovementRequirementSatisfied is false, but postMergeImprovementRequirementSatisfied (TD >= TTD) carries the head forward (BlockTree.cs:1357-1359). ✅
  • The missed TerminalBlockReached edge is benign. The reconcile records metadata from the constructor, where the event still has no subscribers, so the one-shot is consumed unheard. All three consumers also read the predicate — MergeP2PCapabilityResolver.Resolve, MergeBlockProducerRunner.Start/StopAsync/IsProducingBlocks, and AuRaTerminalBlockDisposer (via IsHeadPostMerge plus its BranchProcessor.BlockProcessing fallback) — and the identical situation already existed on any restart with persisted metadata. Not a regression. ✅
  • BatchTrackingTerminalMetadataDb.TrackingWriteBatch satisfies IWriteBatch (IDisposable + IWriteOnlyKeyValueStore + IMergeableKeyValueStore, IKeyValueStore.cs:97-118) — Set/Merge/Clear/Dispose is the complete set, everything else is defaulted. ✅
  • Existing tests still take the paths they claim. I re-traced GetBlockSwitchInfo_returning_expected_results (both TTD cases) and ..._when_td_null_or_zero under the new reconcile — the walk now records block 4 as terminal during construction in both, but every assertion is reached through a branch that predates MergeBlockNumber, so results are unchanged. ✅

Findings

1. Medium — genesis is still recorded as the terminal PoW block when TTD comes from MergeConfig rather than the chain spec (PoSSwitcher.cs:197)

IsPostMergeGenesis reads _chainSpec.Parameters.TerminalTotalDifficulty by design (line 329-331), but this guard needs the effective TTD. ChainSpec.Parameters has no initializer, so new ChainSpec { Genesis = … } leaves it null — and that is precisely MergeTestBlockchain: MergeConfig.TerminalTotalDifficulty ??= "0" (BaseEngineModuleTests.cs:197) with CreateChainSpec() => new() { Genesis = …WithDifficulty(0) } (line 207). IsPostMergeGenesis is then false, genesis.IsTerminalBlock is true (header.IsGenesis short-circuits the difficulty clause), and NewHeadBlock(genesis) from LoadGenesisBlock persists TerminalPoWNumber = 0 and moves MergeBlockNumber 0 → 1.

Master could not reach this: HasPostTerminalTotalDifficultyGenesis(0) set _finalTotalDifficulty = 0, so TransitionFinished was true and the old guard was a permanent no-op. The TransitionFinished → _finalizedBlockHash swap opened it and 21dc039 closed only the chain-spec half. Second consequence: with the new terminalSelectionOpen && isTerminalBlock precedence (line 302), GetBlockConsensusInfo(genesis) on such a setup returns (true, false) and sets header.IsPostMerge = false, where master returned (false, true). One-token guard and a test suggestion are in the inline comment.

2. Low — the recovery walk uses BlockTreeLookupOptions.None, so it can write to the DB and throw out of the constructor (PoSSwitcher.cs:173-183). None requests TD and level creation; FindHeader then calls SetTotalDifficulty + UpdateOrCreateLevel on a miss (BlockTree.cs:554-573, whose own comment calls it out as "readonly method modifies DB"), and SetTotalDifficulty throws InvalidOperationException("An orphaned block on the chain …") when an ancestor is missing (:1686) — escaping PoSSwitcher's constructor and aborting startup, from a scan that is entitled to give up. DoNotCreateLevelIfMissing leaves TD null, which the loop already treats as "stop". Low because the chain-level batch is flushed before the events, so main-chain ancestors normally have their levels — but this is the one path that runs specifically after an unclean shutdown.

Two smaller points in the same block (same thread): the walk gives up silently when the terminal block is more than 256 back, leaving the DB in exactly the state the commit fixes with no log line; and it runs on chains that can never have a terminal PoW block (HasPostTerminalTotalDifficultyGenesis → merged at genesis, HasPostTerminalTotalDifficultyPivot → started post-TTD), where it can only burn 256 lookups per startup.

3. Low — Local_chain_above_ttd_marks_terminal_block_as_reached_without_terminal_block_metadata no longer runs without terminal block metadata (PoSSwitcherTests.cs:333). The reconcile now records block 2 during construction, so the assertion is satisfied by _hasTerminalBlockMetadata and the head-TD clause it was written for is never reached. Still green; the coverage moved.

4. Low — MaxTerminalReconcileDepth's remark asserts a bound it doesn't establish. "Comfortably above one deferred-event processing branch" — the branch handed to TryUpdateMainChain is processingBranch.Blocks (BlockchainProcessor.cs:529), whose length is a reorg depth, not a constant. 256 is a fine pragmatic choice; the remark would be more useful stating that it bounds the scan and that exceeding it is a give-up, rather than implying the branch can't be longer.

5. Low (carried over, unchanged) — the metadata write batch runs under _transitionLock (:202-206), the same monitor ForkchoiceUpdated takes on every FCU with a finalized hash; and the constructor-only Volatile.Writes at :89-90 / :188 read as if concurrent publication mattered. Neither changes behaviour.


Notes (not findings)

  • Best_suggested_header_is_not_durable_ttd_evidence is now DI-backed per @flcl42's test-infrastructure point, and its trailing comment correctly records that the post-processing True comes from the NewHeadBlock metadata path, not from head TD.
  • TransitionFinished is still true whenever FinalTotalDifficulty is configured, so MergeP2PCapabilityResolver advertises eth/69–71 while an archive node full-syncs the pre-merge range. Unchanged by this PR and already in the known follow-ups.
  • The SPDX-year point @flcl42 raised is unaddressed; both files still carry 2022.

Testing caveat

dotnet build and dotnet test are outside the allowed tool set in this job (both were denied), so everything above is by inspection and by tracing against b1b6fa58a7. Finding 1 in particular predicts a change in MergeTestBlockchain behaviour that the PR reports as green — worth confirming locally whether Nethermind.Merge.Plugin.Test genuinely leaves MergeBlockNumber == 0 on those fixtures, since that is the cheapest way to settle it.

Verdict

One Medium to resolve before merge. Finding 1 silently persists incorrect merge-transition metadata and flips genesis classification on every TTD-0-via-MergeConfig setup, it is a change from master, and the fix is a single term in a guard the PR already touches. Everything else is Low: a defensive lookup flag, a silent give-up, a test whose coverage drifted, and two carried-over nits.
• branch fix/final-total-difficulty-ttd-reached

}
// Config-known FinalTotalDifficulty must not block recording the locally observed terminal block,
// so this checks the finalized hash (EIP-3675 step 3) rather than TransitionFinished.
if (_terminalBlockExplicitSpecified || _finalizedBlockHash != Keccak.Zero || IsPostMergeGenesis(header) || !header.IsTerminalBlock(_specProvider))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium — genesis is still recorded as the terminal PoW block when TTD comes from MergeConfig instead of the chain spec.

IsPostMergeGenesis deliberately reads _chainSpec.Parameters.TerminalTotalDifficulty (line 331, with the comment explaining why: MergeConfig overrides must not change genesis classification). But this guard needs the effective TTD, and the two diverge exactly where TTD-0 is supplied on the command line or by a test/devnet harness with Parameters unset:

  • ChainSpec.Parameters is a plain { get; set; } with no initializer, so new ChainSpec { Genesis = … } leaves it null → IsPostMergeGenesis is false.
  • MergeTestBlockchain does exactly that: MergeConfig.TerminalTotalDifficulty ??= "0" (BaseEngineModuleTests.cs:197) plus CreateChainSpec() => new() { Genesis = …WithDifficulty(0) } (line 207).

In that state LoadGenesisBlock raises NewHeadBlock(genesis), and genesis.IsTerminalBlock(_specProvider) is true (IsPostTTD is 0 >= 0, and header.IsGenesis short-circuits the difficulty clause, BlockExtensions.cs:37). So TerminalPoWNumber = 0 / TerminalPoWHash = <genesis> are persisted and MergeBlockNumber moves 0 → 1.

On master this could not happen: HasPostTerminalTotalDifficultyGenesis(0) set _finalTotalDifficulty = 0, so TransitionFinished was true and the old guard made this a permanent no-op. The TransitionFinished → _finalizedBlockHash swap is what opened it, and 21dc039's IsPostMergeGenesis only closed the chain-spec half.

There is a second, observable consequence beyond the persisted metadata: with the new terminalSelectionOpen && isTerminalBlock precedence branch (line 302), GetBlockConsensusInfo(genesis) on such a setup now returns (true, false) and writes header.IsPostMerge = false, where master returned (false, true).

Since the PR settled on "chain-spec TTD zero ⇒ merged at genesis, no terminal PoW block", the same rule should apply when the zero arrives via MergeConfig:

Suggested change
if (_terminalBlockExplicitSpecified || _finalizedBlockHash != Keccak.Zero || IsPostMergeGenesis(header) || !header.IsTerminalBlock(_specProvider))
if (_terminalBlockExplicitSpecified || _finalizedBlockHash != Keccak.Zero || header.IsGenesis && TerminalTotalDifficulty?.IsZero == true || !header.IsTerminalBlock(_specProvider))

A PoSSwitcherTests case with MergeConfig { TerminalTotalDifficulty = "0" }, a ChainSpec with no Parameters, and a genesis head asserting MergeBlockNumber stays 0 would pin it — the existing Post_merge_genesis_is_not_persisted_as_terminal_pow only covers the chain-spec path.

Comment on lines +173 to 183
BlockHeader? header = _blockTree.Head?.Header;
for (int depth = 0; depth < MaxTerminalReconcileDepth; depth++)
{
if (header?.TotalDifficulty is null || header.TotalDifficulty < terminalTotalDifficulty)
return;

if (TryUpdateTerminalBlock(header))
return;

header = _blockTree.FindParentHeader(header, BlockTreeLookupOptions.None);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low — BlockTreeLookupOptions.None makes a crash-recovery scan able to write to the DB, and to throw out of the constructor.

None means both "total difficulty needed" and "create the chain level if it is missing". BlockTree.FindHeader (BlockTree.cs:554-573) documents that combination as "this is here because storing block data is not transactional … we should remove it, readonly method modifies DB": when the block info is absent it calls SetTotalDifficulty(header) and UpdateOrCreateLevel(…), and SetTotalDifficulty throws InvalidOperationException($"An orphaned block on the chain {current}") (BlockTree.cs:1686) when an ancestor header is missing. That exception would escape PoSSwitcher's constructor and abort node startup — from a best-effort recovery walk that this method is entitled to abandon.

The loop already treats a null TD as "stop", so DoNotCreateLevelIfMissing degrades gracefully instead:

Suggested change
BlockHeader? header = _blockTree.Head?.Header;
for (int depth = 0; depth < MaxTerminalReconcileDepth; depth++)
{
if (header?.TotalDifficulty is null || header.TotalDifficulty < terminalTotalDifficulty)
return;
if (TryUpdateTerminalBlock(header))
return;
header = _blockTree.FindParentHeader(header, BlockTreeLookupOptions.None);
}
BlockHeader? header = _blockTree.Head?.Header;
for (int depth = 0; depth < MaxTerminalReconcileDepth; depth++)
{
if (header?.TotalDifficulty is null || header.TotalDifficulty < terminalTotalDifficulty)
return;
if (TryUpdateTerminalBlock(header))
return;
header = _blockTree.FindParentHeader(header, BlockTreeLookupOptions.DoNotCreateLevelIfMissing);
}

Main-chain ancestors normally have their level persisted (the chain-level batch is flushed before NewHeadBlock, so the crash window this walk covers is after those writes), which is why I'm filing this as Low rather than higher — but this is the one code path that runs specifically after an unclean shutdown, so the defensive option is the cheap one.

Two smaller things in the same block:

  • The walk gives up silently. If the terminal block is more than MaxTerminalReconcileDepth behind the head, the loop just falls out and the database stays in exactly the unrecoverable state the commit set out to fix, with no log line. A _logger.IsWarn on exhausting the bound would make that diagnosable.
  • It runs on chains that can never have a terminal PoW block. When HasPostTerminalTotalDifficultyGenesis(...) holds (hoodi, any chain-spec TTD-zero network) the chain merged at genesis, and when HasPostTerminalTotalDifficultyPivot(...) holds the node started post-TTD and never saw the terminal block locally. In both cases the walk can only ever burn 256 FindParentHeader lookups per startup and record nothing — an early return on those two predicates you already have would skip it.

BlockTree blockTree = Build.A.BlockTree(genesisBlock, specProvider).OfChainLength(4).TestObject;
MergeConfig mergeConfig = new() { FinalTotalDifficulty = "3000000" };
PoSSwitcher poSSwitcher = new(mergeConfig, new SyncConfig(), new MemDb(), blockTree, specProvider, new ChainSpec { Genesis = genesisBlock }, LimboLogs.Instance);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low — this test no longer exercises the "without terminal block metadata" state its name pins.

With the startup walk-back in place, constructing the switcher on this tree records metadata before the assertion runs: head is block 3 (TD 4,000,000), the walk steps to block 2 (TD 3,000,000, parent 2,000,000 < TTD 3,000,000) and TryUpdateTerminalBlock succeeds, so _hasTerminalBlockMetadata is true and HasEverReachedTerminalBlock() returns on the first branch. The head-TD clause in HasDurableTerminalTotalDifficultyEvidence() is never reached, and specProvider.MergeBlockNumber is now silently mutated to 3 as a side effect.

The assertion still passes, so this is not a failure — but the case it was written for is now only reachable when the terminal block is further than MaxTerminalReconcileDepth behind the head, or on the pivot/genesis clauses. Adding Assert.That(metadataDb.KeyExists(MetadataDbKeys.TerminalPoWNumber), Is.False) before the call would have caught the drift and would make the intent explicit; a chain deeper than 256 above TTD (or a substituted head with no reachable parents, as Local_ttd_evidence_at_initialization_… now uses) restores the original coverage.

…ve TTD

IsPostMergeGenesis reads the chain-spec TTD on purpose, so MergeConfig
overrides cannot reclassify genesis - GetBlockSwitchInfo_does_not_mark_genesis
_post_merge_for_config_only_ttd_zero pins that. But the recording guard needs
the effective value: ChainSpec.Parameters is null on a command-line or devnet
setup (MergeTestBlockchain builds exactly that), so genesis passed the guard,
IsTerminalBlock is true for genesis at TTD 0, and TerminalPoWNumber/Hash were
persisted with MergeBlockNumber moved 0 -> 1. Master could not reach this: the
zero made TransitionFinished true and the old guard a permanent no-op.

Classification is left alone; only persistence is corrected.
@kamilchodola

Copy link
Copy Markdown
Contributor Author

@claude re-review

@claude

claude Bot commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @kamilchodola's task in 5m 33s —— View job


Re-review: Don't treat configured FinalTotalDifficulty as TTD reached

  • Read the new commit (a4fc08d) and the full PR diff at HEAD
  • Check whether the prior Medium/Low findings were addressed
  • Verify the new guard against chain specs, MergeTestBlockchain and BlockExtensions
  • Review tests
  • Post review

a4fc08d is the only commit since my last round, and it closes the Medium I raised. I re-traced it against the code rather than trusting the commit message, and I also checked the two open threads @kamilchodola answered in prose. No Critical/High/Medium findings remain.


Prior findings — status

Finding Raised by Status
Medium — genesis recorded as terminal PoW when TTD comes from MergeConfig/CLI rather than the chain spec claude ✅ Fixed. Guard now tests the effective TTD (PoSSwitcher.cs:201), so MergeTestBlockchain (MergeConfig.TerminalTotalDifficulty ??= "0" + ChainSpec.Parameters unset) no longer persists TerminalPoWNumber = 0. New test Config_only_ttd_zero_genesis_is_not_persisted_as_terminal_pow asserts both keys absent and MergeBlockNumber == 0
High — persisted FinalizedBlockHash may be a SYNCING hint flcl42 🗣 Answered: unchanged from b1b6fa58a7, and the key is not Engine-API-exclusive, so provenance validation can't work — deferred. Reads as a fair call
High — first provisional candidate disables AuRa finalization flcl42 🗣 Answered: git diff b1b6fa58a7..HEAD -- Nethermind.Merge.AuRa/ is two comments; master already disposes on the first candidate for non-FTD AuRa nodes. Deferred
All High/Medium items from the earlier five rounds (provisional-height rejection, restart resubscription, Hoodi IsPostMergeGenesis, monotonic predicate, event suppression, TryUpdateTerminalBlock synchronisation, batch atomicity, committed-head reconciliation, walk-back) claude, flcl42, AnkushinDaniil, wurdum ✅ Still in place at HEAD, each with a test pinning it

What I verified this round

  • The new term covers both real shapes. _specProvider.TerminalTotalDifficulty is the chain-spec value unless MergeConfig/CLI overrides it (Initialize → UpdateMergeTransitionInfo(_firstPoSBlockNumber, _mergeConfig.TerminalTotalDifficultyParsed)), so hoodi (chain-spec 0x0, genesis difficulty 0x01) is still rejected — the case @AnkushinDaniil and I flagged on 08-04 stays closed — and the config-only zero is now rejected too. ✅
  • The new test is sound, not accidentally green. TestSpecProvider.UpdateMergeTransitionInfo only assigns when the argument is non-null (TestSpecProvider.cs:20-23), so the constructor's UpdateMergeTransitionInfo(null, 0) leaves MergeBlockNumber == 0 and the final assertion is a real check rather than a null comparison. Substitute.For<IBlockTree>() gives a null Head, so the reconcile walk exits before the assertion. ✅
  • Nothing legitimate is now blocked. The term requires a zero effective TTD, so a chain whose genesis is genuinely the terminal PoW block under a positive TTD (IsTerminalBlock_returning_expected_result_for_genesis_block) is still recordable, and HasPostTerminalTotalDifficultyGenesis / Genesis_difficulty_reaching_ttd_marks_terminal_block_as_reached are untouched. ✅
  • The recovery walk cannot loop or misfire on TTD-zero chains. Genesis is now rejected by the guard, and FindParentHeader(genesis, …) passes Keccak.Zero, which FindHeader short-circuits — so the walk ends on the next iteration instead of underflowing Number - 1. ✅
  • SPDX-year thread (@flcl42): I don't think this one is actionable. CONTRIBUTING.md:75-83 and .editorconfig:12 define the header template for new files; 2370 .cs files in src/Nethermind still carry 2022 against 1249 carrying 2026, including heavily-churned ones, so the tree's practice is "year of creation, not of last edit". Bumping it here would also be diff noise against AGENTS.md's minimise-the-diff rule.

Findings (all Low)

1. Low — the fix replaced the chain-spec term instead of adding to it (PoSSwitcher.cs:200-202). Two consequences: Post_merge_genesis_is_not_persisted_as_terminal_pow and Config_only_ttd_zero_genesis_is_not_persisted_as_terminal_pow now exercise the identical predicate (the fixtures differ only in ChainSpec.Parameters, which the guard no longer reads) — a [TestCase] pair per AGENTS.md's DRY rule; and the chain-spec-zero + effective-non-zero combination (hoodi --Merge.TerminalTotalDifficulty=1) is now unguarded, where the guard and GetBlockConsensusInfo's IsPostMergeGenesis used to agree. Keeping both terms is one token.

2. Low — genesis classification on config-only TTD-zero setups changed, and the new test doesn't pin it. With a genesis header that carries a TD, GetBlockConsensusInfo now reaches terminalSelectionOpen && isTerminalBlock (line 307) and returns (true, false), where master returned (false, true). So the switcher calls genesis the terminal PoW block and then declines to record it. GetBlockSwitchInfo_does_not_mark_genesis_post_merge_for_config_only_ttd_zero builds its genesis WithTotalDifficulty(null), so it exits at line 295 and never reaches this branch. Genesis is the only header that can (every other TotalDifficulty == 0 header is caught earlier) and no production consumer classifies genesis — MergeHeaderValidator/MergeSealValidator are not invoked for it and BlockForRpc reads specProvider.MergeBlockNumber, not header.IsPostMerge — hence Low. Detail in the same inline thread.

3. Low (carried, unchanged) — the recovery walk's lookup flag and give-up behaviour (r3883311757). BlockTreeLookupOptions.None (line 182) still requests level creation, so FindHeader can write to the DB and SetTotalDifficulty can throw InvalidOperationException out of PoSSwitcher's constructor on the one path that runs specifically after an unclean shutdown; DoNotCreateLevelIfMissing degrades into the loop's existing "TD null ⇒ stop". Same thread: the walk gives up silently at MaxTerminalReconcileDepth with no log line, and it runs on chains that can never have a terminal PoW block (HasPostTerminalTotalDifficultyGenesis ⇒ merged at genesis, HasPostTerminalTotalDifficultyPivot ⇒ started post-TTD), where it can only burn up to 256 header lookups per startup.

4. Low (carried) — Local_chain_above_ttd_marks_terminal_block_as_reached_without_terminal_block_metadata no longer runs without terminal metadata (r3883312801). The reconcile walk records block 2 during construction, so the assertion is satisfied by _hasTerminalBlockMetadata and the head-TD clause is never reached. Related: the comment at PoSSwitcherTests.cs:377-379 now points at this test as the coverage for that clause, which it no longer provides. Still green; the coverage moved.

5. Low (carried, unchanged) — MaxTerminalReconcileDepth's remark ("comfortably above one deferred-event processing branch") asserts a bound that processingBranch.Blocks doesn't establish; the metadata write batch runs under _transitionLock (lines 207-211), the monitor ForkchoiceUpdated also takes on every FCU with a finalized hash; and the constructor-only Volatile.Writes at lines 89-90 / 188 read as if concurrent publication mattered.


Notes (not findings)

  • Comment quality on the added code reads well: the guard's comment (195-199) explains why the finalized hash and the effective TTD are the right inputs, and the ReconcileTerminalBlockFromHead remark states the crash window it exists for. No restatement-of-code or generation residue in this commit.
  • TransitionFinished is still true whenever FinalTotalDifficulty is configured, so MergeP2PCapabilityResolver advertises eth/69–71 while an archive node full-syncs the pre-merge range. Unchanged by this PR and already in the known follow-ups.

Testing caveat

dotnet build and dotnet test are outside the allowed tool set in this job (both were denied again), so everything above is by inspection and by tracing against b1b6fa58a7. The PR reports Nethermind.Merge.Plugin.Test and Nethermind.Merge.AuRa.Test green; worth one local run since a4fc08d changes a guard that MergeTestBlockchain fixtures pass through on every engine test.

Verdict

Mergeable. Every Critical/High/Medium item raised across the six rounds is either fixed with a test pinning it or explicitly answered with rationale. The five remaining items are a one-token guard belt-and-braces, an inert genesis-classification asymmetry, a defensive lookup flag, a test whose coverage drifted, and three carried-over nits.
• branch fix/final-total-difficulty-ttd-reached

Comment on lines +200 to +202
if (_terminalBlockExplicitSpecified || _finalizedBlockHash != Keccak.Zero
|| header.IsGenesis && TerminalTotalDifficulty?.IsZero == true
|| !header.IsTerminalBlock(_specProvider))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low — the fix is right, but it replaced the chain-spec term instead of adding to it, and the two genesis tests now cover the same clause.

a4fc08d closes the MergeConfig/CLI half of the defect — I re-traced it: MergeTestBlockchain (MergeConfig.TerminalTotalDifficulty ??= "0" + new ChainSpec { Genesis = … } with Parameters unset) now hits header.IsGenesis && TerminalTotalDifficulty?.IsZero == true and no longer persists TerminalPoWNumber = 0. hoodi still goes through the same term (_specProvider.TerminalTotalDifficulty is the chain-spec 0x0 when nothing overrides it), so the previously-closed chain-spec case stays closed. ✅

Two small consequences of dropping IsPostMergeGenesis(header) from the guard rather than OR-ing:

  1. Post_merge_genesis_is_not_persisted_as_terminal_pow and Config_only_ttd_zero_genesis_is_not_persisted_as_terminal_pow now exercise the identical predicate. The only difference between the fixtures is whether ChainSpec.Parameters.TerminalTotalDifficulty is set, and the guard no longer reads it — both are rejected by the effective-TTD term. Per AGENTS.md ("when tests differ only by inputs and expected outputs, parameterize a single test"), these are a [TestCase] pair, unless the chain-spec one also asserts the classification result, which is what actually distinguishes the two setups.
  2. The chain-spec-zero + effective-non-zero combination is no longer guarded. hoodi --Merge.TerminalTotalDifficulty=1: chain-spec TTD is 0x0, effective TTD is 1, genesis TD is 1 → IsTerminalBlock(genesis) is true (header.IsGenesis short-circuits, BlockExtensions.cs:37), the new term does not fire, and genesis is recorded with MergeBlockNumber 0 → 1 — while GetBlockConsensusInfo still classifies genesis post-merge off the chain-spec TTD (line 337). Contrived, and arguably what an operator who typed that asked for, but the two predicates now disagree where they used to agree.

Keeping both terms costs one token and makes the guard "genesis is not a terminal PoW block under either reading of a zero TTD":

Suggested change
if (_terminalBlockExplicitSpecified || _finalizedBlockHash != Keccak.Zero
|| header.IsGenesis && TerminalTotalDifficulty?.IsZero == true
|| !header.IsTerminalBlock(_specProvider))
if (_terminalBlockExplicitSpecified || _finalizedBlockHash != Keccak.Zero
|| IsPostMergeGenesis(header) || header.IsGenesis && TerminalTotalDifficulty?.IsZero == true
|| !header.IsTerminalBlock(_specProvider))

Separately, on the classification half that the commit message says is intentionally left alone: on a config-only TTD-zero setup a genesis header that does carry a TD now reaches the new terminalSelectionOpen && isTerminalBlock branch (line 307) and comes back (isTerminal: true, isPostMerge: false), where master returned (false, true) via TransitionFinished && theMergeEnabled. So the switcher reports genesis as the terminal PoW block and then refuses to record it. GetBlockSwitchInfo_does_not_mark_genesis_post_merge_for_config_only_ttd_zero builds its genesis WithTotalDifficulty((UInt256?)null), so it exits through the TotalDifficulty is null branch and does not pin this. Genesis is the only header this can reach (every other TotalDifficulty == 0 header is caught at line 295) and no production consumer classifies genesis, so Low — but a second case with a TD present would make the intended answer explicit.

@kamilchodola
kamilchodola merged commit 4ac7005 into master Sep 10, 2026
573 of 575 checks passed
@kamilchodola
kamilchodola deleted the fix/final-total-difficulty-ttd-reached branch September 10, 2026 15:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants