feat(client): validator key consistency check once per epoch by darioush · Pull Request #15417 · near/nearcore

darioush · 2026-03-19T19:25:30Z

At key file load time (KeyFile::from_file), verify the private key derives the same public key listed in the file. Returns InvalidData error on mismatch.
At each epoch transition, verify the loaded validator public key matches the key registered in the upcoming epoch's info. Panics on mismatch with an actionable message. The check is skipped during sync to avoid false positives when the node is behind and the key was correctly rotated after the local head's epoch.
The check fires on the first non-syncing head block of each epoch (via check_validator_key_on_new_head), so it also catches the case where the epoch boundary was crossed during sync.
Add validator_signer field to NodeSetupState to support injecting a custom signer for testing.
Add unit test for key file mismatch and a test-loop test for epoch boundary mismatch detection.

Copilot

Pull request overview

Adds validator key consistency validation at multiple lifecycle points (key file load, client startup, and epoch boundaries) to catch misconfigured validator keys early, with accompanying test-loop coverage.

Changes:

Validate KeyFile::from_file by ensuring secret_key derives the declared public_key (error on mismatch).
Add Client::check_validator_key_for_epoch and invoke it on startup and at epoch boundaries (panic on mismatch for actionable failure).
Extend test-loop setup to allow injecting a custom validator_signer, and add new tests for mismatch scenarios.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`core/crypto/src/key_file.rs`	Adds key-file self-consistency check + unit test for mismatched keys.
`chain/client/src/client.rs`	Introduces epoch-based validator key mismatch check and hooks it at epoch boundary.
`chain/client/src/client_actor.rs`	Runs validator key check on startup for validator nodes.
`test-loop-tests/src/setup/state.rs`	Adds injectable `validator_signer` to `NodeSetupState`.
`test-loop-tests/src/setup/setup.rs`	Uses injected signer (or default) when constructing the client’s validator signer.
`test-loop-tests/src/setup/env.rs`	Initializes new `NodeSetupState.validator_signer` field on kill/restart paths.
`test-loop-tests/src/setup/builder.rs`	Initializes new `NodeSetupState.validator_signer` field in builder.
`test-loop-tests/src/tests/validator_key_check.rs`	Adds test-loop tests for startup and epoch-boundary mismatch detection.
`test-loop-tests/src/tests/mod.rs`	Registers the new validator key check tests module.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-19T19:33:31Z

+                if let Ok(next_epoch_id) =
+                    self.epoch_manager.get_epoch_id_from_prev_block(&block_hash)
+                {
+                    self.check_validator_key_for_epoch(&next_epoch_id);


get_epoch_id_from_prev_block errors are silently ignored here, which can cause the validator key check to be skipped at epoch boundaries (reducing the effectiveness of this safety check) and makes diagnosing failures harder. Please at least log/warn on the error (similar to the shard layout error handling below), or propagate it if this condition should be impossible at this point.

Suggested change

if let Ok(next_epoch_id) =

self.epoch_manager.get_epoch_id_from_prev_block(&block_hash)

{

self.check_validator_key_for_epoch(&next_epoch_id);

match self.epoch_manager.get_epoch_id_from_prev_block(&block_hash) {

Ok(next_epoch_id) => {

self.check_validator_key_for_epoch(&next_epoch_id);

}

Err(err) => {

tracing::warn!(target: "client", ?err, "failed to get epoch id from previous block; skipping validator key check for upcoming epoch");

}

Seems not worth it to me, can add it if reviewer prefers to have it.

codecov · 2026-03-19T19:37:01Z

Codecov Report

❌ Patch coverage is 94.82759% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.28%. Comparing base (70222bf) to head (5273f22).
⚠️ Report is 31 commits behind head on master.

Files with missing lines	Patch %	Lines
chain/client/src/client.rs	93.10%	2 Missing ⚠️
core/crypto/src/key_file.rs	95.83%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #15417      +/-   ##
==========================================
+ Coverage   69.03%   69.28%   +0.24%     
==========================================
  Files         934      938       +4     
  Lines      210089   211122    +1033     
  Branches   210089   211122    +1033     
==========================================
+ Hits       145034   146266    +1232     
+ Misses      59056    58819     -237     
- Partials     5999     6037      +38

Flag	Coverage Δ
pytests-nightly	`1.15% <0.00%> (-0.01%)`	⬇️
unittests	`68.56% <94.82%> (+0.18%)`	⬆️
unittests-nightly	`68.78% <94.82%> (+0.21%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

wacban

nice, thanks for adding this!

wacban · 2026-03-20T10:54:10Z

+            let epoch_id = client.chain.head()?.epoch_id;
+            client.check_validator_key_for_epoch(&epoch_id);


Is there a case where the node is behind and wrong epoch is used for the check?

Great point. Let me figure it out before proceeding.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

only set last_validator_key_check_epoch after get_epoch_info succeeds, so a transient store/epoch-manager error doesn't silently skip the check for the rest of the epoch. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

darioush · 2026-03-27T03:41:08Z

+    /// The last epoch for which the validator key was checked. Used to ensure
+    /// the check runs once per epoch on the first non-syncing head block, even
+    /// if the epoch boundary was crossed during sync.
+    last_validator_key_check_epoch: Option<EpochId>,


I'm also happy to just check it at the beginning of an epoch to avoid adding this field to client

code changed

wacban

LGTM

This PR shows there is a gap in nearcore support for rotating the key on chain. It's very rare event and this PR is an improvement so I don't have any recommendations, just sharing this as an observation.

wacban · 2026-03-27T11:10:12Z

+            return;
+        }
+        let validator_id = signer.validator_id();
+        let epoch_info = match self.epoch_manager.get_epoch_info(epoch_id) {


nit: you can also use the let Ok(epoch_info) = ... else { tracing::error!(...); }; pattern here

wacban · 2026-03-27T11:10:46Z

+            return;
+        };
+        let local_key = signer.public_key();
+        let epoch_key = validator_stake.public_key();


epoch_key doesn't quite convince me, how about chain_key or on_chain_key?

darioush · 2026-03-27T12:20:18Z

Making #15469 to track the validator key rotation issue.

darioush added 2 commits March 19, 2026 12:25

feat(client): add validator key consistency checks at load and startup

585e0ae

nit

cb2f916

darioush requested a review from Copilot March 19, 2026 19:28

Copilot started reviewing on behalf of darioush March 19, 2026 19:28 View session

darioush changed the title ~~feat(client): add validator key consistency checks at load and startup~~ feat(client): validator key consistency checks at epoch change and startup Mar 19, 2026

Copilot AI reviewed Mar 19, 2026

View reviewed changes

darioush requested a review from wacban March 19, 2026 19:36

darioush marked this pull request as ready for review March 19, 2026 19:36

darioush requested a review from a team as a code owner March 19, 2026 19:36

wacban previously approved these changes Mar 20, 2026

View reviewed changes

darioush marked this pull request as draft March 23, 2026 13:46

fix(client): avoid false positives in validator key mismatch check

25f1de9

darioush changed the title ~~feat(client): validator key consistency checks at epoch change and startup~~ feat(client): validator key consistency check once per epoch Mar 27, 2026

darioush and others added 3 commits March 26, 2026 18:20

nit: remove extra blank line

b6ce777

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

merge functions

dc488b2

test improvements

48753a6

darioush requested a review from Copilot March 27, 2026 02:21

Copilot started reviewing on behalf of darioush March 27, 2026 02:22 View session

Copilot AI reviewed Mar 27, 2026

View reviewed changes

Comment thread chain/client/src/client.rs Outdated

darioush marked this pull request as ready for review March 27, 2026 02:54

darioush requested a review from wacban March 27, 2026 02:54

darioush commented Mar 27, 2026

View reviewed changes

wacban approved these changes Mar 27, 2026

View reviewed changes

review comments

5d77d3c

darioush enabled auto-merge March 27, 2026 12:20

darioush added this pull request to the merge queue Mar 27, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Mar 27, 2026

darioush added this pull request to the merge queue Mar 27, 2026

darioush removed this pull request from the merge queue due to a manual request Mar 27, 2026

fix publishable separately

60c5c6c

darioush enabled auto-merge March 27, 2026 12:40

fmt

5273f22

darioush added this pull request to the merge queue Mar 27, 2026

Merged via the queue into near:master with commit c7267a3 Mar 27, 2026
32 checks passed

darioush deleted the validator-key-check branch March 27, 2026 13:16

		let epoch_id = client.chain.head()?.epoch_id;
		client.check_validator_key_for_epoch(&epoch_id);

Uh oh!

Conversation

darioush commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

darioush Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

wacban left a comment

Choose a reason for hiding this comment

Uh oh!

wacban Mar 20, 2026

Choose a reason for hiding this comment

Uh oh!

darioush Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

darioush Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

wacban left a comment

Choose a reason for hiding this comment

Uh oh!

wacban Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

wacban Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

darioush commented Mar 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

darioush commented Mar 19, 2026 •

edited

Loading

codecov Bot commented Mar 19, 2026 •

edited

Loading