Skip to content

Deflake "Replica output bytes metric" with atomic stats capture - #4102

Merged
ranshid merged 4 commits into
valkey-io:unstablefrom
Taeknology:fix/replica-output-bytes-metric-3930
Jul 31, 2026
Merged

ranshid merged 4 commits into
valkey-io:unstablefrom
Taeknology:fix/replica-output-bytes-metric-3930

Conversation

@Taeknology

@Taeknology Taeknology commented Jul 6, 2026 •

Copy link
Copy Markdown
Contributor

The test resets stats and asserts total_net_repl_output_bytes is 0. A periodic replication PING can otherwise land between CONFIG RESETSTAT and INFO stats, making the counter 14 (the serialized PING frame) instead of 0.

#4158 added a replication-stream drain and a longer propagation wait. This follow-up keeps those improvements and queues CONFIG RESETSTAT plus INFO stats atomically in MULTI/EXEC, so the zero-baseline check has no interleaving window. It does not mutate repl-ping-replica-period.

Validation:

  • atomic probe: 100/100 passed
  • make -j2 passed
  • integration/replication: 67/67 passed across four runs

Fixes #3930

The test resets stats and asserts total_net_repl_output_bytes is 0.
But the primary sends a periodic PING to the replica (every
repl-ping-replica-period, default 10s). If that PING lands between
`config resetstat` and the `info stats` read, the counter is 14 (the
serialized ping frame) instead of 0 and the test fails on CI:

    Expected '14' to be equal to '0'

The replica is online, so the PING is the only writer of the primary's
replication output bytes in this window. Raise repl-ping-replica-period
to 3600 and drain the stream with wait_for_ofs_sync before resetstat, so
no ping (queued or future) can fire during it. The counter stays 0 after
reset and still grows after the SET. This idiom is already used in this
file.

Fixes valkey-io#3930.

Signed-off-by: Taeknology <[email protected]>
@coderabbitai

coderabbitai Bot commented Jul 6, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Updated the "Replica output bytes metric" test to reset replication statistics and capture the baseline snapshot atomically using MULTI/EXEC on instance $A.

Changes

Replication metric test fix

Layer / File(s) Summary
Atomic replication stats baseline
tests/integration/replication.tcl
Runs config resetstat and info stats in a single MULTI/EXEC sequence after offset synchronization.

Estimated code review effort: 2 (Simple) | ~5 minutes

Possibly related PRs

  • valkey-io/valkey#4158: Updates the same replication metric test to synchronize replication before resetting or sampling stats.

Suggested reviewers: sarthakaggarwal97, enjoy-binbin, dvkashapov

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The change addresses #3930 by removing the race where a PING can land between resetstat and info stats, keeping the zero-baseline check stable.
Out of Scope Changes check ✅ Passed The only change is the targeted replication test fix, with no unrelated code paths or extra behavior changes.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Title check ✅ Passed The title clearly summarizes the main change: deflaking the replica output bytes metric test via atomic stats capture.
Description check ✅ Passed The description directly matches the changeset and explains the atomic reset/stat collection fix and validation.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Jul 6, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.84%. Comparing base (7bb7b7f) to head (b286d84).
⚠️ Report is 14 commits behind head on unstable.

Additional details and impacted files
@@             Coverage Diff              @@
##           unstable    #4102      +/-   ##
============================================
+ Coverage     76.83%   76.84%   +0.01%     
============================================
  Files           162      162              
  Lines         81433    81433              
============================================
+ Hits          62565    62574       +9     
+ Misses        18868    18859       -9     

see 12 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread tests/integration/replication.tcl Resolved
Signed-off-by: Taeknology <[email protected]>

# Conflicts:
#	tests/integration/replication.tcl

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integration/replication.tcl`:
- Around line 198-209: In the test {Replica output bytes metric}, configure
replica pinging to be effectively disabled for the verification window by
setting repl-ping-replica-period to 3600 before resetting stats and capturing
the baseline. Keep the existing wait_for_ofs_sync and atomic stats-reset
sequence intact so only the intended replication traffic affects
total_net_repl_output_bytes.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3d135bf4-e123-4c69-b660-12eb90879a05

📥 Commits

Reviewing files that changed from the base of the PR and between 88c8492 and b286d84.

📒 Files selected for processing (1)
  • tests/integration/replication.tcl

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integration/replication.tcl`:
- Around line 198-209: In the test {Replica output bytes metric}, configure
replica pinging to be effectively disabled for the verification window by
setting repl-ping-replica-period to 3600 before resetting stats and capturing
the baseline. Keep the existing wait_for_ofs_sync and atomic stats-reset
sequence intact so only the intended replication traffic affects
total_net_repl_output_bytes.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3d135bf4-e123-4c69-b660-12eb90879a05

📥 Commits

Reviewing files that changed from the base of the PR and between 88c8492 and b286d84.

📒 Files selected for processing (1)
  • tests/integration/replication.tcl
🛑 Comments failed to post (1)
tests/integration/replication.tcl (1)

198-209: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Add the missing repl-ping-replica-period configuration.

Without this setting, a periodic PING could still arrive during the verification window, incrementing the replication output bytes and allowing the test to pass incorrectly even if the SET command wasn't properly accounted for.

Based on the PR objectives, the test should raise repl-ping-replica-period to 3600 to prevent queued or future PINGs from affecting total_net_repl_output_bytes during the assertion. Based on past review comments, this is needed to prevent a false positive test.

🐛 Proposed fix
         test {Replica output bytes metric} {
+            # Prevent periodic PINGs from inflating the output bytes metric and causing false positives
+            $A config set repl-ping-replica-period 3600
+
             # Make sure no replication traffic (initial sync, backlog writes)
             # is still in flight before resetting stats, so the zero-baseline
             # assertion below doesn't race with it.
             wait_for_ofs_sync $A $B
 
             # Reset stats and read them atomically so replication traffic can't
             # arrive between the reset and the stats snapshot.
             $A multi
             $A config resetstat
             $A info stats
             set info [lindex [$A exec] 1]
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

        test {Replica output bytes metric} {
            # Prevent periodic PINGs from inflating the output bytes metric and causing false positives
            $A config set repl-ping-replica-period 3600

            # Make sure no replication traffic (initial sync, backlog writes)
            # is still in flight before resetting stats, so the zero-baseline
            # assertion below doesn't race with it.
            wait_for_ofs_sync $A $B

            # Reset stats and read them atomically so replication traffic can't
            # arrive between the reset and the stats snapshot.
            $A multi
            $A config resetstat
            $A info stats
            set info [lindex [$A exec] 1]
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/replication.tcl` around lines 198 - 209, In the test
{Replica output bytes metric}, configure replica pinging to be effectively
disabled for the verification window by setting repl-ping-replica-period to 3600
before resetting stats and capturing the baseline. Keep the existing
wait_for_ofs_sync and atomic stats-reset sequence intact so only the intended
replication traffic affects total_net_repl_output_bytes.

@Taeknology

Copy link
Copy Markdown
Contributor Author

Regarding CodeRabbit's suggestion to restore repl-ping-replica-period 3600: not applying it.

The demonstrated failure is the race between CONFIG RESETSTAT and INFO stats, which is removed by executing them atomically through MULTI/EXEC.

The positive assertion runs only after replica B observes the actual SET, so real replication traffic has already occurred. A periodic PING may add noise but cannot be the sole reason the assertion passes.

Setting repl-ping-replica-period to 3600 would mutate shared configuration and would reduce rather than deterministically eliminate PINGs. The atomic variant passed the adversarial probe 100/100 and the full replication suite four times.

@Taeknology Taeknology changed the title Deflake "Replica output bytes metric" by disabling periodic pings Deflake "Replica output bytes metric" with atomic stats capture Jul 19, 2026
@Taeknology
Taeknology requested a review from ranshid July 19, 2026 13:19

@rainsupreme rainsupreme left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fix checks out- thanks for this deflake work! I also appreciate the 100x validation run :)

When we squash merge let's avoid mention of the old ping-period approach in the commit message

@rainsupreme rainsupreme added the to-be-merged Almost ready to merge label Jul 22, 2026
@ranshid
ranshid merged commit 69cd2ef into valkey-io:unstable Jul 31, 2026
101 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

to-be-merged Almost ready to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[TEST-FAILURE] Replica output bytes metric in tests/integration/replication.tcl

3 participants