Conversation
Demonstrates that stale TELEMETRYCNTS Redis keys left over after a plugin upgrade cause RuntimeError in both init_tlm_packet_counts and sync_tlm_packet_counts, which propagates through handle_packet() and triggers an interface disconnect/reconnect loop. Also updates the just test recipe to accept optional args so that `just test <file>` works alongside the bare `just test`. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Wrap System.telemetry.packet() calls in init_tlm_packet_counts and sync_tlm_packet_counts with RuntimeError handling so stale TELEMETRYCNTS keys left over after a plugin upgrade are skipped rather than propagating an exception through handle_packet() into the interface disconnect logic. A WARN is logged once per stale key per interface restart so operators can identify leftover keys without flooding the log. The warned-keys set is reset on each init_tlm_packet_counts call (each interface restart), bounding memory to the stale keys seen in the current epoch only. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2857 +/- ##
==========================================
+ Coverage 78.74% 78.82% +0.08%
==========================================
Files 667 667
Lines 54497 54542 +45
Branches 731 731
==========================================
+ Hits 42913 42993 +80
+ Misses 11504 11469 -35
Partials 80 80
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
jmthomas
requested changes
Feb 24, 2026
Member
jmthomas
left a comment
jmthomas
left a comment
There was a problem hiding this comment.
This same exact code is in Ruby. Please make the equivalent changes there.
…packet to protect against stale redis keys
…is-keys-cause-interface-disconnect-loops
|
Contributor
Author
Updated Ruby implementation to match changes in Python |
jmthomas
approved these changes
Feb 24, 2026
mcosgriff
deleted the
2855-stale-tlmcnt-redis-keys-cause-interface-disconnect-loops
branch
jmthomas
pushed a commit
that referenced
this pull request
Mar 21, 2026
* Add failing tests for stale tlmcnt Redis keys (issue #2855) Demonstrates that stale TELEMETRYCNTS Redis keys left over after a plugin upgrade cause RuntimeError in both init_tlm_packet_counts and sync_tlm_packet_counts, which propagates through handle_packet() and triggers an interface disconnect/reconnect loop. Also updates the just test recipe to accept optional args so that `just test <file>` works alongside the bare `just test`. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Fix stale tlmcnt Redis keys causing interface disconnect loops (#2855) Wrap System.telemetry.packet() calls in init_tlm_packet_counts and sync_tlm_packet_counts with RuntimeError handling so stale TELEMETRYCNTS keys left over after a plugin upgrade are skipped rather than propagating an exception through handle_packet() into the interface disconnect logic. A WARN is logged once per stale key per interface restart so operators can identify leftover keys without flooding the log. The warned-keys set is reset on each init_tlm_packet_counts call (each interface restart), bounding memory to the stale keys seen in the current epoch only. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Ran formatter * Same issue was in the Ruby implementation. Wrapping System.telemetry.packet to protect against stale redis keys --------- Co-authored-by: Claude Sonnet 4.6 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stale
TELEMETRYCNTSRedis keys left over from removed packet definitions caused RuntimeError ininit_tlm_packet_countsandsync_tlm_packet_counts, which propagated throughhandle_packet()intohandle_connection_lost(), triggering a continuous disconnect/reconnect loop.Changes
Closes #2855