Skip to content

fix: update watch_ctx.revision to avoid multiple resyncs#12021

Merged
Revolyssup merged 9 commits into
apache:masterfrom
Revolyssup:revolyssup/resync-etcd
Mar 12, 2025
Merged

fix: update watch_ctx.revision to avoid multiple resyncs#12021
Revolyssup merged 9 commits into
apache:masterfrom
Revolyssup:revolyssup/resync-etcd

Conversation

@Revolyssup

@Revolyssup Revolyssup commented Mar 5, 2025

Copy link
Copy Markdown
Contributor

Description

Fixes # (issue)
Recently this PR was merged #12015

Although it fixed the original issue, it didn't update revision on the watch context which caused resync to happen again and again. This resync will keep happening until the revision will catch up to the watch_ctx.rev.

2025/03/05 22:52:58 [error] 269#269: *7393 [lua] config_etcd.lua:267: received smaller revision, rev=19, watch_ctx.rev=41. etcd may be restarted. resyncing...., context: ngx.timer
2025/03/05 22:52:58 [error] 269#269: *113 [lua] config_etcd.lua:647: sync_data(): waitdir [/apisix/upstreams] err: restarted, will read the configuration again via readdir, context: ngx.timer
2025/03/05 22:53:42 [error] 267#267: *8621 [lua] config_etcd.lua:267: received smaller revision, rev=20, watch_ctx.rev=41. etcd may be restarted. resyncing...., context: ngx.timer
2025/03/05 22:53:42 [error] 272#272: *36 [lua] config_etcd.lua:647: sync_data(): waitdir [/apisix/protos] err: restarted, will read the configuration again via readdir, context: ngx.timer

This change updates watch_ctx.revision to the last observed revision so that once resync takes place it doesn't happen again and again.

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. bug Something isn't working labels Mar 5, 2025
@dosubot dosubot Bot added size:XS This PR changes 0-9 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Mar 5, 2025

@membphis membphis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it easy to add a test case for this case?

@Revolyssup

Copy link
Copy Markdown
Contributor Author

is it easy to add a test case for this case?

not easy because it requires restarting etcd and deleting data which is outside the scope of test cases. I tried mocking etcd related functions and it got really complicated so I decided to halt the efforts in that direction. But this can be tested manually with reproduction steps.

@nic-chen nic-chen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also did a simple test, and the previous error log did not appear again

@Revolyssup
Revolyssup merged commit b4980df into apache:master Mar 12, 2025
laz-xyr pushed a commit to laz-xyr/apisix that referenced this pull request Jun 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working size:XS This PR changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants