fix: batch-processor infinite timer loop prevents graceful shutdown#13288
Merged
nic-6443 merged 2 commits intoApr 27, 2026
Merged
Conversation
dosubot
Bot
added
size:L
nic-6443
force-pushed
the
fix/batch-processor-shutdown-infinite-loop
branch
from
April 24, 2026 02:51
6c9dd9a to
622e955
Compare
There was a problem hiding this comment.
Pull request overview
Fixes a shutdown regression in the batch processor where timer fallback behavior could create a zero-delay timer loop, preventing Nginx workers from exiting gracefully.
Changes:
- Pass
batchinto theexecute_functimer fallback whenngx.timer.atreturns"process exiting". - Update
flush_bufferto flush immediately when called withpremature=trueto avoid extending timers during shutdown. - Add Test::Nginx coverage to verify buffered entries are flushed during worker shutdown and the suite continues (no hang).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
apisix/utils/batch-processor.lua |
Adjusts shutdown behavior for buffer flushing and fixes missing batch arg in timer fallback. |
t/utils/batch-processor.t |
Adds regression tests covering shutdown flush and ensuring the worker exits cleanly. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
nic-6443
force-pushed
the
fix/batch-processor-shutdown-infinite-loop
branch
2 times, most recently
from
April 24, 2026 03:18
fe3fa56 to
ee40382
Compare
dosubot
Bot
added
size:M
nic-6443
force-pushed
the
fix/batch-processor-shutdown-infinite-loop
branch
6 times, most recently
from
April 25, 2026 08:59
1af640f to
8a06608
Compare
During worker shutdown, flush_buffer is called with premature=true but never checked it, causing an infinite zero-delay timer loop that prevents the worker from exiting. Fix by adding premature to the flush condition so shutdown triggers the same process_buffer() path. Also fix schedule_func_exec to pass the batch arg in its 'process exiting' fallback (previously nil, would crash if a retry timer fires during shutdown with retry_delay > 0).
nic-6443
force-pushed
the
fix/batch-processor-shutdown-infinite-loop
branch
from
April 25, 2026 11:32
8a06608 to
1b74a41
Compare
…shutdown Use short inactive_timeout (1s) with continuous entry pushing instead of long timeouts (3600s). This creates the race condition where timers naturally expire during shutdown with premature=false, which is the actual bug path that ngx.worker.exiting() guards against.
nic-6443
requested review from
AlinsRan,
Baoyuantop,
moonming,
nic-chen and
shreemaan-abhishek
April 26, 2026 01:21
moonming
approved these changes
Apr 26, 2026
nic-chen
approved these changes
Apr 26, 2026
AlinsRan
approved these changes
Apr 27, 2026
wistefan
pushed a commit
to wistefan/apisix
that referenced
this pull request
Jun 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
During nginx worker shutdown,
flush_bufferenters an infinite timer loop:flush_buffer(premature=true)— premature flag ignored, time condition not metcreate_buffer_timer()→timer_at(inactive_timeout)fails with "process exiting"timer_at(0, flush_buffer)succeeds → goto 1 (infinite loop)This prevents graceful shutdown and causes
[alert]logs flooding.Additionally,
schedule_func_exechas a bug where the fallbacktimer_at(0, execute_func, self)is missing thebatchargument, causing a nil crash whenretry_delay > 0during shutdown.Fix
1.
flush_buffer: Addpremature or exiting()to the existing time conditionprematurecatches the case where the original pending timer is aborted during shutdownexiting()(ngx.worker.exiting()) catches ALL shutdown cases, including zero-delay fallback timers fromcreate_buffer_timerwherepremature=false—timer_at(0, ...)during shutdown always deliverspremature=falsebecause the timer expires immediately and is never found byabort_pending_timers2.
schedule_func_exec: Passbatchin the fallback timerTest changes
t/utils/batch-processor.t: Added TEST 13 — regression test that hangs if the infinite loop exists (3600s timeouts ensure the time condition is never met normally)t/plugin/prometheus2.t: Added--- error_logassertions on TEST 25, 28, 31, 37 for specific batch-processor shutdown errors (batch-processor now correctly flushes during shutdown, producing expected connection-refused errors when upstream test servers are already stopped)t/plugin/prometheus3.t: Added--- error_logassertion on TEST 5 for batch-processor shutdown errors (error-log-logger flush during HUP worker shutdown)