Repository navigation
Commit 68c5d54
fix: Keep the metrics server running when a Gunicorn worker exits (#7012)
* fix: Keep the metrics server running when a Gunicorn worker exits
Since the metrics server runs in a multiprocessing.Process started in
the Gunicorn master, every forked worker inherits multiprocessing's
registry of child processes. At worker exit,
multiprocessing.util._exit_function terminates every daemon process in
that registry, so the first worker restart (--max-requests, timeout,
reload) sends SIGTERM to the master's metrics server and /metrics stops
responding. It also logs "AssertionError: can only join a child
process" on every worker exit.
Register an at-fork hook when the metrics process is started that
removes it from the registry in every forked child. Workers no longer
terminate it; the master still does on its own exit.
Fixes #7011
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Alexandr Borgatin <[email protected]>
* test: Wait for the metrics server before forking in the worker-exit test
On macOS CI the spawned metrics process can take longer than the 5 s
readiness wait to start listening, so the first /metrics request was
refused. Poll /metrics (up to 30 s) before forking the child.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Alexandr Borgatin <[email protected]>
* test: Use an instant stand-in process in the worker-exit test
The spawned metrics server did not come up within 30 s inside the full
unit suite on macOS CI. The test is about the process surviving a
forked child's exit, not about serving HTTP, so register a fork-context
daemon process via start_metrics_server instead and assert it is still
alive after the child runs multiprocessing.util._exit_function().
Without the fix it is terminated (exitcode -SIGTERM).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Alexandr Borgatin <[email protected]>
* test: Check active_children() in the forked child instead of running _exit_function
Running multiprocessing.util._exit_function() in a fork of the pytest
process runs every registered finalizer. Check what a worker's exit
would act on instead: the metrics process must not be in the child's
active_children(). Also cover _forget_metrics_process_in_child directly,
and kill the stand-in process in cleanup (terminate() can wait out the
join timeout under xdist).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Alexandr Borgatin <[email protected]>
---------
Signed-off-by: Alexandr Borgatin <[email protected]>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>1 parent 1299e4e commit 68c5d54
2 files changed
Lines changed: 89 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
681 | 681 | | |
682 | 682 | | |
683 | 683 | | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
684 | 705 | | |
685 | 706 | | |
686 | 707 | | |
| |||
714 | 735 | | |
715 | 736 | | |
716 | 737 | | |
717 | | - | |
| 738 | + | |
718 | 739 | | |
719 | 740 | | |
720 | 741 | | |
| |||
739 | 760 | | |
740 | 761 | | |
741 | 762 | | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
742 | 767 | | |
743 | 768 | | |
744 | 769 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| 16 | + | |
16 | 17 | | |
17 | 18 | | |
| 19 | + | |
18 | 20 | | |
19 | 21 | | |
20 | 22 | | |
| |||
642 | 644 | | |
643 | 645 | | |
644 | 646 | | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
| 705 | + | |
| 706 | + | |
| 707 | + | |
645 | 708 | | |
646 | 709 | | |
647 | 710 | | |
| |||
0 commit comments