Skip to content

Use Django DB max age connection setting (fixes #4116)#6134

Merged
auvipy merged 1 commit into
celery:masterfrom
whtsky:django_CONN_MAX_AGE
Jun 1, 2020
Merged

Use Django DB max age connection setting (fixes #4116)#6134
auvipy merged 1 commit into
celery:masterfrom
whtsky:django_CONN_MAX_AGE

Conversation

@whtsky

@whtsky whtsky commented Jun 1, 2020

Copy link
Copy Markdown
Contributor

Note: Before submitting this pull request, please review our contributing
guidelines
.

Description

Fixes #4116

#4292 makes Celery respect Django's CONN_MAX_AGE settings, but was reverted in #5515.

This PR tries to respect Django's CONN_MAX_AGE while force close db connections after fork.

@lgtm-com

lgtm-com Bot commented Jun 1, 2020

Copy link
Copy Markdown

This pull request introduces 5 alerts when merging 2db18a8 into 2479f95 - view on LGTM.com

new alerts:

  • 4 for Module is imported with 'import' and 'import from'
  • 1 for Non-exception in 'except' clause

@auvipy auvipy added this to the 4.4.x milestone Jun 1, 2020
@auvipy
auvipy merged commit f0c9b40 into celery:master Jun 1, 2020
@whtsky
whtsky deleted the django_CONN_MAX_AGE branch June 16, 2020 08:44
@auvipy

auvipy commented Jul 8, 2020

Copy link
Copy Markdown
Member

can you comment on #6216

danlamanna added a commit to danlamanna/celery that referenced this pull request Jul 15, 2025
This reverts commit f0c9b40.

This reverts PR celery#6134 and stops using the close_if_unusable_or_obsolete API since there are edge cases where it's
unable to detect if a connection if actually unusable. This is most obvious when Celery interrupts a query in
progress via a time limit handler. Django has marked this issue as
wontfix (https://code.djangoproject.com/ticket/30646). Since this is effectively an optimization for Celery that
can't be reliably used, Celery ought to close the connection after each task instead of trying to manage connections
in a way similar to how the Django application does.
danlamanna added a commit to danlamanna/celery that referenced this pull request Jul 21, 2025
This reverts commit f0c9b40.

This reverts PR celery#6134 and stops using the close_if_unusable_or_obsolete API since there are edge cases where it's
unable to detect if a connection if actually unusable. This is most obvious when Celery interrupts a query in
progress via a time limit handler. Django has marked this issue as
wontfix (https://code.djangoproject.com/ticket/30646). Since this is effectively an optimization for Celery that
can't be reliably used, Celery ought to close the connection after each task instead of trying to manage connections
in a way similar to how the Django application does.
danlamanna added a commit to danlamanna/celery that referenced this pull request Jul 21, 2025
This reverts commit f0c9b40.

This reverts PR celery#6134 and stops using the close_if_unusable_or_obsolete API since there are edge cases where it's
unable to detect if a connection if actually unusable. This is most obvious when Celery interrupts a query in
progress via a time limit handler. Django has marked this issue as
wontfix (https://code.djangoproject.com/ticket/30646). Since this is effectively an optimization for Celery that
can't be reliably used, Celery ought to close the connection after each task instead of trying to manage connections
in a way similar to how the Django application does.
auvipy pushed a commit to danlamanna/celery that referenced this pull request Aug 7, 2025
This reverts commit f0c9b40.

This reverts PR celery#6134 and stops using the close_if_unusable_or_obsolete API since there are edge cases where it's
unable to detect if a connection if actually unusable. This is most obvious when Celery interrupts a query in
progress via a time limit handler. Django has marked this issue as
wontfix (https://code.djangoproject.com/ticket/30646). Since this is effectively an optimization for Celery that
can't be reliably used, Celery ought to close the connection after each task instead of trying to manage connections
in a way similar to how the Django application does.
danlamanna added a commit to danlamanna/celery that referenced this pull request Aug 8, 2025
This reverts commit f0c9b40.

This reverts PR celery#6134 and stops using the close_if_unusable_or_obsolete API since there are edge cases where it's
unable to detect if a connection if actually unusable. This is most obvious when Celery interrupts a query in
progress via a time limit handler. Django has marked this issue as
wontfix (https://code.djangoproject.com/ticket/30646). Since this is effectively an optimization for Celery that
can't be reliably used, Celery ought to close the connection after each task instead of trying to manage connections
in a way similar to how the Django application does.
auvipy pushed a commit to danlamanna/celery that referenced this pull request Aug 9, 2025
This reverts commit f0c9b40.

This reverts PR celery#6134 and stops using the close_if_unusable_or_obsolete API since there are edge cases where it's
unable to detect if a connection if actually unusable. This is most obvious when Celery interrupts a query in
progress via a time limit handler. Django has marked this issue as
wontfix (https://code.djangoproject.com/ticket/30646). Since this is effectively an optimization for Celery that
can't be reliably used, Celery ought to close the connection after each task instead of trying to manage connections
in a way similar to how the Django application does.
auvipy pushed a commit that referenced this pull request Aug 10, 2025
* Revert "Use Django DB max age connection setting"

This reverts commit f0c9b40.

This reverts PR #6134 and stops using the close_if_unusable_or_obsolete API since there are edge cases where it's
unable to detect if a connection if actually unusable. This is most obvious when Celery interrupts a query in
progress via a time limit handler. Django has marked this issue as
wontfix (https://code.djangoproject.com/ticket/30646). Since this is effectively an optimization for Celery that
can't be reliably used, Celery ought to close the connection after each task instead of trying to manage connections
in a way similar to how the Django application does.

* Ensure django fixup never calls close_if_unusable_or_obsolete

This API can fail to close unusable connections in certain scenarios,
namely database failovers and ungraceful terminations (e.g. signal
handler for time limit exceeded tasks). This makes
close_if_unusable_or_obsolete adequate for HTTP request lifecycle
management but inappropriate for use within celery workers.

See also:
https://code.djangoproject.com/ticket/30646
https://forum.djangoproject.com/t/close-if-unusable-or-obsolete-fails-to-close-unusable-connections/41900

* Add test for close_cache
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Django celery fixup doesn't respect Django settings for PostgreSQL connections

2 participants