Skip to content

TypeError raised at AsyncResult destruction time #6164

Closed
#10406
Closed
TypeError raised at AsyncResult destruction time#6164
#10406
Labels
Category: Results BackendIssue Type: Bug ReportStatus: Confirmed ✔Status: Has Testcase ✔
Milestone
5.7.0

Description

@lelit
lelit
opened on Jun 9, 2020

Checklist

Mandatory Debugging Information

  • I have included the output of celery -A proj report in the issue.
    (if you are not able to do this, then at least specify the Celery
    version affected).
  • I have verified that the issue exists against the master branch of Celery.
  • I have included the contents of pip freeze in the issue.
  • I have included all the versions of all the external dependencies required
    to reproduce this bug.

Optional Debugging Information

  • I have tried reproducing the issue on more than one Python version
    and/or implementation.
  • I have tried reproducing the issue on more than one message broker and/or
    result backend.
  • I have tried reproducing the issue on more than one version of the message
    broker and/or result backend.
  • I have tried reproducing the issue on more than one operating system.
  • I have tried reproducing the issue on more than one workers pool.
  • I have tried reproducing the issue with autoscaling, retries,
    ETA/Countdown & rate limits disabled.
  • I have tried reproducing the issue after downgrading
    and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

  • None

Possible Duplicates

  • None

Environment & Settings

Celery version:

celery report Output: 

software -> celery:4.4.5 (cliffs) kombu:4.6.10 py:3.8.3
            billiard:3.6.3.0 redis:3.5.3
platform -> system:Linux arch:64bit
            kernel version:5.6.14 imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://redis:6379/0

broker_url: 'redis://redis:6379/0'
result_backend: 'redis://redis:6379/0'
include: ['etour.hopi.celerer.tasks',
 'etour.hopi.renderer.tasks',
 'etour.hopi.slurper.tasks']

Steps to Reproduce

Required Dependencies

  • Minimal Python Version: N/A or Unknown
  • Minimal Celery Version: I noticed this first in Celery 4.4.2
  • Minimal Kombu Version: N/A or Unknown
  • Minimal Broker Version: N/A or Unknown
  • Minimal Result Backend Version: N/A or Unknown
  • Minimal OS and/or Kernel Version: N/A or Unknown
  • Minimal Broker Client Version: N/A or Unknown
  • Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output: 

amqp==2.6.0
appdirs==1.4.4
attrs==19.3.0
Babel==2.8.0
beautifulsoup4==4.9.1
billiard==3.6.3.0
blessed==1.17.8
cached-property==1.5.1
celery==4.4.5
certifi==2020.4.5.2
Chameleon==3.7.2
chardet==3.0.4
cli-helpers==2.0.1
click==7.1.2
colander==1.7.0
colorlog==4.1.0
configobj==5.0.6
coverage==5.1
deform==2.0.8
defusedxml==0.6.0
enlighten==1.5.2
# Editable install with no version control (etour.hopi.app==1.8)
-e /usr/src/app/src
# Editable install with no version control (etour.hopi.celerer==1.7)
-e /usr/src/celerer/src
# Editable install with no version control (etour.hopi.model==1.7)
-e /usr/src/model/src
# Editable install with no version control (etour.hopi.ptp==1.8)
-e /usr/src/ptp/src
# Editable install with no version control (etour.hopi.renderer==1.7)
-e /usr/src/renderer/src
# Editable install with no version control (etour.hopi.slurper==1.7)
-e /usr/src/slurper/src
future==0.18.2
geographiclib==1.50
geojson==2.5.0
geopy==1.22.0
humanize==2.4.0
hupper==1.10.2
idna==2.9
iso8601==0.1.12
isodate==0.6.0
itsdangerous==1.1.0
Jinja2==2.11.2
kombu==4.6.10
lxml==4.5.1
MarkupSafe==1.1.1
metapensiero.deform.semantic-ui==0.8
metapensiero.markup.semtext==0.24
metapensiero.sphinx.patchdb==3.7
metapensiero.sqlalchemy.dbloady==2.10
metapensiero.sqlalchemy.proxy==5.14
more-itertools==8.3.0
packaging==20.4
PasteDeploy==2.1.0
pathtools==0.1.2
peppercorn==0.6
pgcli @ https://github.com/dbcli/pgcli/archive/master.zip
pglast==1.12
pgspecial==1.11.10
Pillow==7.1.2
plaster==1.0
plaster-pastedeploy==0.7
pluggy==0.13.1
progressbar2==3.51.3
prompt-toolkit==3.0.5
psycopg2 @ file:///tmp/psycopg2-2.8.5-cp38-cp38-linux_x86_64.whl
py==1.8.1
pycountry==19.8.18
Pygments==2.6.1
pyowm==3.0.0
pyparsing==2.4.7
Pyphen==0.9.5
pyramid==1.10.4
pyramid-chameleon==0.3
pyramid-jinja2==2.8
pyramid-tm==2.4
pyRXP==2.2.0
PySocks==1.7.1
pytest==5.4.3
python-rapidjson==0.9.1
python-utils==2.4.0
pytz==2020.1
redis==3.5.3
reportlab==3.5.42
requests==2.23.0
requests-toolbelt==0.9.1
ruamel.yaml==0.16.10
ruamel.yaml.clib==0.2.0
setproctitle @ file:///tmp/setproctitle-1.1.10-cp38-cp38-linux_x86_64.whl
six==1.15.0
sly==0.4
soupsieve==2.0.1
SQLAlchemy @ file:///tmp/SQLAlchemy-1.3.17-cp38-cp38-manylinux2010_x86_64.whl
sqlparse==0.3.1
tabulate==0.8.7
terminaltables==3.1.0
toposort==1.5
transaction==3.0.0
translationstring==1.3
urllib3==1.25.9
venusian==3.0.0
vine==1.3.0
waitress==1.4.4
watchdog==0.10.2
wcwidth==0.2.4
WebOb==1.8.6
WebTest==2.0.35
XlsxWriter==1.2.9
zeep==3.4.0
zope.deprecation==4.4.0
zope.interface==5.1.0
zope.sqlalchemy==1.3

Other Dependencies

Details

N/A

Minimally Reproducible Test Case

Details 







Expected Behavior



Actual Behavior



Exception ignored in:
<function AsyncResult.__del__ at 0x7fc30a8640d0>
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/celery/result.py", line 389, in __del__
self.backend.remove_pending_result(self)
File "/usr/local/lib/python3.8/site-packages/celery/backends/asynchronous.py", line 187, in remove_pending_result
self.on_result_fulfilled(result)
File "/usr/local/lib/python3.8/site-packages/celery/backends/asynchronous.py", line 195, in on_result_fulfilled
self.result_consumer.cancel_for(result.id)
File "/usr/local/lib/python3.8/site-packages/celery/backends/redis.py", line 180, in cancel_for
key = self._get_key_for_task(task_id)
File "/usr/local/lib/python3.8/site-packages/celery/backends/base.py", line 753, in get_key_for_task
return key_t('').join([
TypeError
:
sequence item 1: expected a bytes-like object, UUID found



As reported on mailing list, I was able to get rid of the error by installing the following monkey patches:


from celery.backends.base import BaseKeyValueStoreBackend as BKVSB  # noqa


def get_key_for_task(self, task_id, key=b''):
    """Get the cache key for a task by id."""
    key_t = self.key_t
    return (self.task_keyprefix
            + key_t(str(task_id) if isinstance(task_id, UUID) else task_id)
            + key_t(key))


def get_key_for_group(self, group_id, key=b''):  # pragma: no cover
    """Get the cache key for a group by id."""
    key_t = self.key_t
    return (self.group_keyprefix
            + key_t(str(group_id) if isinstance(group_id, UUID) else group_id)
            + key_t(key))


def get_key_for_chord(self, group_id, key=b''):  # pragma: no cover
    """Get the cache key for the chord waiting on group with given id."""
    key_t = self.key_t
    return (self.chord_keyprefix
            + key_t(str(group_id) if isinstance(group_id, UUID) else group_id)
            + key_t(key))


BKVSB.get_key_for_task = get_key_for_task
BKVSB.get_key_for_group = get_key_for_group
BKVSB.get_key_for_chord = get_key_for_chord
del BKVSB, get_key_for_task, get_key_for_group, get_key_for_chord
Metadata
Metadata
Assignees
No one assigned
    Labels
    Category: Results BackendIssue Type: Bug ReportStatus: Confirmed ✔Status: Has Testcase ✔
    Type
    No type
    Projects
    No projects
    Milestone
    Relationships
    None yet
    Development
    No branches or pull requests
    Issue actions