Skip to content

[FilesystemBackend] Failure in unpickling a FilesystemBackend instance for a group #6108

Closed
Closed
[FilesystemBackend] Failure in unpickling a FilesystemBackend instance for a group#6108
Labels
Issue Type: Bug Report
Milestone
4.4.x

Description

@singlaive
singlaive
opened on May 18, 2020

Checklist

  • [ x] I have verified that the issue exists against the master branch of Celery.
  • This has already been asked to the discussion group first.
  • [x ] I have read the relevant section in the
    contribution guide
    on reporting bugs.
  • [x ] I have checked the issues list
    for similar or identical bug reports.
  • I have checked the pull requests list
    for existing proposed fixes.
  • I have checked the commit log
    to find out if the bug was already fixed in the master branch.
  • I have included all related issues and possible duplicate issues
    in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

  • I have included the output of celery -A proj report in the issue.
    (if you are not able to do this, then at least specify the Celery
    version affected).
  • I have verified that the issue exists against the master branch of Celery.
  • I have included the contents of pip freeze in the issue.
  • I have included all the versions of all the external dependencies required
    to reproduce this bug.

Optional Debugging Information

  • I have tried reproducing the issue on more than one Python version
    and/or implementation.
  • I have tried reproducing the issue on more than one message broker and/or
    result backend.
  • I have tried reproducing the issue on more than one version of the message
    broker and/or result backend.
  • I have tried reproducing the issue on more than one operating system.
  • I have tried reproducing the issue on more than one workers pool.
  • I have tried reproducing the issue with autoscaling, retries,
    ETA/Countdown & rate limits disabled.
  • I have tried reproducing the issue after downgrading
    and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

  • None

Possible Duplicates

  • None

Environment & Settings

Celery version:

celery report Output: 

software -> celery:4.4.2 (cliffs) kombu:4.6.8 py:3.7.5
            billiard:3.6.3.0 py-amqp:2.5.2
platform -> system:Darwin arch:64bit
            kernel version:18.7.0 imp:CPython
loader   -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled

Steps to Reproduce

Required Dependencies

  • Minimal Python Version: N/A or Unknown
  • Minimal Celery Version: N/A or Unknown
  • Minimal Kombu Version: N/A or Unknown
  • Minimal Broker Version: N/A or Unknown
  • Minimal Result Backend Version: N/A or Unknown
  • Minimal OS and/or Kernel Version: N/A or Unknown
  • Minimal Broker Client Version: N/A or Unknown
  • Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

Other Dependencies

Details

N/A

Minimally Reproducible Test Case

Details 







Expected Behavior



Backend: file system

steriliser: pickle


Scenario: Define a group and retrieve the result, for instance, group([add.s(1,2), add.s(3,4)])().get()

This should give the correct result of two adding calculation


Actual Behavior




_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
venv3/lib/python3.7/site-packages/celery/result.py:229: in get
    on_message=on_message,
venv3/lib/python3.7/site-packages/celery/backends/base.py:579: in wait_for_pending
    no_ack=no_ack,
venv3/lib/python3.7/site-packages/celery/backends/base.py:606: in wait_for
    meta = self.get_task_meta(task_id)
venv3/lib/python3.7/site-packages/celery/backends/base.py:457: in get_task_meta
    meta = self._get_task_meta_for(task_id)
venv3/lib/python3.7/site-packages/celery/backends/base.py:791: in _get_task_meta_for
    return self.decode_result(meta)
venv3/lib/python3.7/site-packages/celery/backends/base.py:322: in decode_result
    return self.meta_from_decoded(self.decode(payload))
venv3/lib/python3.7/site-packages/celery/backends/base.py:333: in decode
    accept=self.accept)
venv3/lib/python3.7/site-packages/kombu/serialization.py:264: in loads
    return decode(data)
/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py:130: in __exit__
    self.gen.throw(type, value, traceback)
venv3/lib/python3.7/site-packages/kombu/serialization.py:54: in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
venv3/lib/python3.7/site-packages/vine/five.py:194: in reraise
    raise value.with_traceback(tb)
venv3/lib/python3.7/site-packages/kombu/serialization.py:50: in _reraise_errors
    yield
venv3/lib/python3.7/site-packages/kombu/serialization.py:264: in loads
    return decode(data)
venv3/lib/python3.7/site-packages/kombu/serialization.py:338: in unpickle
    return pickle_loads(str_to_bytes(s))
venv3/lib/python3.7/site-packages/kombu/serialization.py:59: in pickle_loads
    return load(BytesIO(s))
venv3/lib/python3.7/site-packages/celery/backends/base.py:69: in unpickle_backend
    return cls(*args, app=current_app._get_current_object(), **kwargs)
venv3/lib/python3.7/site-packages/celery/backends/filesystem.py:53: in __init__
    path = self._find_path(url)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <celery.backends.filesystem.FilesystemBackend object at 0x10b2c7c90>, url = None

    def _find_path(self, url):
        if not url:
>           raise ImproperlyConfigured(E_NO_PATH_SET)
E           kombu.exceptions.DecodeError: You need to configure a path for the file-system backend

venv3/lib/python3.7/site-packages/celery/backends/filesystem.py:77: DecodeError



Solution


FilesystemBackend class defined in https://github.com/celery/celery/blob/master/celery/backends/filesystem.py#L34, misses __reduce__ method to pickle the instance properly. The following example would make it work.


def __reduce__(self, args=(), kwargs={}):
    kwargs.update(
        dict(url=self.url))
    return super(FilesystemBackend, self).__reduce__(args, kwargs)

Metadata
Metadata
Assignees
No one assigned
    Labels
    Issue Type: Bug Report
    Type
    No type
    Projects
    No projects
    Milestone
    Relationships
    None yet
    Development
    No branches or pull requests
    Issue actions