Checklist
Mandatory Debugging Information
Optional Debugging Information
Related Issues and Possible Duplicates
Related Issues
Possible Duplicates
Environment & Settings
Celery version:
celery report Output:
software -> celery:4.4.0 (cliffs) kombu:4.6.7 py:3.7.6
billiard:3.6.1.0 redis:3.3.11
platform -> system:Linux arch:64bit
kernel version:5.4.13-3-MANJARO imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://redis:6379/2
Steps to Reproduce
Required Dependencies
- Minimal Python Version: N/A or Unknown
- Minimal Celery Version: N/A or Unknown
- Minimal Kombu Version: N/A or Unknown
- Minimal Broker Version: N/A or Unknown
- Minimal Result Backend Version: N/A or Unknown
- Minimal OS and/or Kernel Version: N/A or Unknown
- Minimal Broker Client Version: N/A or Unknown
- Minimal Result Backend Client Version: N/A or Unknown
Python Packages
pip freeze Output:
amqp==2.5.2
apipkg==1.5
attrs==19.3.0
backcall==0.1.0
billiard==3.6.1.0
blinker==1.4
boltons==19.3.0
cachetools==4.0.0
celery==4.4.0
certifi==2019.11.28
cffi==1.13.2
chardet==3.0.4
Click==7.0
cryptography==2.8
decorator==4.4.1
defusedxml==0.6.0
dnspython==1.16.0
dotted==0.1.8
elasticsearch==7.1.0
elasticsearch-dsl==7.1.0
execnet==1.7.1
Flask==1.1.1
Flask-Caching==1.4.0
Flask-Cors==3.0.8
Flask-Limiter==1.1.0
Flask-Login==0.4.1
flask-mongoengine==0.9.5
flask-shell-ipython==0.4.1
Flask-WTF==0.14.2
gevent==1.4.0
google-api-core==1.15.0
google-auth==1.10.0
google-cloud-core==1.1.0
google-cloud-pubsub==1.1.0
google-cloud-storage==1.23.0
google-resumable-media==0.5.0
googleapis-common-protos==1.6.0
greenlet==0.4.15
grpc-google-iam-v1==0.12.3
grpcio==1.26.0
httpagentparser==1.9.0
httplib2==0.15.0
idna==2.8
importlib-metadata==1.4.0
ipython==7.10.2
ipython-genutils==0.2.0
itsdangerous==1.1.0
jedi==0.15.2
Jinja2==2.10.3
kombu==4.6.7
libthumbor==1.3.2
limits==1.3
MarkupSafe==1.1.1
marshmallow==3.3.0
mixpanel==4.5.0
mmh3==2.5.1
mongoengine==0.18.2
more-itertools==8.1.0
multidict==4.7.4
ndg-httpsclient==0.5.1
newrelic==5.4.1.134
nexmo==2.4.0
oauth2client==4.1.3
oauthlib==3.1.0
packaging==20.0
parso==0.5.2
pexpect==4.7.0
phonenumbers==8.11.1
pickleshare==0.7.5
Pillow-SIMD==6.0.0.post0
pluggy==0.13.1
prompt-toolkit==3.0.2
protobuf==3.9.0
ptyprocess==0.6.0
pusher==2.1.4
py==1.8.1
py-cpuinfo==5.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pybase62==0.4.3
pycparser==2.19
Pygments==2.5.2
PyJWT==1.7.1
pymongo==3.10.0
PyNaCl==1.3.0
pyOpenSSL==19.1.0
pyparsing==2.4.6
pytelegraf==0.3.3
pytest==5.3.2
pytest-benchmark==3.2.3
pytest-forked==1.1.3
pytest-mock==1.13.0
pytest-sugar==0.9.2
pytest-xdist==1.31.0
python-dateutil==2.8.1
python-rapidjson==0.9.1
python3-openid==3.1.0
pytz==2019.3
PyYAML==5.2
redis==3.3.11
requests==2.22.0
requests-oauthlib==1.3.0
rsa==4.0
semantic-version==2.8.3
sentry-sdk==0.13.5
six==1.13.0
social-auth-app-flask==1.0.0
social-auth-core==3.2.0
social-auth-storage-mongoengine==1.0.1
termcolor==1.1.0
traitlets==4.3.3
twilio==6.35.1
urllib3==1.25.7
uWSGI==2.0.18
vine==1.3.0
wcwidth==0.1.8
webargs==5.5.2
Werkzeug==0.15.5
WTForms==2.2.1
yarl==1.4.2
zipp==0.6.0
Other Dependencies
Details
N/A
Minimally Reproducible Test Case
Details
import time
from flask import Flask
from celery import Celery, group
app = Flask(__name__)
celery = Celery('app', broker='redis://redis:6379/1', backend='redis://redis:6379/2')
@celery.task
def debug():
return
@app.route('/', methods={'GET', 'POST'})
def hello_world():
task = group([
debug.si() for i in range(10)
]).apply_async()
start = time.perf_counter()
task.get(timeout=5, interval=0.01)
print('END', (time.perf_counter() - start) * 1000)
return {}
Expected Behavior
Scheduling noop tasks and setting interval should make the response time near the set interval in ideal clean environments.
Example: Setting task.get(interval=0.1) with 5x noop tasks, I would expect near 100ms response.
Actual Behavior
Regardless of the setting of interval, the response time is at least 500ms.
The cause is that interval is not passed all the way to get_many() where it defaults to 500ms which is where the minimum latency is comming from.
|
def _iter_meta(self): |
|
return (meta for _, meta in self.backend.get_many( |
|
{r.id for r in self.results}, max_iterations=1, |
|
)) |
|
def get_many(self, task_ids, timeout=None, interval=0.5, no_ack=True, |
|
on_message=None, on_interval=None, max_iterations=None, |
|
READY_STATES=states.READY_STATES): |
|
interval = 0.5 if interval is None else interval |
Checklist
I have verified that the issue exists against the
masterbranch of Celery. 90fe53fThis has already been asked to the discussion group first.
I have read the relevant section in the
contribution guide
on reporting bugs.
I have checked the issues list
for similar or identical bug reports.
I have checked the pull requests list
for existing proposed fixes.
I have checked the commit log
to find out if the bug was already fixed in the master branch.
I have included all related issues and possible duplicate issues
in this issue (If there are none, check this box anyway).
Mandatory Debugging Information
celery -A proj reportin the issue.(if you are not able to do this, then at least specify the Celery
version affected).
masterbranch of Celery.pip freezein the issue.to reproduce this bug.
Optional Debugging Information
and/or implementation.
result backend.
broker and/or result backend.
ETA/Countdown & rate limits disabled.
and/or upgrading Celery and its dependencies.
Related Issues and Possible Duplicates
Related Issues
Possible Duplicates
Environment & Settings
Celery version:
celery reportOutput:Steps to Reproduce
Required Dependencies
Python Packages
pip freezeOutput:Other Dependencies
Details
N/A
Minimally Reproducible Test Case
Details
Expected Behavior
Scheduling noop tasks and setting
intervalshould make the response time near the setintervalin ideal clean environments.Example: Setting
task.get(interval=0.1)with 5x noop tasks, I would expect near100msresponse.Actual Behavior
Regardless of the setting of
interval, the response time is at least 500ms.The cause is that
intervalis not passed all the way toget_many()where it defaults to500mswhich is where the minimum latency is comming from.celery/celery/result.py
Lines 837 to 840 in cf82930
celery/celery/backends/base.py
Lines 663 to 666 in dc03b6d