Add a concurrency model with ThreadPoolExecutor#5099
Conversation
Codecov Report
@@ Coverage Diff @@
## master #5099 +/- ##
=========================================
Coverage ? 84.02%
=========================================
Files ? 144
Lines ? 17426
Branches ? 2401
=========================================
Hits ? 14643
Misses ? 2585
Partials ? 198
Continue to review full report at Codecov.
|
thedrow
left a comment
There was a problem hiding this comment.
Generally speaking, this looks great.
I still have to review the unit tests more carefully.
In the meanwhile, there are a few more points I'd like you to address.
|
@thedrow perhaps the thread pool should be marked as experimental in the documentation? |
|
It hasn't been tested in Production so you might be right. |
|
I use this PR in production for several months now, it works correctly. |
Celery 4.3
Merge pull request #1 from celery/master
|
is there any plan to merge to master? or can I use this branch for experimentation at my side. I am acutally looking for execution pool with threads. Thanks, |
|
Yes we can merge this in my opinion. |
|
Hello, For what it worth, we use it in production for a year now, seamlessly. |
|
thanks a lot! |
|
Thanks a lot people. You guys are awesome.! |
* Fix serialization and deserialization of nested exception classes (#5717) * Fix #5597: chain priority (#5759) * adding `worker_process_shutdown` to __all__ (#5762) * Fix typo (#5769) * Reformat code. * Simplify commands to looking for celery worker processes (#5778) * update doc- celery supports storage list. (#5776) * Update introduction.rst * Update introduction.rst * Fail xfailed tests if the failure is unexpected. * Added integration coverage for link_error (#5373) * Added coverage for link_error. * Use pytest-rerunfailed plugin instead of rolling our own custom implementation. * Added link_error with retries. This currently fails. * Remove unused import. * Fix import on Python 2.7. * retries in link_error do not hang the worker anymore. * Run error callbacks eagerly when the task itself is run eagerly. Fixes #4899. * Adjust unit tests accordingly. * Grammar in documentation (#5780) * Grammar in documentation * Address review. * pypy 7.2 matrix (#5790) * removed extra slashes in CELERY_BROKER_URL (#5792) The Celery broker URL in settings.py had 2 slashes in the end which are not required and can be misleading. so I changed :- CELERY_BROKER_URL = 'amqp://guest:guest@localhost//' to CELERY_BROKER_URL = 'amqp://guest:guest@localhost' * Fix #5772 task_default_exchange & task_default_exchange_type not work (#5773) * Fix #5772 task_default_exchange & task_default_exchange_type not work * Add unit test: test_setting_default_exchange * Move default_exchange test to standalone class * Run integration suite with memcached results backend. (#5739) * Fix hanging forever when fetching results from a group(chain(group)) canvas. (#5744) PR #5739 uncovered multiple problems with the cache backend. This PR should resolve one of them. PR #5638 fixed the same test case for our async results backends that support native join. However, it did not fix the test case for sync results backends that support native join. * Fix regression in PR #5681. (#5753) See comment in the diff for details. * Grammatical fix to CONTRIBUTING.rst doc (#5794) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs (#5795) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs * Add unit test: test_get_connection_with_authmechanism * Add unit test: test_get_connection_with_authmechanism_no_username * Fix errors in Python 2.7 Remove "," after "**" operator * Revert "Revert "Revert "Added handle of SIGTERM in BaseTask in celery/task.py to prevent kill the task" (#5577)" (#5586)" (#5797) This reverts commit f79894e. * Add Python 3.8 Support (#5785) * Added Python 3.8 to the build matrix. * Ensure a supported tblib version is installed for Python 3.8 and above. In addition, modernize the relevant tests. * Workaround patching problem in test. * py 3.8 in clasifier * ubuntu bionic (#5799) * ubuntu bionic * fast finish * sync bumversion with pypi release * Dev.req (#5803) * update docker config * undo hardpin * devr req install from github master * update docker config (#5801) * update docker config * make dockerfile to install from github master dev branch by default * update download link * Isort. * Grammatical & punctuation fixes for CONTRIBUTING.rst document (#5804) * update dockerfile * switched to ubuntu bionic * update docker * keep it empty until we reconfigure it again with autopep8 * Fixed Dockerfile (#5809) * Update document CONTRIBUTING.rst & fix Dockerfile typo (#5813) * Added an issue template for minor releases. * reference gocelery Go Client/Server for Celery (#5815) * Add enterprise language (#5818) * Fix/correct minor doc typos (#5825) * Correct a small typo * Correct bad contributing documentation links * Preserve the task priority in case of a retry (#5820) * Preserve the task priority in case of a retry * Created test case for retried tasks with priority * Implement an integration test for retried tasks with priorities * bump kombu * basic changelog for celery 4.4.0rc4 * bump celery 4.4.0rc4 * events bootstep disabled if no events (#5807) * events bootstep disabled if no events * Added unit tests. * update bug report template * fixing ascii art to look nicer (#5831) * Only rerun flaky tests when failures can be intermediary. * Rename Changelog to Changelog.rst * The test_nested_group_chain test can run without native_join support. (#5838) * Run integration tests with Cassandra (#5834) * Run integration tests with Cassandra. * Configure cassandra result backend * Pre-create keyspace and table * Fix deprecation warning. * Fix path to cqlsh. * Increase connection timeout. * Wait until the cluster is available. * SQS - Reject on failure (#5843) * reject on failure * add documentation * test fix * test fix * test fix * Add a concurrency model with ThreadPoolExecutor (#5099) * Add a concurrency model with ThreadPoolExecutor * thread model test for pypy * Chain primitive's code example fix in canvas documentation (Regression PR#4444) (#5845) * Changed multi-line string (#5846) This string wasn't rendering properly and was printing the python statement too. Although the change isn't as pretty code-wise, it gets rid of an annoyance for the user. * Add auto expiry for DynamoDB backend (#5805) * Add auto expiry for DynamoDB backend This adds auto-expire support for the DynamoDB backend, via the DynamoDB Time to Live feature. * Require boto3>=1.9.178 for DynamoDB TTL support boto3 version 1.9.178 requires botocore>=1.12.178. botocore version 1.12.178 introduces support for the DynamoDB UpdateTimeToLive call. The UpdateTimeToLive call is used by the DynamoDB backend to enable TTL support on a newly created table. * Separate TTL handling from table creation Handle TTL enabling/disabling separately from the table get-or-create function. Improve handling of cases where the TTL is already set to the desired state. DynamoDB only allows a single TTL update action within a fairly long time window, so some problematic cases (changing the TTL attribute, enabling/disabling TTL when it was recently modified) will raise exceptions that have to be dealt with. * Handle older boto3 versions If the boto3 TTL methods are not found, log an informative error. If the user wants to enable TTL, raise an exception; if TTL should be disabled, simply return. * Improve logging - Handle exceptions by logging the error and re-raising - Log (level debug) when the desired TTL state is already in place * Add and use _has_ttl() convenience method Additional changes: - Handle exceptions when calling boto3's describe_time_to_live() - Fix test cases for missing TTL methods * Update ttl_seconds documentation * Log invalid TTL; catch and raise ValueError * Separate method _get_table_ttl_description * Separate ttl method validation function * Clarify tri-state TTL value * Improve test coverage * Fix minor typo in comment * Mark test as xfail when using the cache backend. (
Duplicate #5011