Skip to content

Add a concurrency model with ThreadPoolExecutor#5099

Merged
auvipy merged 5 commits into
celery:masterfrom
alfred-sa:threadpool
Nov 24, 2019
Merged

Add a concurrency model with ThreadPoolExecutor#5099
auvipy merged 5 commits into
celery:masterfrom
alfred-sa:threadpool

Conversation

@alfred-sa

Copy link
Copy Markdown
Contributor

Duplicate #5011

@codecov

codecov Bot commented Oct 8, 2018

Copy link
Copy Markdown

Codecov Report

❗ No coverage uploaded for pull request base (master@52d6343). Click here to learn what that means.
The diff coverage is 81.81%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master    #5099   +/-   ##
=========================================
  Coverage          ?   84.02%           
=========================================
  Files             ?      144           
  Lines             ?    17426           
  Branches          ?     2401           
=========================================
  Hits              ?    14643           
  Misses            ?     2585           
  Partials          ?      198
Impacted Files Coverage Δ
celery/concurrency/__init__.py 8.33% <0%> (ø)
celery/concurrency/thread.py 96.42% <96.42%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 52d6343...9887592. Read the comment docs.

@thedrow thedrow left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally speaking, this looks great.
I still have to review the unit tests more carefully.
In the meanwhile, there are a few more points I'd like you to address.

Comment thread t/unit/concurrency/test_thread.py Outdated
Comment thread docs/getting-started/introduction.rst
@thedrow thedrow added this to the v4.3 milestone Oct 9, 2018
@georgepsarakis

Copy link
Copy Markdown
Contributor

@thedrow perhaps the thread pool should be marked as experimental in the documentation?

@thedrow

thedrow commented Dec 16, 2018

Copy link
Copy Markdown
Contributor

It hasn't been tested in Production so you might be right.

@auvipy auvipy modified the milestones: v4.3, v5.0.0 Feb 1, 2019
@alfred-sa

Copy link
Copy Markdown
Contributor Author

I use this PR in production for several months now, it works correctly.

Merge pull request #1 from celery/master
@dduraipandian

dduraipandian commented Nov 23, 2019

Copy link
Copy Markdown

is there any plan to merge to master? or can I use this branch for experimentation at my side. I am acutally looking for execution pool with threads.

https://github.com/whuji/celery/tree/threadpool

Thanks,
Durai Pandian

@auvipy auvipy modified the milestones: v5.0.0, 4.5 Nov 23, 2019
@thedrow

thedrow commented Nov 24, 2019

Copy link
Copy Markdown
Contributor

Yes we can merge this in my opinion.

@auvipy auvipy closed this Nov 24, 2019
@auvipy auvipy reopened this Nov 24, 2019
@auvipy auvipy modified the milestones: 4.5, 4.4.0 Nov 24, 2019
@auvipy
auvipy merged commit f6bb847 into celery:master Nov 24, 2019
@alfred-sa

Copy link
Copy Markdown
Contributor Author

Hello,

For what it worth, we use it in production for a year now, seamlessly.

@auvipy

auvipy commented Nov 24, 2019

Copy link
Copy Markdown
Member

thanks a lot!

@dduraipandian

Copy link
Copy Markdown

Thanks a lot people. You guys are awesome.!

auvipy added a commit that referenced this pull request Feb 26, 2020
* Fix serialization and deserialization of nested exception classes (#5717)

* Fix #5597: chain priority (#5759)

* adding `worker_process_shutdown` to __all__ (#5762)

* Fix typo (#5769)

* Reformat code.

* Simplify commands to looking for celery worker processes (#5778)

* update doc- celery supports storage list. (#5776)

* Update introduction.rst

* Update introduction.rst

* Fail xfailed tests if the failure is unexpected.

* Added integration coverage for link_error (#5373)

* Added coverage for link_error.

* Use pytest-rerunfailed plugin instead of rolling our own custom implementation.

* Added link_error with retries. This currently fails.

* Remove unused import.

* Fix import on Python 2.7.

* retries in link_error do not hang the worker anymore.

* Run error callbacks eagerly when the task itself is run eagerly.

Fixes #4899.

* Adjust unit tests accordingly.

* Grammar in documentation (#5780)

* Grammar in documentation

* Address review.

* pypy 7.2 matrix (#5790)

* removed extra slashes in CELERY_BROKER_URL (#5792)

The Celery broker URL in settings.py had 2 slashes in the end which are not required and can be misleading.
so I changed :-
CELERY_BROKER_URL = 'amqp://guest:guest@localhost//' to CELERY_BROKER_URL = 'amqp://guest:guest@localhost'

* Fix #5772 task_default_exchange & task_default_exchange_type not work (#5773)

* Fix #5772 task_default_exchange & task_default_exchange_type not work

* Add unit test: test_setting_default_exchange

* Move default_exchange test to standalone class

* Run integration suite with memcached results backend. (#5739)

* Fix hanging forever when fetching results from a group(chain(group)) canvas. (#5744)

PR #5739 uncovered multiple problems with the cache backend.
This PR should resolve one of them.

PR #5638 fixed the same test case for our async results backends that support native join.
However, it did not fix the test case for sync results backends that support native join.

* Fix regression in PR #5681. (#5753)

See comment in the diff for details.

* Grammatical fix to CONTRIBUTING.rst doc (#5794)

* Fix #5734 Celery does not consider authMechanism on mongodb backend URLs (#5795)

* Fix #5734 Celery does not consider authMechanism on mongodb backend URLs

* Add unit test: test_get_connection_with_authmechanism

* Add unit test: test_get_connection_with_authmechanism_no_username

* Fix errors in Python 2.7

Remove "," after "**" operator

* Revert "Revert "Revert "Added handle of SIGTERM in BaseTask in celery/task.py to prevent kill the task" (#5577)" (#5586)" (#5797)

This reverts commit f79894e.

* Add Python 3.8 Support (#5785)

* Added Python 3.8 to the build matrix.

* Ensure a supported tblib version is installed for Python 3.8 and above.

In addition, modernize the relevant tests.

* Workaround patching problem in test.

* py 3.8 in clasifier

* ubuntu bionic (#5799)

* ubuntu bionic

* fast finish

* sync bumversion with pypi release

* Dev.req (#5803)

* update  docker config



* undo hardpin

* devr req install from github master

* update  docker config (#5801)

* update  docker config

* make dockerfile to install from github master dev branch by default

* update download link

* Isort.

* Grammatical & punctuation fixes for CONTRIBUTING.rst document (#5804)

* update dockerfile

* switched to ubuntu bionic

* update docker

* keep it empty until we reconfigure it again with autopep8

* Fixed Dockerfile (#5809)

* Update document CONTRIBUTING.rst & fix Dockerfile typo (#5813)

* Added an issue template for minor releases.

* reference gocelery Go Client/Server for Celery (#5815)

* Add enterprise language (#5818)

* Fix/correct minor doc typos (#5825)

* Correct a small typo

* Correct bad contributing documentation links

* Preserve the task priority in case of a retry (#5820)

* Preserve the task priority in case of a retry

* Created test case for retried tasks with priority

* Implement an integration test for retried tasks with priorities

* bump kombu

* basic changelog for celery 4.4.0rc4

* bump celery 4.4.0rc4

* events bootstep disabled if no events (#5807)

* events bootstep disabled if no events

* Added unit tests.

* update bug report template

* fixing ascii art to look nicer (#5831)

* Only rerun flaky tests when failures can be intermediary.

* Rename Changelog to Changelog.rst

* The test_nested_group_chain test can run without native_join support. (#5838)

* Run integration tests with Cassandra (#5834)

* Run integration tests with Cassandra.

* Configure cassandra result backend

* Pre-create keyspace and table

* Fix deprecation warning.

* Fix path to cqlsh.

* Increase connection timeout.

* Wait until the cluster is available.

* SQS - Reject on failure (#5843)

* reject on failure

* add documentation

* test fix

* test fix

* test fix

* Add a concurrency model with ThreadPoolExecutor (#5099)

* Add a concurrency model with ThreadPoolExecutor

* thread model test for pypy

* Chain primitive's code example fix in canvas documentation (Regression PR#4444) (#5845)

* Changed multi-line string (#5846)

This string wasn't rendering properly and was printing the python statement too. Although the change isn't as pretty code-wise, it gets rid of an annoyance for the user.

* Add auto expiry for DynamoDB backend (#5805)

* Add auto expiry for DynamoDB backend

This adds auto-expire support for the DynamoDB backend, via the DynamoDB
Time to Live feature.

* Require boto3>=1.9.178 for DynamoDB TTL support

boto3 version 1.9.178 requires botocore>=1.12.178.

botocore version 1.12.178 introduces support for the DynamoDB
UpdateTimeToLive call.

The UpdateTimeToLive call is used by the DynamoDB backend to enable TTL
support on a newly created table.

* Separate TTL handling from table creation

Handle TTL enabling/disabling separately from the table get-or-create
function.

Improve handling of cases where the TTL is already set to the desired
state.

DynamoDB only allows a single TTL update action within a fairly long
time window, so some problematic cases (changing the TTL attribute,
enabling/disabling TTL when it was recently modified) will raise
exceptions that have to be dealt with.

* Handle older boto3 versions

If the boto3 TTL methods are not found, log an informative error. If the
user wants to enable TTL, raise an exception; if TTL should be disabled,
simply return.

* Improve logging

- Handle exceptions by logging the error and re-raising

- Log (level debug) when the desired TTL state is already in place

* Add and use _has_ttl() convenience method

Additional changes:

- Handle exceptions when calling boto3's describe_time_to_live()

- Fix test cases for missing TTL methods

* Update ttl_seconds documentation

* Log invalid TTL; catch and raise ValueError

* Separate method _get_table_ttl_description

* Separate ttl method validation function

* Clarify tri-state TTL value

* Improve test coverage

* Fix minor typo in comment

* Mark test as xfail when using the cache backend. (