good start, from first skimming

@celery/technical-board The first round of reviews can begin.
I've described most of the publisher side.

If you have any ideas, comments, thoughts etc. I'd love some input.

@njsmith

I was thinking about trio/ayncio based concurrency in celery 4.5 with or without multiprocessing as an experimental feature only for celery 4.5 and python 3.5, what yo guys think? and some basic async compatibility in 4.5 task-based workflows?

I'm not sure it's possible given the fact that we will have to duplicate all of our code for sync/async paths.

OK

How much duplication would there be? Remember django is undergoing something similar in 3.x

How much duplication would there be? Remember django is undergoing something similar in 3.x

I will check that after 4.4 stable release

@jheld:

How much duplication would there be? Remember django is undergoing something similar in 3.x

A pretty good amount. The I/O and logic layers in Celery aren't really split properly to allow this change. I believe that the internals could be refactored to allow it, but it would likely be a lot of work. I've played with this a bit in https://github.com/clokep/twistedcelery. There's also a short write-up of how I came to the conclusion that this was the way forward for my use-case. (This example is for Twisted, but I think the idea is similar for asyncio/trio.)

can you guys check this goodboy/tractor#83

@auvipy is the question/interest around the work via trio?

my first question (as someone who has never used trio or the other task queue libraries mentiones there), is related to code/design sharing? advantages/disadvantages with how celery would use/compare with what they do.

Trio uses structured concurrency concepts and allows configuring timeouts to any action, limit coroutines/threads capacity or grouping tasks in nurseries which ensures all tasks are complete before proceeding among other things.
It's much easier to reason about such code.

Tractor is an implementation that allows us to use multiprocessing and trio which is useful for implementing workers.

@thedrow @auvipy can we get a document which is more representative of celery 4.5, 4.6 plans? celery 5 definitely will be py3 only (and eyeing asyncio/async libs likely, depending on timing), and it's trying to do a lot of other redesigns. Are there pieces we agree can be done before the py3 upgrade? Should those be CEP on their own?

It just would be great to button up some of the remaining nice/powerful improvements that py27 can also allow, and then cut the rope to get us to the future (giving some older projects something nice but not looking back anymore). Is there a document for any of that? If not, we should create that so we can clip off py27 in a nice, concrete, way. Even though it's "nice" of us to support py27 (or really, anything before any big redesigns), it's holding back the project.

@thedrow @auvipy can we get a document which is more representative of celery 4.5, 4.6 plans? celery 5 definitely will be py3 only (and eyeing asyncio/async libs likely, depending on timing), and it's trying to do a lot of other redesigns. Are there pieces we agree can be done before the py3 upgrade? Should those be CEP on their own?

It just would be great to button up some of the remaining nice/powerful improvements that py27 can also allow, and then cut the rope to get us to the future (giving some older projects something nice but not looking back anymore). Is there a document for any of that? If not, we should create that so we can clip off py27 in a nice, concrete, way. Even though it's "nice" of us to support py27 (or really, anything before any big redesigns), it's holding back the project.

that's a good point!!! we planed to support 2.7 as long as we can and I get some sponsor. I have plans to implement some of the celery 5 things to celery 4.5,4.6 which will not be breaking changes. like amqp 1.0 etc....

@auvipy glad to hear it.

Very psyched to try out amqp 1.0 (hoping for AWS' ActiveMQ support). Although I do wonder if redis streams would be another worthwhile implementation, but it would be...certainly not AMQP, very much just queues, I would think.

So we think we can implement AMQP 1.0 (and keeping 0.9.1 (did I get that right?) support, too?) before v5?

How close do we think we're feeling with the planned PRs for 4.5? 16 open PRs, though I wonder how many of those are actually serious contenders for 4.5

If this is the wrong spot for this discussion please let me know.

@auvipy glad to hear it.

Very psyched to try out amqp 1.0 (hoping for AWS' ActiveMQ support). Although I do wonder if redis streams would be another worthwhile implementation, but it would be...certainly not AMQP, very much just queues, I would think.

So we think we can implement AMQP 1.0 (and keeping 0.9.1 (did I get that right?) support, too?) before v5?

How close do we think we're feeling with the planned PRs for 4.5? 16 open PRs, though I wonder how many of those are actually serious contenders for 4.5

If this is the wrong spot for this discussion please let me know.

you should keep me pushing on that front :D Omer is too busy with c5 stuff at Bloomberg.

I have a lot of backlog for 4.5: 1. amqp 1.0 keeping 0.9.1 intact

btw @jheld it would be great if you could convince your employer to sponsor so of my time for celery it would be awesome. money is a great motivation for me If I get that for open source work, no matter how small it is.

@auvipy via the opencollective/celery, right?

@auvipy via the opencollective/celery, right?

or direct payment to me via my EU bank account in Belgium :) as EURO. you can shot me a line at [email protected] for more detail. :)

@auvipy I also lost access to most of the repositories.
Hopefully by accident.

@auvipy I also lost access to most of the repositories.
Hopefully by accident.

I will check. it seems you didnt join the core developer team! i am going to invite you

@auvipy I also lost access to most of the repositories.
Hopefully by accident.

it's due to the confusing github role management stuff i believe but you should have it all again. check and let me know

@auvipy I also lost access to most of the repositories.
Hopefully by accident.

it's due to the confusing github role management stuff i believe but you should have it all again. check and let me know

It works. Thank you.
We do need to chat soon. Please e-mail me.

We should review OPA for configuring Celery and enforcing different policies.

The build is failing because the website is down.

Hi everyone.

I'm merging this PR but this does not mean that this CEP has been accepted.
Upon reading it, it is way too detailed to provide a high-level overview of the architecture of Celery NextGen.

It also seems that we have a lot of ideas and lots of new features to implement. Therefore, we require multiple major versions to have them all released.

I'm going to break the document into multiple documents (= multiple PRs).
In the meanwhile, this document will remain in the drafts folder until we no longer need it.

good move.