-
-
Notifications
You must be signed in to change notification settings - Fork 48
Description
Our current implementation of asynchronous processing is built on top of React/AMPhp. Flow is simply spawning new processes and it's sending the pipeline into it, later those processes are receiving rows over TCP, transforming them and sending back also through TCP to the main process.
This approach, even that is pretty cool is not brining enough value in order to justify the maintenance around it.
The main problem comes from the fact that processing is not the biggest bottleneck, our main problem comes from extraction, which is not included in our current async processing model.
Another problem is that async processing is currently impossible through phar/docker.
For now we are going to remove the AsyncPipeline and react/amphp adapters, which should reduce the complicity of our codebase allowing us to easier refactor Pipelines (we are still missing Executor). One that is done, we should look again into async processing, however this time we need to start from async extraction.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status