-
Notifications
You must be signed in to change notification settings - Fork 16.3k
Description
Apache Airflow version
2.5.1
What happened
I'd used airflow 2.3.1 and everything was fine.
Then I decided to move to airflow 2.5.1.
I can't start even webserver, airflow on my laptop consumes the entire memory (32Gb) and OOM killer comes.
I investigated a bit. So it starts with airflow 2.3.4. Only using official docker image (apache/airflow:2.3.4) and only on linux laptop, mac is ok.
Memory leak starts when source code tries to import for example airflow.cli.commands.webserver_command module using airflow.utils.module_loading.import_string.
I dived deeply and found that it happens when "import daemon" is performed.
You can reproduce it with this command: docker run --rm --entrypoint="" apache/airflow:2.3.4 /bin/bash -c "python -c 'import daemon'". Once again, reproducec only on linux (my kernel is 6.1.12).
That's weird considering daemon hasn't been changed since 2018.
What you think should happen instead
No response
How to reproduce
docker run --rm --entrypoint="" apache/airflow:2.3.4 /bin/bash -c "python -c 'import daemon'"
Operating System
Arch Linux (kernel 6.1.12)
Versions of Apache Airflow Providers
No response
Deployment
Docker-Compose
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct