Skip to content

high memory leak, cannot start even webserver #29841

@antonio-antuan

Description

@antonio-antuan

Apache Airflow version

2.5.1

What happened

I'd used airflow 2.3.1 and everything was fine.
Then I decided to move to airflow 2.5.1.
I can't start even webserver, airflow on my laptop consumes the entire memory (32Gb) and OOM killer comes.

I investigated a bit. So it starts with airflow 2.3.4. Only using official docker image (apache/airflow:2.3.4) and only on linux laptop, mac is ok.

Memory leak starts when source code tries to import for example airflow.cli.commands.webserver_command module using airflow.utils.module_loading.import_string.
I dived deeply and found that it happens when "import daemon" is performed.
You can reproduce it with this command: docker run --rm --entrypoint="" apache/airflow:2.3.4 /bin/bash -c "python -c 'import daemon'". Once again, reproducec only on linux (my kernel is 6.1.12).
That's weird considering daemon hasn't been changed since 2018.

What you think should happen instead

No response

How to reproduce

docker run --rm --entrypoint="" apache/airflow:2.3.4 /bin/bash -c "python -c 'import daemon'"

Operating System

Arch Linux (kernel 6.1.12)

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions