Airflow is great in a specific DAG checks. Meaning if you know there is an issue with a specific DAG you have everything you need to check what is going on but if you have a system of 1000+ dags your needs are a bit different.
It's missing some kind of dashboard that will show what is going on in system wise manner.
For example:
TOP X dags that are still in running mode
Pools that are max occupied
last scheduler heartbeat
I can think of many other metrics
The idea here is a dashboard that will show what is going on in Airflow right now (not calculating over history).
There are many ways to get it externally with Grafana and other tools but I think it comes to a time where Airflow should provide it itself.
Airflow is great in a specific DAG checks. Meaning if you know there is an issue with a specific DAG you have everything you need to check what is going on but if you have a system of 1000+ dags your needs are a bit different.
It's missing some kind of dashboard that will show what is going on in system wise manner.
For example:
TOP X dags that are still in running mode
Pools that are max occupied
last scheduler heartbeat
I can think of many other metrics
The idea here is a dashboard that will show what is going on in Airflow right now (not calculating over history).
There are many ways to get it externally with Grafana and other tools but I think it comes to a time where Airflow should provide it itself.