This is the meta issue for tracking performance optimization of the UI and API to improve airflow operating at high scale.

Done

I - API

• Add gunicorn support for API server with rolling worker restarts #60940 (memory optimization and worker rotation)

• API - Improve historical_metrics_data endpoint #62021

• API - Improve public get_dag_runs endpoint #62025 perf: use load_only() in get_dag_runs eager loading to reduce data fetched per task instance #62482

• Optimize /ui/dags endpoint serialization #61483

• Mention http2 in the API/UI performance documentation #62032

• Add log buffering to increase performance when streaming big logs #63180 (Add log endpoint buffering for streaming)

II - UI

• Add virtualization to grid view #60241
• Optimize grid node merging with ID index #61656
• UI - Improve refresh rate strategy of the Grid view #62033
• Replace node-sql-parser with sqlparser-ts #61111
• Stream grid task-instance summaries as NDJSON to eliminate N+1 requests #62369

Targeted for 3.2+

• API - Improve public get_task_instances endpoint #62027

  • fix: eliminate duplicate JOINs in get_task_instances endpoint #62910
  • Change default sort to 'id' for list task instances #64168
  • Update search parameters to better leverage DB indexes #64963

• Cache requires_access_dag checks and other auth manager redundant checks #61485 Cache dag access and teams check to be reused in grid ti_summary API calls #61623

• Add log buffering to increase performance when streaming big logs #63180

• improve grid/ti_summaries and grid/runs improve grid/ti_summaries and grid/runs #64034

• Improve Graph View Performance Improve Graph View Performance #65031

• List task instances Add cursor based pagination for get_task_instances endpoint #64845

• Cursor based pagination for dag runs Add cursor based pagination for get_dag_runs endpoint #65604

Attached is a zip file with some dummy dags that will generate a high number of tasks and runs in your local development setup. This is useful to 'simulate' a high scale installation with millions of TIs and thousands of dag runs, to help you work on the linked issues or identify other bottlenecks.

serdagexpensive.zip