Re-organize repo into multiple projects using uv's workspaces "core" (scheduler + web/api servers), "providers" (a single project for all providers for now), "task-sdk" (new dist for this project) #42857
Description
List discussion https://lists.apache.org/thread/dyv5jhvt65xs6l5o2byc2b67f4wlwf6r
We will try out uv's workspace feature for this.
The projects we want are:
- "core", containing the scheduler, web/api servers, dag parser etc.
- "providers", a single project containing all the provivders
- "task-sdk", the new dist that is key to this AIP.
There will be other top level folders such as the helm chart etc.
Tests should probably be split to live alongside the projects.
Example code layout (picking a few representative files in each group):
core/pyproject.toml/
core/src/airflow/__init__.py
core/src/airflow/models/…
core/src/airflow/api_fastapi/…
core/src/airflow/ui/package.json # React UI etc.
core/tests/jobs/test_scheduler_job.py
providers/src/airflow/providers/celery/provider.yaml
providers/src/airflow/providers/celery/__init__.py
providers/tests/airflow/providers/celery/executors/test_celery_executor.py
providers/src/airflow/providers/cncf/kubernetes/provuder.yaml
providers/src/airflow/providers/cncf/kubernetes/__init__.py
task-sdk/pyproject.toml
task-sdk/src/airflow/sdk/__init__.py
task-sdk/src/airflow/sdk/defintions/dag.py
task-sdk/src/airflow/sdk/defintions/task_group.py
task-sdk/tests/airflow/sdk/defintions/test_task_group.py