Repository navigation
Comparing changes
Open a pull request
base repository: apache/airflow
base: dag-processor-activation
head repository: apache/airflow
compare: dag-processor-deployment
- 9 commits
- 18 files changed
- 1 contributor
Commits on Nov 11, 2026
-
Add API-mode Dag processor deployment support
API-mode processors need health probes that work without database credentials. Each processor session must also remain exclusive while pods are replaced, and isolated deployments need explicit credential and network boundaries around the trusted API server.
Configuration menu - View commit details
-
Copy full SHA for c337246 - Browse repository at this point
Copy the full SHA c337246View commit details -
Rotate Dag processor session tokens in Kubernetes Secrets
Dag processors using the Execution API need renewable credentials without access to signing keys. Kubernetes deployments also need to retain their session when the token publisher restarts or an API response is lost.
Configuration menu - View commit details
-
Copy full SHA for e8de12c - Browse repository at this point
Copy the full SHA e8de12cView commit details -
Clarify isolated Dag processor credential and migration guidance
Catalog synchronization and token provisioning have different credential requirements. Bundle migrations must also avoid having trusted and isolated processors reconcile different inventories for the same bundle.
Configuration menu - View commit details
-
Copy full SHA for 1fc3ea9 - Browse repository at this point
Copy the full SHA 1fc3ea9View commit details -
Clarify the Dag processor rollout status check
Deployment checks should name the command being described so operators can match the guidance to the example and distinguish process startup from continued token validity.
Configuration menu - View commit details
-
Copy full SHA for b5c4966 - Browse repository at this point
Copy the full SHA b5c4966View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0a14701 - Browse repository at this point
Copy the full SHA 0a14701View commit details -
Handle invalid input in the Dag processor token provisioner
A token whose subject is not a string, written to the Secret by anything other than the provisioner, made the session lookup raise AttributeError. Kubernetes rotation then re-read that token on every iteration and never replaced it, so the processor's token eventually expired, and file mode crashed at startup the same way. The provisioner now treats such a token as having no session and starts a new one. A bad Secret name, namespace or API port, or a missing cluster CA, ended the command with a traceback, and the CA error did not say which file it tried to load. These now exit with a message that names the problem, including the CA path. The name-validation test could not fail: without the in-cluster environment variables, the store raised a different ValueError that the test's match also accepted. It now sets those variables and matches the DNS-label and DNS-subdomain messages.
Configuration menu - View commit details
-
Copy full SHA for 92d1d33 - Browse repository at this point
Copy the full SHA 92d1d33View commit details -
Fix the isolated Dag processor deployment examples and chart checks
The token provisioner example signed with the key in the chart's generated <release>-jwt-secret. Every helm upgrade of the trusted release writes a new key there and rolls the API server onto it, but the provisioner keeps the old key in its environment until it restarts, so the API server rejected every token it issued after an upgrade. The example now reads a pre-created Secret that the trusted release also uses as jwtSecretName. The processor-only Helm values set LocalExecutor just to pass the chart's validation, which also made the processor parse Dags with a different [core] executor from the trusted deployment. Tasks naming another executor then failed to import. The example now sets the processor's executor to match, and the guide says it must. The chart chooses probes, update strategy and the replica limit only from config.dag_processor.execution_api_token_file. Setting the token file through AIRFLOW__DAG_PROCESSOR__EXECUTION_API_TOKEN_FILE instead kept a liveness probe that fails without database credentials, so the chart now rejects that variable. The chart tests now also cover rendering without a token file, including a null config.dag_processor. The guide overstated how close to token expiry a Kubernetes API outage can end without stopping the processor, because it left out the retry backoff and the Secret projection delay. Its install command also lacked a chart version, and a few smaller statements were wrong. No CI job ran the Compose deployment example tests, so the quick-start Compose job now runs them.
Configuration menu - View commit details
-
Copy full SHA for 0e10cab - Browse repository at this point
Copy the full SHA 0e10cabView commit details -
Select the chart's Dag processor API mode from use_execution_api
The Dag processor now enters API mode only when [dag_processor] use_execution_api is enabled; execution_api_token_file is just the credential source. The chart still chose the local probes, the Recreate strategy and the one-replica limit from the token file, so a release that set only the token file ran a database-mode processor with API-mode probes. "airflow dag-processor --check-health" then exits because local health probes need API mode, and Kubernetes restarts the pod in a loop. A processor with API mode enabled and no token file fails at startup, so the chart refuses to render that combination instead of deploying a pod that cannot start. Setting AIRFLOW__DAG_PROCESSOR__USE_EXECUTION_API in env or dagProcessor.env would bypass the chart's API-mode settings the same way the token-file variable did, so the chart rejects it too. The quick-start Compose healthcheck tested the token file in the same way, so it now follows use_execution_api as well.
Configuration menu - View commit details
-
Copy full SHA for 1a65e04 - Browse repository at this point
Copy the full SHA 1a65e04View commit details -
Keep Dag processor deployment probes consistent with API mode
Airflow accepts whitespace and inline comments in boolean configuration. Deployment selectors must recognize the same values so processors receive matching probes, rollout behavior and session-token safeguards.
Configuration menu - View commit details
-
Copy full SHA for 91bbfd1 - Browse repository at this point
Copy the full SHA 91bbfd1View commit details
This comparison is taking too long to generate.
Unfortunately it looks like we can’t render this comparison for you right now. It might be too big, or there might be something weird with your repository.
You can try running this command locally to see the comparison on your machine:
git diff dag-processor-activation...dag-processor-deployment