Fix race condition in AWS CLI cache creation during parallel KubernetesPodOperator auth (#60943) by Vamsi-klu · Pull Request #1 · Vamsi-klu/airflow

Vamsi-klu · 2026-02-15T03:56:39Z

Summary

Pre-create ~/.aws/cli/cache directory in KubernetesHook.get_conn() to prevent a FileExistsError race condition when multiple KPO tasks authenticate via aws eks get-token concurrently on the same Celery worker
Older botocore versions (<1.40.2) call os.makedirs() without exist_ok=True, causing intermittent task failures before pod creation

Root Cause

When parallel KubernetesPodOperator tasks invoke exec-based EKS authentication on the same worker, the AWS CLI races to create ~/.aws/cli/cache. The losing process gets FileExistsError (errno 17), which surfaces as a 403 Forbidden from the Kubernetes API — the task fails before the pod is even created.

Fixed upstream in botocore 1.40.2, but this defensive fix protects users on older versions.

Why this approach (and not something else)

We considered several alternatives before landing on defensive directory pre-creation:

Approach	Why we rejected it
Retry on 403 in `generic_api_retry`	403 is normally a permanent permissions error. Adding it to `TRANSIENT_STATUS_CODES` would mask real auth failures and add retry latency to every legitimate 403. Distinguishing transient exec-auth 403s from real permission denials is not reliably possible — the Kubernetes client's ExecProvider silently swallows the subprocess error and proceeds with a bad token, so the 403 looks identical to a genuine RBAC denial.
`threading.Lock` around config loading	The exec plugin (`aws eks get-token`) runs lazily during the first API call, not during `config.load_kube_config()`. A lock around config loading wouldn't prevent the race. Locking around every API call would serialize all K8s operations — unacceptable for performance.
Parse kubeconfig to detect exec-based auth	Over-engineered for a one-line fix. Would add complexity, fragile YAML parsing, and still need per-tool knowledge of which cache dirs to create.
Pin `botocore >= 1.40.2` as a dependency	The Kubernetes provider has no direct dependency on botocore and shouldn't. AWS is just one of many possible exec-based auth backends.
Documentation-only (recommend botocore upgrade)	Doesn't help users who can't control their botocore version (e.g., managed Airflow platforms like Astronomer).

Why pre-creation wins:

It's a single os.makedirs(..., exist_ok=True) call — the exact same fix botocore 1.40.2 applied, just done earlier in the call chain
exist_ok=True is inherently safe for concurrent invocations — no race between our pre-creation and the AWS CLI
Zero performance overhead (one syscall, idempotent)
Zero risk of masking real errors — we don't change retry behavior or error handling
Protects all users regardless of their botocore version

Changes

hooks/kubernetes.py: Added _ensure_exec_plugin_cache_dirs() function called from get_conn() before any kube config loading. Uses os.makedirs(..., exist_ok=True) to pre-create the cache directory.
test_kubernetes.py: 3 new test cases verifying directory creation, idempotency, and integration with get_conn().

closes: apache#60943

Test plan

New unit tests verify directory creation, idempotency, and integration
Manual: Run parallel KPO tasks on same Celery worker with EKS auth and botocore < 1.40.2

Note to users: Upgrading to botocore >= 1.40.2 also resolves this at the source. This fix provides a safety net for environments that cannot upgrade immediately.

…che#60977)

…#61253) Co-authored-by: Anton Nitochkin <[email protected]>

…integration tests (apache#61311)

* Fix Keycloak double-slash URL bug (apache#61121) - Normalize server_url in _get_token_url to prevent double-slashes - Add .rstrip('/') to handle trailing slashes in server_url configuration - Add comprehensive tests for URL normalization scenarios - Resolves compatibility issue with Keycloak 26.4+ strict path validation When server_url has a trailing slash (e.g., 'https://host/auth/'), the previous implementation would create invalid URLs with double-slashes (e.g., 'https://host/auth//realms/...'), which Keycloak 26.4+ rejects with HTTP 400 'missingNormalization' error. This fix allows users to configure server_url with or without trailing slashes while ensuring properly normalized URLs are always generated. Fixes apache#61121 * Refactor URL normalization tests to use parametrize - Consolidate 4 separate test methods into a single parametrized test - Improves maintainability and reduces code duplication - Covers same scenarios: no trailing slash, single slash, multiple slashes, root path

…put (apache#61302)

apache#61287) * Fix OAuth session race condition causing false 401 errors during login Fixes apache#57981 When users authenticate via Azure OAuth SSO (and other OAuth providers), the UI briefly displays an authentication error message during the OAuth redirect flow. The error appears for approximately 1 second before disappearing once authentication successfully completes. Root Cause: The issue stems from a race condition during the OAuth authentication flow. After the OAuth callback completes and the user is authenticated, the Flask session containing OAuth tokens and user data may not be fully committed to the session backend (cookie or database) before the redirect response is sent to the client. When the UI loads and immediately makes API requests (like /ui/config), these requests arrive before the session is available, causing temporary 401 Unauthorized errors. Solution: This commit introduces a CustomAuthOAuthView that extends Flask-AppBuilder's AuthOAuthView to explicitly ensure the session is committed before redirecting. The fix: 1. Created providers/fab/src/airflow/providers/fab/auth_manager/views/auth_oauth.py with CustomAuthOAuthView class 2. Override oauth_authorized() method to mark session.modified = True after parent's OAuth callback handling completes 3. Updated security_manager/override.py to use CustomAuthOAuthView instead of the default AuthOAuthView This ensures Flask's session interface saves the session via the after_request handler before the HTTP redirect response is sent to the client, eliminating the race condition. The fix addresses the root cause as suggested by maintainer feedback on PR apache#58037, rather than masking the error in the UI. Testing: - Syntax validated with py_compile - Works with both session backends (database and securecookie) - Maintains backward compatibility with existing OAuth flows Related Issues: - apache#55612 - Airflow UI initial XHR returns 401 before session cookie is set - apache#57534 - Airflow 3.1.1 oauth login failure - apache#57485 - Airflow 3.1.1 oauth login broken - PR apache#58037 - Previous UI-based workaround attempt (closed) Signed-off-by: Jgprog117 <[email protected]> * Fix logging to use %-formatting instead of f-strings * Add tests for CustomAuthOAuthView * Fix linting and formatting issues in OAuth session race condition fix Remove unused imports, fix import ordering, and apply ruff formatting: - Remove unused pytest import from test_auth_oauth.py - Remove unused AuthOAuthView import from override.py - Fix import ordering to comply with ruff formatting rules - Apply ruff format to test file * Address PR review feedback from SameerMesiah97 - Remove redundant if/else branching that did the same thing in both paths - Fix misleading "completed successfully" log message to neutral wording - Replace brittle __class__.__bases__[0] mocking with explicit AuthOAuthView - Consolidate duplicate backend tests into a single parametrized test * Fix test RuntimeError by avoiding Flask session LocalProxy access Use mock.patch with new= as context manager instead of decorator to prevent mock from inspecting the Flask session LocalProxy, which requires an active request context. --------- Signed-off-by: Jgprog117 <[email protected]>

Extract TriggerDAGRunPostBody logical_date defaulting from _get_func into a new _apply_datamodel_defaults method. This addresses review feedback to reduce cyclomatic complexity in the already-complex _get_func method and provides a centralized location for managing datamodel-specific defaults.

…ache#60640)

…pache#61352) If VERSION_SUFFIX is exported in the current terminal, it can be accidentally used as `--version-suffix` by distribution preparation commands - which in case of the PMC verification might cause preparing of `PyPI` variants of the distributions, not the `SVN` ones (without RC suffix). This PR updates the relevant commands to override the suffix with empty suffix - which also is an explicit signal that this is intended.

* doc: clarify policy for exposing sensitive data * Update airflow-core/docs/security/security_model.rst --------- Co-authored-by: Jarek Potiuk <[email protected]>

…pache#61350)

* Refactor configuration parser to respect deprecated options in default parser * Add tests to ensure deprecated options are skipped in configuration parser * Add test for handling default values of deprecated options in AirflowConfigParser * Add test case without fallback param

There were a few things raising warnings during uv sync that should be fixed: * no lower limits on few dependencies * old pcycopg2 dev dependency for postgres provider. * botocore/aiobotocore depsndencies not updated for a while * lack of quotes in some sqlalchemy deps * optional deps for sqlalchemy not sorted properly

Signed-off-by: Guan-Ming (Wesley) Chiu <[email protected]>

* Add data lineage graph to AssetGraph * Update airflow-core/src/airflow/api_fastapi/core_api/routes/ui/dependencies.py Co-authored-by: Jason(Zhe-You) Liu <[email protected]> * Refactor to use deque * Apply review comments * Add task icon * Extract get_scheduling_dependencies --------- Co-authored-by: Jason(Zhe-You) Liu <[email protected]>

…e#61385)

The operator was incorrectly ignoring the `wait_policy` argument and always defaulting to waiting for cluster completion. This change ensures the `wait_policy` is correctly persisted and used to select the appropriate waiter (e.g., for step completion), fixing the hardcoded behavior.

…es (apache#61404) * Bump the fab-ui-package-updates group across 1 directory with 3 updates Bumps the fab-ui-package-updates group with 3 updates in the /providers/fab/src/airflow/providers/fab/www directory: [jquery-ui](https://github.com/jquery/jquery-ui), [css-loader](https://github.com/webpack/css-loader) and [stylelint](https://github.com/stylelint/stylelint). Updates `jquery-ui` from 1.14.1 to 1.14.2 - [Release notes](https://github.com/jquery/jquery-ui/releases) - [Commits](jquery/jquery-ui@1.14.1...1.14.2) Updates `css-loader` from 7.1.2 to 7.1.3 - [Release notes](https://github.com/webpack/css-loader/releases) - [Changelog](https://github.com/webpack/css-loader/blob/main/CHANGELOG.md) - [Commits](webpack/css-loader@v7.1.2...v7.1.3) Updates `stylelint` from 17.0.0 to 17.1.0 - [Release notes](https://github.com/stylelint/stylelint/releases) - [Changelog](https://github.com/stylelint/stylelint/blob/main/CHANGELOG.md) - [Commits](stylelint/stylelint@17.0.0...17.1.0) --- updated-dependencies: - dependency-name: jquery-ui dependency-version: 1.14.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: fab-ui-package-updates - dependency-name: css-loader dependency-version: 7.1.3 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: fab-ui-package-updates - dependency-name: stylelint dependency-version: 17.1.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: fab-ui-package-updates ... Signed-off-by: dependabot[bot] <[email protected]> * Update hash --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: vincbeck <[email protected]>

Bumps @isaacs/brace-expansion from 5.0.0 to 5.0.1. --- updated-dependencies: - dependency-name: "@isaacs/brace-expansion" dependency-version: 5.0.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Update timer.duration labels and examples to milliseconds, and fix the OTel timing docstring to match runtime behavior. Historical context: apache#39908 introduced milliseconds alignment with a compatibility flag, and apache#43975 removed that flag and standardized milliseconds everywhere. Co-authored-by: Cursor <[email protected]>

* refactor: Added deferrable support to WinRMOperator * Update providers/microsoft/winrm/src/airflow/providers/microsoft/winrm/triggers/winrm.py Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: Copilot <[email protected]>

…pache#61831)

…e#61813) * Fix scheduler crash when enqueuing TI with null dag_version_id After upgrade from AF2, TIs might be without dag_version_id since we don't enforce this at the DB level. The solution here is to skip enqueing such TIs until the verify_integrity runs which would update the dag_version_id of the TI. Initially, the TI would be stuck but would later be cleared when the handle tasks stuck in queued deems it fit. * update warning log * update warning log * Update the caplog level in test

* flush in-memory otel metrics at process shutdown * fix module for test

… key (apache#61837)

Signed-off-by: Guan-Ming (Wesley) Chiu <[email protected]>

…61872)

…61865)

…he#61885)

…o a shared constant (apache#61891)

…pache#61842)

Release version check for Python started to fail with rate limit, this PR changes Python version retrieval to use static information hosted on ftp site of Python (which is behind CDN and can handle much bigger traffic and is read-only so there are no rate limits involved.

@jscheffl

* Added Informatica Plugin Provider * Added Informatica Plugin Provider * changed the informatica version which uploaded on pypi * fix: restore accidentally deleted symlink files * added tests requirements and missing tests * Fix provider compat errors in prek/breeze checks * fix prex listener issue * Apply suggestions from code review Co-authored-by: Jens Scheffler <[email protected]> * Apply suggestions from code review Co-authored-by: Copilot <[email protected]> * refactor(providers/informatica): address review comments from @jscheffl * refactor(providers/informatica): address review comments from copilot * refactor(providers/informatica): address review comments from copilot * refactor(providers/informatica&openlineage): Breeze documentation has been updated due to a provider version change and fixing tests * refactor(providers/informatica&openlineage): Breeze documentation has been updated due to a provider version change and fixing tests * Update providers/informatica/docs/index.rst Co-authored-by: Jarek Potiuk <[email protected]> * refactor(providers/informatica): address review comments from @jscheffl * Regenerate Breeze output after main merge * Fix:(providers/informatica) build-docs error for removed tests & changed CODEOWNERS * fix: removed informatica integration tests in GroupOfTests.INTEGRATION_PROVIDERS * refactor(providers/informatica) regenerated breeze docs after the common.ai provider was added * fix: deleted not necessarly init file --------- Co-authored-by: Jens Scheffler <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Jarek Potiuk <[email protected]>

Corrected the spelling of 'VERSION_SUFFIX' in the command and clarified the note regarding its temporary unsetting.

…atus (apache#61426) - Adds a new 'airflowctl auth list-envs' command that scans AIRFLOW_HOME for environment config files and reports each environment's name, API URL, and authentication status (authenticated / not authenticated). - Filters out internal files (debug_creds_*.json, *_generated.json) so only real environments are listed.

apache#61281) * fix(api): disable uvloop if PYTHONASYNCIODEBUG=1 to prevent segfault (issue apache#61214) * docs(newsfragment): add newsfragment for uvloop/asyncio debug segfault fix (apache#61214) * fix(api): add stacklevel for debug warning and fix rst literals * fix: remove else block and correct newsfragment per review feedback * docs: document PYTHONASYNCIODEBUG/PYTHONDEVMODE incompatibility (Python 3.12+) * docs: fix RST formatting and spellcheck errors - Add blank line after warning directive to fix RST error - Quote asyncio with backticks to fix spellcheck * refactor: simplify to warning-only approach per reviewer feedback * refactor: apply reviewer suggestions --------- Co-authored-by: Jason(Zhe-You) Liu <[email protected]>

…k openlineage (apache#60918)

) * Fix JDBC provider conf import to use compat sdk * Add comment for common-compat dependency version

) Co-authored-by: Stephen Bracken <email-protected>

…esPodOperator auth (apache#60943) Pre-create ~/.aws/cli/cache directory in KubernetesHook.get_conn() to prevent a FileExistsError race condition when multiple KPO tasks authenticate via `aws eks get-token` concurrently on the same Celery worker. Older botocore versions (<1.40.2) call os.makedirs() without exist_ok=True, causing intermittent task failures before pod creation. closes: apache#60943

henry3260 and others added 30 commits February 2, 2026 10:58

Migrate Flask based role_and_permission_endpoint APIs to Fastapi (apa…

8c96236

…che#60977)

Change serialization inside GenAIGeminiCreateBatchJobOperator (apache…

1129231

…#61253) Co-authored-by: Anton Nitochkin <[email protected]>

Construct task instances with ExecuteTask.make for celery executor …

784e1a9

…integration tests (apache#61311)

Fix typo in breeze cmd help (apache#61359)

7d32061

Chart: Remove breaking change for createUser (apache#61337)

698157f

Print debug mode warning to stderr to avoid polluting stdout JSON out…

74af16f

…put (apache#61302)

Update providers metadata 2026-02-02 (apache#61361)

baecfdd

Adds a "private_key" alternative extra field to GitHub connection (ap…

e4400f4

…ache#60640)

limit 0.39 version due to missing package (apache#61387)

ba1d473

Adding Bugra to Helm Charts as CODEOWNERS (apache#61367)

b78efb0

Clarify policy for exposing sensitive data (apache#59864)

8a4f38f

* doc: clarify policy for exposing sensitive data * Update airflow-core/docs/security/security_model.rst --------- Co-authored-by: Jarek Potiuk <[email protected]>

Introducing prek hook to detect airflow imports in shared libraries (a…

b3e3bc5

…pache#61350)

Fix verification process for Helm Chart (apache#61388)

c70f857

fix memray decorator error handling (apache#60624)

37d890b

Execution API: document RTIF endpoint (apache#60488)

e43dff0

Exclude unnecessary ui build artifacts from wheel (apache#61335)

8edb313

Signed-off-by: Guan-Ming (Wesley) Chiu <[email protected]>

Fix host in AWS auth manager (apache#61368)

1a86aec

fix: Fix grid view tooltip z-index issue (apache#61275)

f701351

Add JWT issuer config to docker-compose and breeze environment (apach…

c3b5c66

…e#61385)

jedcunningham and others added 30 commits February 13, 2026 08:20

fix: apply missing partition_key filter in PALK when creating DagRun (a…

a4a6eb2

…pache#61831)

CI: Upgrade important CI environment (apache#61840)

0fb9527

Flush in-memory OTel metrics at process shutdown (apache#61808)

a9df122

* flush in-memory otel metrics at process shutdown * fix module for test

Persist table columns visibility in local storage (apache#61858)

e4c4e36

feat(asset-partition): distinguish partition key and mapped partition…

1465124

… key (apache#61837)

Remove redundant debounce-promise dependency from UI (apache#61832)

721b0ed

Signed-off-by: Guan-Ming (Wesley) Chiu <[email protected]>

Update helm chart 1.19.0 release date (apache#61836)

f17d6ff

Fix ProvidersManager test assertions and add cleanup fixture (apache#…

24cc215

…61872)

Clarify that you should not forget about stable version (apache#61875)

2ada67a

Add missing template fields to DatabricksWorkflowTaskGroup (apache#…

850628e

…61865)

Fix breeze ui check-translation-completeness verification in ci (apac…

3060cea

…he#61885)

Fix EKS operator failures by consolidating API version references int…

fc73e0b

…o a shared constant (apache#61891)

Fix serialization for return value in GenAIGeminiGetBatchJobOperator (a…

2162e32

…pache#61842)

Remove duplicated import in roles.py (apache#61896)

ce36195

Fix mypy type error in MsSqlToHiveOperator (apache#61902)

048059b

Fix typo in upgrading_to_airflow3.rst (apache#60746)

65919a2

Update providers metadata 2026-02-14 (apache#61911)

8ced184

Remove comments for missing features (apache#61922)

cf5e0a4

Fix VERSION_SUFFIX typo and update note in README (apache#61918)

4d004b6

Corrected the spelling of 'VERSION_SUFFIX' in the command and clarified the note regarding its temporary unsetting.

Added validation for missing redshift connection host in postgres hoo…

9837cee

…k openlineage (apache#60918)

Providers: JDBC provider updated for conf import migration (apache#61413

1573e40

) * Fix JDBC provider conf import to use compat sdk * Add comment for common-compat dependency version

add max_line_per_pages setting to ElasticsearchTaskHandler (apache#61492

34359a8

) Co-authored-by: Stephen Bracken <email-protected>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix race condition in AWS CLI cache creation during parallel KubernetesPodOperator auth (#60943)#1

Fix race condition in AWS CLI cache creation during parallel KubernetesPodOperator auth (#60943)#1
Vamsi-klu wants to merge 640 commits into
mainfrom
fix/aws-cli-cache-race-condition-60943

Vamsi-klu commented Feb 15, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

Vamsi-klu commented Feb 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Root Cause

Why this approach (and not something else)

Changes

Test plan

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Vamsi-klu commented Feb 15, 2026 •

edited

Loading