Skip to content

Fixes #57515 - Fix test_async_write_logs_should_execute_successfully test - #1

Merged
hwang-cadent merged 720 commits into
mainfrom
fix-57515-kubernetes-pod-operator-test
Nov 22, 2025
Merged

hwang-cadent merged 720 commits into
mainfrom
fix-57515-kubernetes-pod-operator-test

Conversation

@hwang-cadent

Copy link
Copy Markdown
Owner

Description

This PR fixes the broken test test_async_write_logs_should_execute_successfully() in the KubernetesPodOperator test suite. The test was incorrectly implemented and was not properly testing the async log writing functionality.

Changes

  • Added @patch(KUB_OP_PATH.format("client")) decorator to properly mock self.client.read_namespaced_pod_log() which is the actual method used by _write_logs()
  • Fixed mock setup to use mocked_client.read_namespaced_pod_log instead of incorrectly mocking mock_manager.read_pod_logs (which is never called)
  • Fixed mock return value to return an iterable of bytes: [test_logs.encode("utf-8")] as expected by the implementation
  • Added mock_manager.await_pod_completion.return_value for proper test flow
  • Fixed assertions to verify actual log output using caplog instead of a meaningless string assertion
  • Fixed get_logs=False assertion to check mocked_client.read_namespaced_pod_log.assert_not_called() instead of the incorrect pod_manager.read_pod_logs check

Root Cause

The test had three main issues:

  1. Wrong mock target: It mocked pod_manager.read_pod_logs but _write_logs() actually calls self.client.read_namespaced_pod_log() directly
  2. Incorrect assertion: The assertion assert f"Container logs: {test_logs}" always passes (it's just checking if a string exists, not if it was logged)
  3. Wrong verification for get_logs=False: It checked pod_manager.read_pod_logs which is never called by _write_logs()

Testing

  • Both parameterized test cases now pass:
    • test_async_write_logs_should_execute_successfully[True] ✅
    • test_async_write_logs_should_execute_successfully[False] ✅
  • All existing tests continue to pass

Type of Change

  • Bug fix (non-breaking change which fixes an issue)

Fixes apache#57515

KoviAnusha and others added 30 commits November 6, 2025 10:03
…/tests/unit/amazon/aws/operators/) (apache#57903)

Co-authored-by: Kalyan R <[email protected]>
apache#56272)

* Propogate dagrun timeout failure context

* add unit test

* pass started msg for notify dagrun start state change

* make msg required for notify_dagrun_state_changed

* fix type unit test failure
…57619)

* Fix scheduler using stale max_active_runs from SerializedDAG

When max_active_runs is updated via versioned bundles, the scheduler
was checking the limit against stale SerializedDAG data instead of
the latest DagModel value. This caused queued DAG runs to be incorrectly
blocked even when the limit had been increased.

The scheduler now uses max_active_runs from DagModel (accessed via
dag_run association proxy) to ensure versioned bundle updates to
max_active_runs are respected immediately.

closes: apache#57604

* fixup! Fix scheduler using stale max_active_runs from SerializedDAG

* fixup! fixup! Fix scheduler using stale max_active_runs from SerializedDAG
…apache#57912)

The system tests just needs to run the minimum batch size to test
functionality. The shorter batch should decrease test runtime and
resource usage.

Also fix a bug that caused the prompts list to be an unreliable size,
there was a missing flush on the temp file.
potiuk and others added 29 commits November 14, 2025 09:51
)

It's useful, but it's missing an option to attach non-asset related metadata collected from hooks
(f.e. query_id, or query_text), that can be also useful in context of lineage and desired by hook
lineage readers. For example, having the executed SQL query enables SQL parsing on the reader
side, retrieving input and output datasets, which can significantly improve lineage extraction
accuracy.
…58264)

Add logging to test_example_dag_hashes_are_always_consistent
…pache#58148)

(This is the second attempt at this PR)

This change is quite simple, as it simply calls the function added in apache#57212
and apache#58147 if it's available. Up until the point that it is released this code
will not hit the getattr will return None and be a no-op.

I didn't use the walrus operator here as theoretically someone could be using
this to run in a Python that doesn't support it.

Fixes apache#51422, fixes apache#54706.

This doesn't have any unit tests, as a) it's relatively simple, and b) due to
how we run in the same process in the tests, there is no socket to reconnect
to, so we can't actually exercise this code
…7710)

* Display static asset metadata (extra) in Asset details page

* Add i18n translation for static Asset Metadata and minor UI cleanup

* Add i18n translation key 'ststic_asset_metadata' to all locale asset.jason files

* Rename Static Asset Metadata to Additional Data
…ache#57863) (apache#58070)

- Changed AwaitMessageSensor and AwaitMessageTriggerFunctionSensor to inherit from BaseSensorOperator instead of BaseOperator
- This enables both sensors to accept standard sensor parameters like timeout and soft_fail
- Added comprehensive test coverage for timeout and soft_fail parameters
- Updated documentation to reflect the new parameters

Fixes apache#57863
apache#57756) (apache#57968)

When TriggerDagRunOperator is used with deferrable=True, wait_for_completion=True,
  reset_dag_run=True, and a fixed trigger_run_id, the operator becomes permanently
  stuck in deferred state after clearing and re-running.

  Root cause:
  When reset_dag_run=True is used with a fixed run_id, the database preserves the
  original logical_date from the first run. However, on subsequent runs after clearing,
  the operator calculates a NEW logical_date based on the current time. The DagStateTrigger
  was being created with this newly calculated logical_date, causing a mismatch when
  querying the database - the trigger looked for a DAG run with the new logical_date
  but the database contained the original logical_date, causing the query to return
  zero results indefinitely.

  Solution:
  - Modified _handle_trigger_dag_run() in task_runner.py to pass execution_dates=None
    to DagStateTrigger when run_ids is provided, since run_id alone is sufficient and
    globally unique
  - Added test test_handle_trigger_dag_run_deferred_with_reset_uses_run_id_only to
    verify the fix and prevent regression

  The fix ensures that both deferrable and non-deferrable modes use identical logic
  for determining DAG run completion - querying by run_id and state only, without
  filtering by logical_date which can become stale when resets are involved.
* Updates to release process of providers

There are several updates to the release process in preparation for
rotational release management process:

* documentation has been simplified and unnecessary steps and digressions
  are move out of the way
* some commands had been simplified, optional steps have been moved
  at the end of the doc
* commented out PRs in release notes are automatically removed
  from the Issue created for users
* better guideliens on how to iterate with the issue by removing
  unneeded PRs
* gh issue create used instead of manually hand-crafting the URL
  is used to trigger issue creation
* github token retrieval is simplified for prepare-issues command
* stale ".. Please review " comments were removed from the changelogs
* We are using now ${RELEASE_DATE} subfolder in SVN to store the providers -
  this will allow several parallel releases to be run.

* Update providers/amazon/docs/changelog.rst

Co-authored-by: Bugra Ozturk <[email protected]>

---------

Co-authored-by: Bugra Ozturk <[email protected]>
* fix: Prevent preserving tabs when navigating to Task Groups

* refactor: Centralize keyboard navigation logic to use

* test: update test,  Groups should not preserve tabs
…mode (apache#57846) (apache#58080)

* Docs: Add note for WSL users about CHOKIDAR_USEPOLLING for UI hot reloading (apache#57846)

* Breeze: Automatically set CHOKIDAR_USEPOLLING for WSL users in --dev-mode (apache#57846)

* Breeze: Automatically set CHOKIDAR_USEPOLLING for WSL users in --dev-mode (apache#57846)

* Update dev/breeze/src/airflow_breeze/commands/developer_commands.py

Co-authored-by: LIU ZHE YOU <[email protected]>

---------

Co-authored-by: LIU ZHE YOU <[email protected]>
…57169)

* Add JSON serialization for kubeconfig in AsyncKubernetesHook

* Add debug logging for kubeconfig serialization in AsyncKubernetesHook

* Ran perk

---------

Co-authored-by: Kalyan R <[email protected]>
apache#58306)

The instructions were partially outdated and quite a bit too chatty, they
did not focus on actual tasks to do but also contained some digressions
and more detailed description of some provider principles.

This change makes it simpler to follow the process by the release manager,
moves the docuentation of details to PROVIDERS.rst and removes largely
duplicated PROVIDER_DISTRIBUTION_DETAILS.md

The "--incremental-update" flag is specifically added and mentioned
in the doc on what should be done when new commits are added after
initial release notes have been prepared, guiding the release
manager more explicitly what to do and providing instructions.
The default is 6s which is quite aggressive and causes throttling issues
in the test harness. Tests don't need that kind of fidelity and can wait
much longer between attempts to fire the trigger.
@hwang-cadent
hwang-cadent merged commit 9fdb2a4 into main Nov 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

KubernetesPodOperator Test test_async_write_logs_should_execute_successfully is not working