Repository navigation
Fixes #57515 - Fix test_async_write_logs_should_execute_successfully test - #1
Merged
Merged
Conversation
…57844) Co-authored-by: Kalyan R <[email protected]>
Co-authored-by: Kalyan R <[email protected]>
Co-authored-by: Kalyan R <[email protected]>
…cutors, datasets) (apache#57913) Co-authored-by: Kalyan R <[email protected]>
…et method (apache#55289) Co-authored-by: Tzu-ping Chung <[email protected]>
…/tests/unit/amazon/aws/operators/) (apache#57903) Co-authored-by: Kalyan R <[email protected]>
…che#57911) Co-authored-by: Kalyan R <[email protected]>
…#57888) Co-authored-by: Kalyan R <[email protected]> Signed-off-by: Xch1 <[email protected]>
Co-authored-by: Kalyan R <[email protected]> Signed-off-by: Xch1 <[email protected]>
apache#56272) * Propogate dagrun timeout failure context * add unit test * pass started msg for notify dagrun start state change * make msg required for notify_dagrun_state_changed * fix type unit test failure
…57619) * Fix scheduler using stale max_active_runs from SerializedDAG When max_active_runs is updated via versioned bundles, the scheduler was checking the limit against stale SerializedDAG data instead of the latest DagModel value. This caused queued DAG runs to be incorrectly blocked even when the limit had been increased. The scheduler now uses max_active_runs from DagModel (accessed via dag_run association proxy) to ensure versioned bundle updates to max_active_runs are respected immediately. closes: apache#57604 * fixup! Fix scheduler using stale max_active_runs from SerializedDAG * fixup! fixup! Fix scheduler using stale max_active_runs from SerializedDAG
…apache#57912) The system tests just needs to run the minimum batch size to test functionality. The shorter batch should decrease test runtime and resource usage. Also fix a bug that caused the prompts list to be an unreliable size, there was a missing flush on the temp file.
) It's useful, but it's missing an option to attach non-asset related metadata collected from hooks (f.e. query_id, or query_text), that can be also useful in context of lineage and desired by hook lineage readers. For example, having the executed SQL query enables SQL parsing on the reader side, retrieving input and output datasets, which can significantly improve lineage extraction accuracy.
…58264) Add logging to test_example_dag_hashes_are_always_consistent
…pache#58148) (This is the second attempt at this PR) This change is quite simple, as it simply calls the function added in apache#57212 and apache#58147 if it's available. Up until the point that it is released this code will not hit the getattr will return None and be a no-op. I didn't use the walrus operator here as theoretically someone could be using this to run in a Python that doesn't support it. Fixes apache#51422, fixes apache#54706. This doesn't have any unit tests, as a) it's relatively simple, and b) due to how we run in the same process in the tests, there is no socket to reconnect to, so we can't actually exercise this code
…7710) * Display static asset metadata (extra) in Asset details page * Add i18n translation for static Asset Metadata and minor UI cleanup * Add i18n translation key 'ststic_asset_metadata' to all locale asset.jason files * Rename Static Asset Metadata to Additional Data
…ache#57863) (apache#58070) - Changed AwaitMessageSensor and AwaitMessageTriggerFunctionSensor to inherit from BaseSensorOperator instead of BaseOperator - This enables both sensors to accept standard sensor parameters like timeout and soft_fail - Added comprehensive test coverage for timeout and soft_fail parameters - Updated documentation to reflect the new parameters Fixes apache#57863
apache#57756) (apache#57968) When TriggerDagRunOperator is used with deferrable=True, wait_for_completion=True, reset_dag_run=True, and a fixed trigger_run_id, the operator becomes permanently stuck in deferred state after clearing and re-running. Root cause: When reset_dag_run=True is used with a fixed run_id, the database preserves the original logical_date from the first run. However, on subsequent runs after clearing, the operator calculates a NEW logical_date based on the current time. The DagStateTrigger was being created with this newly calculated logical_date, causing a mismatch when querying the database - the trigger looked for a DAG run with the new logical_date but the database contained the original logical_date, causing the query to return zero results indefinitely. Solution: - Modified _handle_trigger_dag_run() in task_runner.py to pass execution_dates=None to DagStateTrigger when run_ids is provided, since run_id alone is sufficient and globally unique - Added test test_handle_trigger_dag_run_deferred_with_reset_uses_run_id_only to verify the fix and prevent regression The fix ensures that both deferrable and non-deferrable modes use identical logic for determining DAG run completion - querying by run_id and state only, without filtering by logical_date which can become stale when resets are involved.
* Updates to release process of providers
There are several updates to the release process in preparation for
rotational release management process:
* documentation has been simplified and unnecessary steps and digressions
are move out of the way
* some commands had been simplified, optional steps have been moved
at the end of the doc
* commented out PRs in release notes are automatically removed
from the Issue created for users
* better guideliens on how to iterate with the issue by removing
unneeded PRs
* gh issue create used instead of manually hand-crafting the URL
is used to trigger issue creation
* github token retrieval is simplified for prepare-issues command
* stale ".. Please review " comments were removed from the changelogs
* We are using now ${RELEASE_DATE} subfolder in SVN to store the providers -
this will allow several parallel releases to be run.
* Update providers/amazon/docs/changelog.rst
Co-authored-by: Bugra Ozturk <[email protected]>
---------
Co-authored-by: Bugra Ozturk <[email protected]>
* fix: Prevent preserving tabs when navigating to Task Groups * refactor: Centralize keyboard navigation logic to use * test: update test, Groups should not preserve tabs
…mode (apache#57846) (apache#58080) * Docs: Add note for WSL users about CHOKIDAR_USEPOLLING for UI hot reloading (apache#57846) * Breeze: Automatically set CHOKIDAR_USEPOLLING for WSL users in --dev-mode (apache#57846) * Breeze: Automatically set CHOKIDAR_USEPOLLING for WSL users in --dev-mode (apache#57846) * Update dev/breeze/src/airflow_breeze/commands/developer_commands.py Co-authored-by: LIU ZHE YOU <[email protected]> --------- Co-authored-by: LIU ZHE YOU <[email protected]>
…57169) * Add JSON serialization for kubeconfig in AsyncKubernetesHook * Add debug logging for kubeconfig serialization in AsyncKubernetesHook * Ran perk --------- Co-authored-by: Kalyan R <[email protected]>
apache#58306) The instructions were partially outdated and quite a bit too chatty, they did not focus on actual tasks to do but also contained some digressions and more detailed description of some provider principles. This change makes it simpler to follow the process by the release manager, moves the docuentation of details to PROVIDERS.rst and removes largely duplicated PROVIDER_DISTRIBUTION_DETAILS.md The "--incremental-update" flag is specifically added and mentioned in the doc on what should be done when new commits are added after initial release notes have been prepared, guiding the release manager more explicitly what to do and providing instructions.
The default is 6s which is quite aggressive and causes throttling issues in the test harness. Tests don't need that kind of fidelity and can wait much longer between attempts to fire the trigger.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR fixes the broken test
test_async_write_logs_should_execute_successfully()in the KubernetesPodOperator test suite. The test was incorrectly implemented and was not properly testing the async log writing functionality.Changes
@patch(KUB_OP_PATH.format("client"))decorator to properly mockself.client.read_namespaced_pod_log()which is the actual method used by_write_logs()mocked_client.read_namespaced_pod_loginstead of incorrectly mockingmock_manager.read_pod_logs(which is never called)[test_logs.encode("utf-8")]as expected by the implementationmock_manager.await_pod_completion.return_valuefor proper test flowcaploginstead of a meaningless string assertionget_logs=Falseassertion to checkmocked_client.read_namespaced_pod_log.assert_not_called()instead of the incorrectpod_manager.read_pod_logscheckRoot Cause
The test had three main issues:
pod_manager.read_pod_logsbut_write_logs()actually callsself.client.read_namespaced_pod_log()directlyassert f"Container logs: {test_logs}"always passes (it's just checking if a string exists, not if it was logged)pod_manager.read_pod_logswhich is never called by_write_logs()Testing
test_async_write_logs_should_execute_successfully[True]✅test_async_write_logs_should_execute_successfully[False]✅Type of Change
Fixes apache#57515