Repository navigation
Honor download_num_attempts and upload_num_attempts in GCSTimeSpanFileTransformOperator - #72953
Conversation
…eTransformOperator Co-Authored-By: Claude Fable 5.1 <[email protected]>
|
Thanks for the review, @molcay. The helper is gone in favour of On system tests: I don't have a GCP project, so I haven't run If you have a moment, #72954 in the same provider (the IdP extra parameters that never reach the token request) has been waiting for a first review since the 11th. |
molcay
left a comment
There was a problem hiding this comment.
Hi @bingqin2,
Thank you for the changes.
LGTM.
For the #72954, It seems like @VladaZakharova already reviewed it yesterday.
GCSTimeSpanFileTransformOperatoracceptsdownload_num_attemptsandupload_num_attempts("Number of attempts to try to download/upload a single file"), but since #62196 moved the transfers fromGCSHook.download()/GCSHook.upload()to directblob.download_to_filename()/blob.upload_from_filename()calls, both values are stored and never read. Every blob gets exactly one attempt regardless of the setting, so a transientGoogleCloudErroreither fails the task or, with*_continue_on_fail, silently drops that file.This restores the retry behaviour the operator had through the hook: a small
_run_with_attemptshelper retriesGoogleCloudErrorup to the configured number of attempts with the hook's exponential back-off (2**attemptseconds), and both worker functions go through it. The default of one attempt keeps today's behaviour, and the*_continue_on_failhandling around the futures is unchanged.Changes
providers/google/src/airflow/providers/google/cloud/operators/gcs.py:_run_with_attemptshelper;_downloadand_uploadhonordownload_num_attemptsandupload_num_attemptsproviders/google/tests/unit/google/cloud/operators/test_gcs.py:test_download_honors_download_num_attemptsandtest_upload_honors_upload_num_attempts(retry, give-up and single-attempt cases;time.sleepis patched and its calls asserted), plus a shared_setup_transform_processhelperTesting
providers/google:tests/unit/google/cloud/operators/test_gcs.py(TestGCSTimeSpanFileTransformOperator, 28 tests)Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Claude Fable 5.1) following the guidelines. I reviewed and understand all changes; the tests were run locally as listed above.
🤖 Generated with Claude Code