Skip to content

Honor download_num_attempts and upload_num_attempts in GCSTimeSpanFileTransformOperator - #72953

Merged
shahar1 merged 2 commits into
apache:mainfrom
bingqin2:gcs-timespan-retry-attempts
Oct 3, 2026
Merged

shahar1 merged 2 commits into
apache:mainfrom
bingqin2:gcs-timespan-retry-attempts

Conversation

@bingqin2

Copy link
Copy Markdown
Contributor

GCSTimeSpanFileTransformOperator accepts download_num_attempts and upload_num_attempts ("Number of attempts to try to download/upload a single file"), but since #62196 moved the transfers from GCSHook.download() / GCSHook.upload() to direct blob.download_to_filename() / blob.upload_from_filename() calls, both values are stored and never read. Every blob gets exactly one attempt regardless of the setting, so a transient GoogleCloudError either fails the task or, with *_continue_on_fail, silently drops that file.

This restores the retry behaviour the operator had through the hook: a small _run_with_attempts helper retries GoogleCloudError up to the configured number of attempts with the hook's exponential back-off (2**attempt seconds), and both worker functions go through it. The default of one attempt keeps today's behaviour, and the *_continue_on_fail handling around the futures is unchanged.

Changes

  • providers/google/src/airflow/providers/google/cloud/operators/gcs.py: _run_with_attempts helper; _download and _upload honor download_num_attempts and upload_num_attempts
  • providers/google/tests/unit/google/cloud/operators/test_gcs.py: test_download_honors_download_num_attempts and test_upload_honors_upload_num_attempts (retry, give-up and single-attempt cases; time.sleep is patched and its calls asserted), plus a shared _setup_transform_process helper

Testing

  • providers/google: tests/unit/google/cloud/operators/test_gcs.py (TestGCSTimeSpanFileTransformOperator, 28 tests)
  • mypy on the changed module, prek hooks on the changed files

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Claude Fable 5.1) following the guidelines. I reviewed and understand all changes; the tests were run locally as listed above.


🤖 Generated with Claude Code

@molcay molcay left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @bingqin2,

Thank you for your PR!

Can you have a look at my comments?

Also, please share your system tests run results.

@bingqin2

Copy link
Copy Markdown
Contributor Author

Thanks for the review, @molcay. The helper is gone in favour of tenacity.retry (details in the threads), and the 28 time-span unit tests pass, including the retry, give-up and single-attempt cases for both downloads and uploads.

On system tests: I don't have a GCP project, so I haven't run example_gcs_transform_timespan.py. That test covers the happy path, where the default of one attempt keeps today's behaviour; the retry path only runs on transient GoogleCloudErrors, which the unit tests simulate. If your team can run it on the system-test dashboard, that would confirm the real transfer path.

If you have a moment, #72954 in the same provider (the IdP extra parameters that never reach the token request) has been waiting for a first review since the 11th.

@molcay molcay left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @bingqin2,

Thank you for the changes.

LGTM.

For the #72954, It seems like @VladaZakharova already reviewed it yesterday.

@shahar1
shahar1 merged commit 9135355 into apache:main Oct 3, 2026
86 checks passed
@bingqin2
bingqin2 deleted the gcs-timespan-retry-attempts branch October 5, 2026 23:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants