Data Fusion pipeline state polling no longer retries transient 404 responses #72385

Under which category would you file this issue?

Providers

Apache Airflow version

2.11.1

What happened and how to reproduce it?

• Running a bunch of CloudDataFusionStartPipelineOperator in parallel
• Observing a third to half of them failing with error:

  File "/opt/python3.11/lib/python3.11/site-packages/airflow/providers/google/cloud/hooks/datafusion.py", line 185, in _check_response_status_and_data
    raise HTTPError(message)
requests.exceptions.HTTPError: Retrieving a pipeline state failed with code 404

What you think should happen instead?

• It should not fail because this introduces a regression of a fix delivered 6 years ago: Add try clause to DataFusionHook.wait_for_pipeline_state #10031
• Clearly the PR In GCP Datafusion hook handle start_pipeline() when response is None / Invalid #60688 makes no sense because the "new exceptions chosen" are not in sync: One function raises HTTPError while the one calling that function has an except KeyError... I guess AI has at least this advantage that it would catch such simple error as it's 100% a logic bug and 100% contained in a single PR... Maybe activate CodeRabbit or whatever competitor that is free for OSS?

Operating System

No response

Deployment

Google Managed Service for Apache Airflow

Apache Airflow Provider(s)

google

Versions of Apache Airflow Providers

apache-airflow-providers-google==22.2.2
https://docs.cloud.google.com/composer/docs/versions-packages#composer-2-17-9-airflow-2-11-1

Official Helm Chart version

Not Applicable

Kubernetes Version

Not Applicable

Helm Chart configuration

Not Applicable

Docker Image customizations

Not Applicable

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct