Skip to content

Commit e8ae850

Browse files
Reject empty match_glob in GCSToS3Operator on old google provider (#71591)
match_glob is a template field, so its value is not available until Jinja rendering happens on the worker, well after __init__ has run. The guard that rejects match_glob on an unsupported google provider tested it for truthiness, which the validate-operators-init prek hook flags. The check only asks whether the argument was supplied; it never inspects the value. Per the false-positive guidance on #70296 that makes it a provision check, which is fixed in place rather than moved to execute(). Moving it would break the check under render_template_as_native_obj=True, where a supplied field can render to None and so becomes indistinguishable from an omitted one, and would defer a static authoring mistake from Dag parse time to every task instance and retry. Switching to an explicit is not None comparison changes one case: match_glob="" is now rejected on a google provider older than 10.3.0, where truthiness previously read the supplied empty string as absent. An empty glob is not a valid pattern and was never honoured on those versions. Narrow the raised exception from AirflowException to ValueError, which is what an invalid argument warrants, and drop the file from generated/known_airflow_exceptions.txt accordingly. This follows the merged #70359 precedent for S3DeleteObjectsOperator, which made the same one-for-one change to the same four kinds of file. Also remove the class from the prek exemption list, since the hook fails on stale entries, and add tests covering supplied, empty-string and omitted match_glob. Related: #70296
1 parent df9a3e7 commit e8ae850

4 files changed

Lines changed: 36 additions & 7 deletions

File tree

‎generated/known_airflow_exceptions.txt‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -112,7 +112,6 @@ providers/amazon/src/airflow/providers/amazon/aws/sensors/sagemaker.py::8
112112
providers/amazon/src/airflow/providers/amazon/aws/sensors/sagemaker_unified_studio.py::1
113113
providers/amazon/src/airflow/providers/amazon/aws/sensors/sqs.py::2
114114
providers/amazon/src/airflow/providers/amazon/aws/sensors/step_function.py::1
115-
providers/amazon/src/airflow/providers/amazon/aws/transfers/gcs_to_s3.py::1
116115
providers/amazon/src/airflow/providers/amazon/aws/transfers/redshift_to_s3.py::1
117116
providers/amazon/src/airflow/providers/amazon/aws/transfers/s3_to_dynamodb.py::3
118117
providers/amazon/src/airflow/providers/amazon/aws/transfers/s3_to_redshift.py::3

‎providers/amazon/src/airflow/providers/amazon/aws/transfers/gcs_to_s3.py‎

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@
2626
from packaging.version import Version
2727

2828
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
29-
from airflow.providers.common.compat.sdk import AirflowException, BaseOperator
29+
from airflow.providers.common.compat.sdk import BaseOperator
3030
from airflow.providers.google.cloud.hooks.gcs import GCSHook
3131

3232
if TYPE_CHECKING:
@@ -146,10 +146,8 @@ def __init__(
146146
self.__is_match_glob_supported = False
147147
except ImportError: # __version__ was added in 10.1.0, so this means it's < 10.3.0
148148
self.__is_match_glob_supported = False
149-
if not self.__is_match_glob_supported and match_glob:
150-
raise AirflowException(
151-
"The 'match_glob' parameter requires 'apache-airflow-providers-google>=10.3.0'."
152-
)
149+
if not self.__is_match_glob_supported and match_glob is not None:
150+
raise ValueError("The 'match_glob' parameter requires 'apache-airflow-providers-google>=10.3.0'.")
153151
self.match_glob = match_glob
154152
self.gcp_user_project = gcp_user_project
155153

‎providers/amazon/tests/unit/amazon/aws/transfers/test_gcs_to_s3.py‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,39 @@ def test_execute__match_glob(self, mock_hook):
7575
user_project=None,
7676
)
7777

78+
@pytest.mark.parametrize(
79+
"match_glob",
80+
[
81+
pytest.param("**/*.csv", id="value_supplied"),
82+
pytest.param("", id="empty_string_is_still_supplied"),
83+
],
84+
)
85+
@mock.patch("airflow.providers.google.__version__", "10.2.0")
86+
def test_init__match_glob_rejected_on_older_google_provider(self, match_glob):
87+
"""Supplying match_glob requires apache-airflow-providers-google>=10.3.0.
88+
89+
The check asks whether the argument was supplied, not whether its value is truthy,
90+
so an empty string is rejected too - it was supplied.
91+
"""
92+
with pytest.raises(ValueError, match=r"requires 'apache-airflow-providers-google>=10\.3\.0'"):
93+
GCSToS3Operator(
94+
task_id=TASK_ID,
95+
gcs_bucket=GCS_BUCKET,
96+
dest_s3_key=S3_BUCKET,
97+
match_glob=match_glob,
98+
)
99+
100+
@mock.patch("airflow.providers.google.__version__", "10.2.0")
101+
def test_init__match_glob_omitted_is_accepted_on_older_google_provider(self):
102+
"""Omitting match_glob leaves nothing to reject, whatever the google provider version."""
103+
operator = GCSToS3Operator(
104+
task_id=TASK_ID,
105+
gcs_bucket=GCS_BUCKET,
106+
dest_s3_key=S3_BUCKET,
107+
)
108+
109+
assert operator.match_glob is None
110+
78111
@mock.patch("airflow.providers.amazon.aws.transfers.gcs_to_s3.GCSHook")
79112
def test_execute_incremental(self, mock_hook):
80113
mock_hook.return_value.list.return_value = MOCK_FILES

‎scripts/ci/prek/validate_operators_init_exemptions.txt‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,6 @@
77
# Burn-down tracked at https://github.com/apache/airflow/issues/70296
88
providers/amazon/src/airflow/providers/amazon/aws/operators/neptune.py::NeptuneStartDbClusterOperator
99
providers/amazon/src/airflow/providers/amazon/aws/operators/neptune.py::NeptuneStopDbClusterOperator
10-
providers/amazon/src/airflow/providers/amazon/aws/transfers/gcs_to_s3.py::GCSToS3Operator
1110
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py::KubernetesPodOperator
1211
providers/google/src/airflow/providers/google/cloud/operators/cloud_build.py::CloudBuildCreateBuildOperator
1312
providers/google/src/airflow/providers/google/cloud/operators/cloud_storage_transfer_service.py::CloudDataTransferServiceCreateJobOperator

0 commit comments

Comments
 (0)