Repository navigation
Add deferrable mode to QuickSight operator and sensor - #70218
Merged
Merged
Conversation
boring-cyborg
Bot
added
area:providers
kind:documentation
provider:amazon
o-nikolas
reviewed
Jul 25, 2026
o-nikolas
left a comment
o-nikolas
left a comment
Contributor
There was a problem hiding this comment.
Looks like there are conflicts and build failures that need looking at. Also, I'm not sure of the breaking change (the new 30min timeout). I think large ingestions will easily pass that and fail dags for folks using this operator. We could make the default much higher, or go through a deprecation process where we log that the user should switch to setting a high default before we remove it. Or wait until a major release for the amazon provider package.
vincbeck
reviewed
Jul 27, 2026
Contributor
Author
Raised the default wait to 36 hours: 4320 attempts at the 30s delay, or 2160 at the sensor's 60s poke interval. That's how long AWS itself waits on a SPICE ingestion in the CloudFormation wait policy, where 36 hours is both the default and the maximum: |
ozemin
force-pushed
the
quicksight-deferrable
branch
from
July 27, 2026 19:24
ca7307e to
5e254c3
Compare
Contributor
|
Static checks are failing |
The QuickSight create-ingestion wait path used a hand-written polling loop that held a worker slot with time.sleep and had no upper bound, so a stuck ingestion could block a worker indefinitely. Moving the wait to a custom boto waiter lets both the operator and the sensor defer to the triggerer with deferrable=True, matching the deferrable pattern used across the Amazon provider, and bounds the synchronous wait as well.
AWS waits up to 36 hours for a SPICE ingestion in its own CloudFormation wait policy, where 36 hours is both the default and the maximum. Refreshes that run for several hours are common, so a shorter cap fails Dags that previously waited without a limit. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-quicksight-dataset-ingestionwaitpolicy.html
check_interval is a template field, so a constructor that branches on it reads the un-rendered Jinja expression rather than the value the task will actually use. Emitting the warning where the deprecated value is resolved keeps it tied to the real input.
ozemin
force-pushed
the
quicksight-deferrable
branch
from
August 21, 2026 21:47
5e254c3 to
71ab048
Compare
o-nikolas
approved these changes
Aug 28, 2026
|
Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions. |
imrichardwu
pushed a commit
to imrichardwu/airflow
that referenced
this pull request
Sep 11, 2026
The QuickSight create-ingestion wait path used a hand-written polling loop that held a worker slot with time.sleep and had no upper bound, so a stuck ingestion could block a worker indefinitely. Moving the wait to a custom boto waiter lets both the operator and the sensor defer to the triggerer with deferrable=True, matching the deferrable pattern used across the Amazon provider, and bounds the synchronous wait as well. AWS waits up to 36 hours for a SPICE ingestion in its own CloudFormation wait policy, where 36 hours is both the default and the maximum. Refreshes that run for several hours are common, so a shorter cap fails Dags that previously waited without a limit.
1 task done
xvega
pushed a commit
to xvega/airflow
that referenced
this pull request
Sep 13, 2026
The QuickSight create-ingestion wait path used a hand-written polling loop that held a worker slot with time.sleep and had no upper bound, so a stuck ingestion could block a worker indefinitely. Moving the wait to a custom boto waiter lets both the operator and the sensor defer to the triggerer with deferrable=True, matching the deferrable pattern used across the Amazon provider, and bounds the synchronous wait as well. AWS waits up to 36 hours for a SPICE ingestion in its own CloudFormation wait policy, where 36 hours is both the default and the maximum. Refreshes that run for several hours are common, so a shorter cap fails Dags that previously waited without a limit.
dlactin
pushed a commit
to dlactin/airflow
that referenced
this pull request
Sep 14, 2026
The QuickSight create-ingestion wait path used a hand-written polling loop that held a worker slot with time.sleep and had no upper bound, so a stuck ingestion could block a worker indefinitely. Moving the wait to a custom boto waiter lets both the operator and the sensor defer to the triggerer with deferrable=True, matching the deferrable pattern used across the Amazon provider, and bounds the synchronous wait as well. AWS waits up to 36 hours for a SPICE ingestion in its own CloudFormation wait policy, where 36 hours is both the default and the maximum. Refreshes that run for several hours are common, so a shorter cap fails Dags that previously waited without a limit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
deferrable=Truesupport toQuickSightCreateIngestionOperatorandQuickSightSensor, following the same waiter-based pattern as the rest of the Amazon provider. QuickSight has no official boto3 waiters (boto/boto3#2769), so this adds a customingestion_completewaiter and aQuickSightIngestionCompletedTrigger, and replaces the hook's hand-written polling loop, which held a worker slot withtime.sleepand had no upper bound.One thing without a direct precedent:
DescribeIngestionrequiresAwsAccountId, so it is resolved on the worker before deferring and passed to the trigger — the triggerer never makes a sync STS call.Behaviour changes: the synchronous wait is now bounded by
waiter_max_attempts(default 60 attempts × 30s) instead of waiting forever; noted in the provider changelog.QuickSightHook.wait_for_stateand the operator'scheck_intervalare deprecated (still working) in favour ofwait_for_ingestion/waiter_delay. Failures raiseQuickSightIngestionFailedError, anAirflowExceptionsubclass.Was generative AI tooling used to co-author this PR?