Skip to content

Restore argument-provision checks to __init__ in operators fixed by the burn-down #70503

Closed

Description

@shahar1

Background

validate-operators-init flags every read of a template-field parameter in __init__. A companion PR narrows that: a field is None / field is not None check reads whether the argument was passed, not its value, and belongs in the constructor. (With render_template_as_native_obj=True a provided field renders to None, so the same check in execute() reports a supplied argument as missing; and raising at construction surfaces a static authoring mistake as a Dag import error rather than per task instance.)

Replaying both checker versions over every exemption entry removed so far (31 PRs, 48 entries) shows 8 entries across 5 PRs where the old rule forced a change that was not needed. Those PRs are merged and green — nothing is broken — but the files now carry comments and regression tests asserting the placement the docs discourage, and contributors working #70296 learn the rule by reading them.

This issue tracks putting them back. Each is independent; separate PRs are fine.

Blocked on #70505. Until it merges, validate-operators-init still flags every read of a template-field parameter in __init__, so a restored check fails the hook — a revert landed earlier would have to re-add the exemption entry it just removed.

Revert

Split, don't revert

  • providers/common/ai/.../operators/document_loader.py (Validate template fields after rendering in common.ai operators #70338) — the source_path/source_bytes exclusivity is provision and returns to __init__. The third check, source_bytes is not None and file_type is None, backs the downstream assert self.file_type is not None; file_type is a template field, so that assumption only holds after rendering and it stays in execute(). Update the comment to say which is which — this is the canonical example of the boundary.

Rewrite, don't revert

The replay above only finds reverts that are free — entries whose findings drop to zero with no edit. Provision checks written as truthiness (if a and b: raise) score the same under both checkers, so they never showed up in it, but they are the same kind of check and moving them to execute() cost more: on rendered values a mutual-exclusion check silently accepts the conflict when one side renders empty. Restoring these means rewriting and to is not None and, which is strictly stricter — it also rejects field="" alongside a second argument, and both were supplied.

Leave alone (listed so nobody reverts them by pattern-match)

Required-argument checks written as truthiness stay in execute() too. if not field: raise there still raises — it is later feedback, never a silent accept — and rewriting it to if field is None: raise in __init__ would loosen it, letting field="" through. Splitting each into a provision check plus a value check is more churn than the placement is worth:

Not in scope

The other 40 exemption entries removed so far are genuine value reads (ruleset.strip(), isinstance(mongo_query, list), source not in SUPPORTED_SOURCES, ...). They stay flagged under the narrowed rule and must not be reverted.

Verifying a revert

Run the hook on the file after the change — all classes above are clean under the narrowed rule, so no exemption entries need re-adding:

uv run --script scripts/ci/prek/validate_operators_init.py <path>

related: #70296


Drafted-by: Claude Code (Opus 5) (no human review before posting)

Activity

  1. ColtenOuO commented on Jul 27, 2026

    @ColtenOuO
    Contributor

    I'd like to try the neo4j one >< (I have open PR #70508 as draft and wait #70505 merged)

  2. ColtenOuO commented on Jul 27, 2026

    @ColtenOuO
    Contributor

    Hi @shahar1 , sorry for the direct ping, and no rush at all.

    Since #70508 no reviewer is assigned yet, would you have time to take a look whenever it suits you? I'd mainly like to confirm the approach is what you had in mind - in particular, I moved both provision checks rather than only cypher is None: after #70373, Neo4jOperator(sql=...) is valid with cypher unset, so the constructor check has to be cypher is None and sql is None, and
    splitting it from the mutual-exclusivity check felt arbitrary once written that way.

    If the approach looks right, I'd be glad to take databricks_repos.py, agent.py and document_loader.py as well.

  3. baha-bouali commented on Jul 27, 2026

    @baha-bouali
    Contributor

    Taking agent.py(revert) and document_loader.py (split) since they are both under providers/common/ai. I will be submitting a PR soon.

  4. ahilashsasidharan commented on Jul 28, 2026

    @ahilashsasidharan
    Contributor

    Hey @shahar1, wanted to check if my PR #70450 falls into the PRs needed to be restored, because it seems to fall into the class of the usual "exactly one of a or b" guard — belongs in __init__ and must not be moved.

    I already have a change to revert my PR and to restore it back to init, but it fails the prek hook:

    No templated field logic checks in operator __init__...........................................................Failed
    - hook id: validate-operators-init
    - exit code: 1
    
      amazon/src/airflow/providers/amazon/aws/operators/dms.py
      DmsModifyTaskOperator's constructor applies logic to template fields. Template fields are rendered after the constructor runs, so validation or transformation here acts on the un-rendered Jinja
      expression and should move to execute():
        line 197: if cdc_start_time is not None and cdc_start_position is not None:  (cdc_start_position, cdc_start_time)
      DmsStartReplicationOperator's constructor applies logic to template fields. Template fields are rendered after the constructor runs, so validation or transformation here acts on the un-rendered Jinja
      expression and should move to execute():
        line 791: if cdc_start_time is not None and cdc_start_pos is not None:  (cdc_start_pos, cdc_start_time)
    

    I am not sure if there is a separate PR to planned to narrow the hook so it doesn't flag this, and this was a case that was missed from your list. I could also be misunderstanding, so please correct me if I am wrong.

    If it needs to be restored I can push my change I have locally once the a PR to fix the rule is merged.

  5. shahar1 commented on Jul 28, 2026

    @shahar1
    ContributorAuthor

    Since #70508 no reviewer is assigned yet, would you have time to take a look whenever it suits you?

    Yes, currently it's drafted - ping me there when it's ready.

    I already have a change to revert my PR and to restore it back to init, but it fails the prek hook:

    I haven't merged #70505 yet, it shouldn't fail afterwards.

  6. ahilashsasidharan commented on Jul 29, 2026

    @ahilashsasidharan
    Contributor

    I've completed the DMS operators rewrite and it should be ready for review. I will take the providers/databricks/.../sensors/databricks.py::DatabricksSQLStatementsSensor and providers/snowflake/.../operators/snowpark_containers.py::SnowparkContainerJobOperator leftover ones from rewrite to help close this out. I should have a PR for those in the next day or two.

  7. eladkal commented on Sep 11, 2026

    @eladkal
    Contributor

    @shahar1 I think all items are adressed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:providerskind:metaHigh-level information important to the community

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions