Repository navigation
Description
Background
validate-operators-init flags every read of a template-field parameter in __init__. A companion PR narrows that: a field is None / field is not None check reads whether the argument was passed, not its value, and belongs in the constructor. (With render_template_as_native_obj=True a provided field renders to None, so the same check in execute() reports a supplied argument as missing; and raising at construction surfaces a static authoring mistake as a Dag import error rather than per task instance.)
Replaying both checker versions over every exemption entry removed so far (31 PRs, 48 entries) shows 8 entries across 5 PRs where the old rule forced a change that was not needed. Those PRs are merged and green — nothing is broken — but the files now carry comments and regression tests asserting the placement the docs discourage, and contributors working #70296 learn the rule by reading them.
This issue tracks putting them back. Each is independent; separate PRs are fine.
Blocked on #70505. Until it merges, validate-operators-init still flags every read of a template-field parameter in __init__, so a restored check fails the hook — a revert landed earlier would have to re-add the exemption entry it just removed.
Revert
-
providers/common/ai/.../operators/agent.py(Validate template fields after rendering in common.ai operators #70338) —message_history is not None and enable_hitl_reviewis pure provision. Restore to__init__, reverttest_agent.py::test_message_history_with_hitl_review_raisesto constructor-raises. The comment# message_history is a template field; validate the combination after rendering.states the rule wrongly and must go. -
providers/databricks/.../operators/databricks_repos.py(Validate Databricks Repos operators' template fields after rendering #70341) — all fourbranch/tag/repo_id/repo_pathchecks inDatabricksReposCreate/Update/DeleteOperatorare pure provision. Restore to__init__; revert the two tests docstringed "rejected at execute, after rendering". -
providers/neo4j/.../operators/neo4j.py(Fix Neo4jOperator templated query validation #70348) —if cypher is None: raisereturns to__init__. The deprecation-warning placement is a separate question; Show the deprecated Neo4jOperator sql query in Rendered Templates #70373's rationale stands independently.
Split, don't revert
-
providers/common/ai/.../operators/document_loader.py(Validate template fields after rendering in common.ai operators #70338) — thesource_path/source_bytesexclusivity is provision and returns to__init__. The third check,source_bytes is not None and file_type is None, backs the downstreamassert self.file_type is not None;file_typeis a template field, so that assumption only holds after rendering and it stays inexecute(). Update the comment to say which is which — this is the canonical example of the boundary.
Rewrite, don't revert
The replay above only finds reverts that are free — entries whose findings drop to zero with no edit. Provision checks written as truthiness (if a and b: raise) score the same under both checkers, so they never showed up in it, but they are the same kind of check and moving them to execute() cost more: on rendered values a mutual-exclusion check silently accepts the conflict when one side renders empty. Restoring these means rewriting and to is not None and, which is strictly stricter — it also rejects field="" alongside a second argument, and both were supplied.
-
providers/amazon/.../aws/operators/dms.py(Move amazon dms operators template-field validation to execute() #70450) —DmsModifyTaskOperator(cdc_start_time and cdc_start_position) andDmsStartReplicationOperator(self.cdc_start_time and self.cdc_start_pos). Revert the two tests to constructor-raises; while there,DmsStartReplicationOperatorshould raiseValueErrorrather thanAirflowException. The commit is not inproviders-amazon/9.33.0rc1, so fixing before the next release costs no changelog entry. -
providers/databricks/.../sensors/databricks.py(Validate DatabricksSQLStatementsSensor statement fields after rendering #70340) —DatabricksSQLStatementsSensor, thestatement and statement_idexclusivity check only. Itsnot statement and not statement_idsibling is a required-argument check — see below. -
providers/snowflake/.../operators/snowpark_containers.py(Move SnowparkContainerJobOperator template-field validation to execute #70332) —SnowparkContainerJobOperator, theself.spec_text and (self.spec or self.spec_stage)exclusivity check only. Same split as above for its inverse.
Leave alone (listed so nobody reverts them by pattern-match)
providers/weaviate/.../operators/weaviate.py(Fix WeaviateIngestOperator input_data validation before rendering #70326) — makinginput_datarequired is already released; reverting would be a second API break. Theexecute()check becomes belt-and-braces and is harmless.providers/teradata/.../transfers/teradata_to_teradata.py(Fix Teradata transfer template field initialization #70392) —self.sql_params or {}at the use site is equivalent. No value in churning it back.
Required-argument checks written as truthiness stay in execute() too. if not field: raise there still raises — it is later feedback, never a silent accept — and rewriting it to if field is None: raise in __init__ would loosen it, letting field="" through. Splitting each into a provision check plus a value check is more churn than the placement is worth:
providers/papermill/.../operators/papermill.py(Fix template-field validation in PapermillOperator #70435) —not input_nb,not output_nbproviders/apache/kafka/.../operators/produce.py(Validate ProduceToTopicOperator topic after rendering #70333) —not (self.topic and self.producer_function)providers/databricks/.../sensors/databricks.py(Validate DatabricksSQLStatementsSensor statement fields after rendering #70340) —not statement and not statement_idproviders/cncf/kubernetes/.../operators/resource.py(Validate KubernetesResourceBaseOperator yaml_conf after rendering #70337) —not any([self.yaml_conf, self.yaml_conf_file])providers/snowflake/.../operators/snowpark_containers.py(Move SnowparkContainerJobOperator template-field validation to execute #70332) —not self.spec_text and not (self.spec and self.spec_stage)
Not in scope
The other 40 exemption entries removed so far are genuine value reads (ruleset.strip(), isinstance(mongo_query, list), source not in SUPPORTED_SOURCES, ...). They stay flagged under the narrowed rule and must not be reverted.
Verifying a revert
Run the hook on the file after the change — all classes above are clean under the narrowed rule, so no exemption entries need re-adding:
uv run --script scripts/ci/prek/validate_operators_init.py <path>
related: #70296
Drafted-by: Claude Code (Opus 5) (no human review before posting)
Activity
ColtenOuO commentedon Jul 27, 2026
I'd like to try the neo4j one >< (I have open PR #70508 as draft and wait #70505 merged)
ColtenOuO commentedon Jul 27, 2026
Hi @shahar1 , sorry for the direct ping, and no rush at all.
Since #70508 no reviewer is assigned yet, would you have time to take a look whenever it suits you? I'd mainly like to confirm the approach is what you had in mind - in particular, I moved both provision checks rather than only
cypher is None: after #70373,Neo4jOperator(sql=...)is valid withcypherunset, so the constructor check has to becypher is None and sql is None, andsplitting it from the mutual-exclusivity check felt arbitrary once written that way.
If the approach looks right, I'd be glad to take
databricks_repos.py,agent.pyanddocument_loader.pyas well.baha-bouali commentedon Jul 27, 2026
Taking
agent.py(revert) anddocument_loader.py(split) since they are both underproviders/common/ai. I will be submitting a PR soon.ahilashsasidharan commentedon Jul 28, 2026
Hey @shahar1, wanted to check if my PR #70450 falls into the PRs needed to be restored, because it seems to fall into the class of
the usual "exactly one of a or b" guard — belongs in __init__ and must not be moved.I already have a change to revert my PR and to restore it back to init, but it fails the prek hook:
I am not sure if there is a separate PR to planned to narrow the hook so it doesn't flag this, and this was a case that was missed from your list. I could also be misunderstanding, so please correct me if I am wrong.
If it needs to be restored I can push my change I have locally once the a PR to fix the rule is merged.
shahar1 commentedon Jul 28, 2026
Yes, currently it's drafted - ping me there when it's ready.
I haven't merged #70505 yet, it shouldn't fail afterwards.
ahilashsasidharan commentedon Jul 29, 2026
I've completed the DMS operators rewrite and it should be ready for review. I will take the
providers/databricks/.../sensors/databricks.py::DatabricksSQLStatementsSensorandproviders/snowflake/.../operators/snowpark_containers.py::SnowparkContainerJobOperatorleftover ones from rewrite to help close this out. I should have a PR for those in the next day or two.Fail Neo4jOperator argument mistakes at Dag parse time (#70508)
Fail Neo4jOperator argument mistakes at Dag parse time (apache#70508)
eladkal commentedon Sep 11, 2026
@shahar1 I think all items are adressed?
Fail Neo4jOperator argument mistakes at Dag parse time (apache#70508)
Fix `DocumentLoaderOperator` validation errors naming the wrong argum…
Fix `DocumentLoaderOperator` validation errors naming the wrong argum…