docs: adopt issue #4419 terminology in Understanding Comet Plans guide - #4650

…s guide (apache#4650)

…s guide (apache#4650)

apache#4650 rewrote this page. Fix the uses of "native" that remain: say
"Rust" where the text means the Rust implementation or the Rust plan,
"in the Comet pipeline" where it means Comet as a whole, and "Spark's
built-in shuffle" instead of "Spark's native shuffle". Config keys, API
names, and quoted explain output are unchanged.

prettier realigns the Scans table because two of its rows changed.

Part of apache#4419.

Generated-by: OpenCode

apache#4650 rewrote this page. Fix the uses of "native" that remain: say
"Rust" where the text means the Rust implementation or the Rust plan,
"in the Comet pipeline" where it means Comet as a whole, and "Spark's
built-in shuffle" instead of "Spark's native shuffle". Config keys, API
names, and quoted explain output are unchanged.

prettier realigns the Scans table because two of its rows changed.

Part of apache#4419.

Generated-by: OpenCode

apache#4650 rewrote this page. Fix the uses of "native" that remain: say
"Rust" where the text means the Rust implementation or the Rust plan,
"in the Comet pipeline" where it means Comet as a whole, and "Spark's
built-in shuffle" instead of "Spark's native shuffle". Config keys, API
names, and quoted explain output are unchanged.

Also say that consecutive Rust-implemented operators are usually, not
always, combined into one block, and add CometTakeOrderedAndProject to
the exceptions: it runs its own final Rust plan, after a single-partition
shuffle when its input has several partitions.

prettier realigns the Scans table because two of its rows changed.

Part of apache#4419.

Generated-by: OpenCode

apache#4650 rewrote this page. Fix the uses of "native" that remain: say
"Rust" where the text means the Rust implementation or the Rust plan,
"in the Comet pipeline" where it means Comet as a whole, and "Spark's
built-in shuffle" instead of "Spark's native shuffle". Config keys, API
names, and quoted explain output are unchanged.

Also say that consecutive Rust-implemented operators are usually, not
always, combined into one block, and add CometTakeOrderedAndProject to
the exceptions: it runs its own final Rust plan, after a single-partition
shuffle when its input has several partitions.

prettier realigns the Scans table because two of its rows changed.

Part of apache#4419.

Generated-by: OpenCode

apache#4650 rewrote this page. Fix the uses of "native" that remain: say
"Rust" where the text means the Rust implementation or the Rust plan,
"in the Comet pipeline" where it means Comet as a whole, and "Spark's
built-in shuffle" instead of "Spark's native shuffle". Config keys, API
names, and quoted explain output are unchanged.

Also say that consecutive Rust-implemented operators are usually, not
always, combined into one block, and add CometTakeOrderedAndProject to
the exceptions: it runs its own final Rust plan, after a single-partition
shuffle when its input has several partitions.

Two small fixes on the same page: the explain section documents six
configs, not five, so add the missing spark.comet.explain.planOnly.enabled
row to its table; and CometCollectLimit is not purely JVM-side, since it
runs Rust limit plans when a Comet operator reads its output.

prettier realigns the Scans table because two of its rows changed.

Part of apache#4419.

Generated-by: OpenCode

apache#4650 rewrote this page. Fix the uses of "native" that remain: say
"Rust" where the text means the Rust implementation or the Rust plan,
"in the Comet pipeline" where it means Comet as a whole, and "Spark's
built-in shuffle" instead of "Spark's native shuffle". Config keys, API
names, and quoted explain output are unchanged.

Also say that consecutive Rust-implemented operators are usually, not
always, combined into one block, and add CometTakeOrderedAndProject to
the exceptions: it runs its own final Rust plan, after a single-partition
shuffle when its input has several partitions.

Two small fixes on the same page: the explain section documents seven
configs, not five, so add the missing spark.comet.explain.planOnly.enabled
and spark.comet.explain.rules rows to its table; and CometCollectLimit is
not purely JVM-side, since it runs Rust limit plans unless Spark collects
it at the plan root, so say so and adjust the JVM-side section's intro.

prettier realigns the Scans table because two of its rows changed.

Part of apache#4419.

Generated-by: OpenCode

…ages (apache#7151)

* docs: add a documentation style guide

Add docs/source/contributor-guide/style_guide.md, which writes down the
vocabulary from apache#4419 (pipeline membership, implementation language and
data format), the rule for "native", how to name operators in prose,
and the page conventions that the docs-review skill in apache#4420 cites.

Each rule has a stable ID so that reviews can cite it. The guide follows
the code where apache#4419 and the code differ: "Hybrid" keeps the meaning it
already has in expressions.md, expressions are described separately
from plan nodes, and "Arrow-native" is a claim about data format only.
The Generated Content section names the generated text exactly, by the
markers GenerateDocs fills in, including the config tables built from
QueryPlanSerde and contrib config classes, and the cast tables built
from CometCast. P18 covers attribution of third-party content, which
apache#4420 lists among its checks. The guide's scope covers .doc strings in
contrib config classes, and its vocabulary and accuracy rules also apply
to doc comments, AGENTS.md and agent skills.

Link the page from the contributor guide toctree and from AGENTS.md.

Part of apache#4419.

Generated-by: OpenCode

* docs: correct the plugin overview and adopt the apache#4419 terminology

Correct statements in plugin_overview.md that no longer match the code:

- CometSparkSessionExtensions registers more than CometRule: it also
  registers rules that run after columnar transitions are inserted, AQE
  query stage optimizer rules on Spark 3.5 and later, and
  IcebergWriteStrategy.
- Comet does replace part of a query stage, and an unsupported operator
  stays in Spark, not the whole plan. Describe the post-transition rules
  in the order CometRule runs them: RevertNativeForTransitionHeavyStages
  first, which can decline an unsafe revert, then
  EliminateRedundantTransitions, which uses CometColumnarToRowExec, or
  CometNativeColumnarToRowExec when that opt-in config is set and the
  converter supports the schema.
- Not every leaf of the Rust plan is a ScanExec. ScanExec takes Arrow
  batches from the JVM; a direct shuffle read of a Comet shuffle with
  supported types is a ShuffleScanExec; the Parquet, Iceberg and CSV
  scans read their files in Rust.
- Blocks are formed from consecutive CometNativeExec operators, except
  that the write operators start a new block for their child. The native
  shuffle writer absorbs a CometNativeExec child.

Add a paragraph on the JVM codegen dispatcher, which apache#4419 asks this
page to cover: some expressions in a Rust plan run Spark's generated JVM
code over Arrow batches through JNI, and the operator around them stays
in the Comet pipeline. Scala and Java UDFs take the same path.

Also replace the vague uses of "native" with the apache#4419 terms. The scan
section (Phase 1) is left alone, because apache#6052 rewrites it.

Part of apache#4419.

Generated-by: OpenCode

* docs: adopt the apache#4419 terminology in the native shuffle guide

Replace vague uses of "native" in native_shuffle.md. Native shuffle
accepts any CometPlan child, so call the child a Comet operator rather
than a native one. Where "native" was used as a noun, name the Rust
component instead.

Also correct statements this touched:

- When RevertNativeForTransitionHeavyStages restores a stage, a native
  shuffle whose input came from spark.comet.convert.shuffleInput.enabled
  switches back to JVM columnar shuffle.
- When direct read cannot serve a shuffle stage, the stage stays as it
  is and the operators that read it cannot join a Rust plan. The check
  is QueryPlanSerde.supportedDataType; supportedSinkDataType does not
  exist.
- Drop the unverified reason given for rejecting nested range keys.
- Describe spark.comet.shuffle.directRead.enabled the same way as the
  JVM shuffle guide.

Contributor-guide forms such as "native code" and "native plan", the
"native shuffle" name, and config keys are unchanged.

Part of apache#4419.

Generated-by: OpenCode

* docs: adopt the apache#4419 terminology in the JVM shuffle guide

Replace vague uses of "native" in jvm_shuffle.md. Say which parts of
JVM shuffle run on the JVM (partitioning) and which run in Rust (Arrow
IPC encoding), and call a shuffle's child a Comet operator, since native
shuffle accepts any CometPlan child, including JVM-side ones.

Say that direct read applies when a Comet block reads the shuffle as a
ShuffleScan input. Also describe spark.comet.shuffle.directRead.enabled
accurately: both read
paths decode in Rust, and direct read decodes inside the Rust plan,
bypassing Arrow FFI, as the config's own doc string says.

Part of apache#4419.

Generated-by: OpenCode

* docs: adopt the apache#4419 terminology in the Gluten comparison

Replace vague uses of "native" with the term each sentence means: the
Comet pipeline, Rust, or Velox on the Gluten side. Describe Velox and
DataFusion as columnar query engines rather than vectorized ones.

Two factual fixes came up while rewording:

- The table formats section said Comet has no Delta Lake integration.
  main now has an experimental, opt-in Delta Lake scan (apache#5365), so say
  that and link the Delta Lake guide.
- Link the compatibility guide by its path in the docs rather than by its
  published URL, and point the two Gluten limitations links, which
  return 404, at the page in Gluten's v1.7.0 sources.

Part of apache#4419.

Generated-by: OpenCode

* docs: finish the apache#4419 terminology in Understanding Comet Plans

apache#4650 rewrote this page. Fix the uses of "native" that remain: say
"Rust" where the text means the Rust implementation or the Rust plan,
"in the Comet pipeline" where it means Comet as a whole, and "Spark's
built-in shuffle" instead of "Spark's native shuffle". Config keys, API
names, and quoted explain output are unchanged.

Also say that consecutive Rust-implemented operators are usually, not
always, combined into one block, and add CometTakeOrderedAndProject to
the exceptions: it runs its own final Rust plan, after a single-partition
shuffle when its input has several partitions.

Two small fixes on the same page: the explain section documents seven
configs, not five, so add the missing spark.comet.explain.planOnly.enabled
and spark.comet.explain.rules rows to its table; and CometCollectLimit is
not purely JVM-side, since it runs Rust limit plans unless Spark collects
it at the plan root, so say so and adjust the JVM-side section's intro.

prettier realigns the Scans table because two of its rows changed.

Part of apache#4419.

Generated-by: OpenCode

* docs: replace "DataFusion native engine" on the docs landing page

Use the apache#4419 terminology on docs/source/index.md, which the README
rewrite in apache#4428 did not cover. The page now says what Comet does with
DataFusion rather than calling it a native engine, and the opening line
carries the README's Arrow framing, as apache#4419 asks.

Say that Comet hands DataFusion the operators it runs in Rust, not all
supported operators, since some run on the JVM. Also drop "powerful" from the edited paragraph, and link the
benchmarking guide by its relative path rather than its published URL.

Part of apache#4419.

Generated-by: OpenCode

* docs: qualify direct read, explain.rules, and shuffle child rules

- plugin_overview.md: a shuffle input is a ShuffleScanExec only when it
  is serialized as a ShuffleScan leaf, which needs direct read, an AQE
  query stage, and supported output types. Without AQE, the shuffle is
  read through ScanExec.
- understanding-comet-plans.md: spark.comet.explain.rules logs the plan
  before and after CometScanRule, CometExecRule and
  EliminateRedundantTransitions, not stage reverts. Point to
  spark.comet.explain.fallback.log.enabled for those.
- jvm_shuffle.md: scope the columnar-to-row diagram to children that
  produce Arrow batches. A CometSparkToColumnarExec child returns its
  input rows directly.
- native_shuffle.md: with the Celeborn shuffle manager, native shuffle
  needs a CometNativeExec child, not any CometPlan.

Part of apache#4419.

Generated-by: OpenCode

* docs: match the shuffle read condition, drop native and seamless from the home page

The Shuffle Reads paragraph in plugin_overview.md now gives the same
ShuffleScan condition as the paragraph above it and jvm_shuffle.md. The
overview figure and the index.md compatibility blurb write Rust instead
of native (N3) and drop seamlessly (P10).

Generated-by: OpenCode

* docs: say what decides the CometCollectLimit path, fix two overview figure labels

CometCollectLimit takes rows on the JVM for collect, show and take,
through executeCollect, and runs Rust limit plans otherwise; where it
sits in the plan does not decide this. The overview figure no longer
says every supported operator runs in Rust (CometUnion does not), and
drops vectorized as a synonym for columnar (T10).

Generated-by: OpenCode

* docs: give the full Celeborn shuffle condition, address the reader on the Gluten page

With the Celeborn shuffle manager, native shuffle also needs
spark.comet.shuffle.mode=native and the manager's own support checks
(cometCelebornShuffleFallbackReason), not only a CometNativeExec child.
Two changed lines on the Gluten page now address the reader (P9).

Generated-by: OpenCode

* docs: make the native rules testable

N1 now covers every form of the banned phrases, such as ran natively and
Comet-native. N2 allows the native shuffle path without pairing it with
JVM shuffle. N3 replaces and similar with a test, names user-guide/, and
says pages outside the contributor guide allow only the N2 compounds.
Agents running the guide could not apply the old wording consistently.

Generated-by: OpenCode

* docs: complete the CometRule and Celeborn conditions, say Rust plan for a fused block

Under Adaptive Query Execution, CometRule also runs as the columnar
rule on each new query stage, not only as a stage preparation rule. The
Celeborn condition names nativeShuffleFallbackReason, which holds the
manager's checks, instead of a user-guide section that only covers
released clients. The pages already say Rust plan for a group of
operators that run together in Rust, so the three Comet block mentions
use it too.

Generated-by: OpenCode

* docs: allow Arrow-native in N2 and close the native rules

T7 lists Arrow-native, which N3's limit to N2 compounds outside the
contributor guide would otherwise forbid. N3 also says any other use of
native is a finding, so a phrase outside the lists has an outcome.

Generated-by: OpenCode

* docs: limit the N2 scan compounds to Rust-implemented scans

A dry run read native cache scan as allowed by N2, but the in-memory
cache scan runs on the JVM. N2 now says the scan and writer compounds
apply only to Rust-implemented ones.

Generated-by: OpenCode

* docs: point N1 at T1 for not-a-fallback and at N5 for Spark

A dry run rewrote "falls back to its native execution" and "executes
natively (not falling back to Spark)" wrongly, because N1 offered only
T1 or T3. N1 now says not falling back is T1, since codegen dispatch is
not a fallback, and that Spark's own behavior is N5.

Generated-by: OpenCode

* docs: apply the style guide to new and changed text

Existing pages move over as they are edited, so a review does not ask a
change to restyle text it leaves alone, unless the change makes that
text wrong (P16). Requested in review.

Generated-by: OpenCode

* docs: call Velox and DataFusion vectorized query engines again

The sentence is about batch-at-a-time execution, which T10 allows
vectorized for, and both projects describe themselves that way.
Columnar is a claim about data layout. Requested in review.

Generated-by: OpenCode