docs(optimizer): add generated optimizer rules reference - #21824

cc @comphead

Hi @kumarUjjawal , i don't know if i am write but as per @comphead's comment

The gap: There is no reference doc that lists and describes what each built-in rule actually does. The codebase has 27 logical optimizer rules (e.g., push_down_filter, eliminate_cross_join,
common_subexpr_eliminate) and 21 physical optimizer rules (e.g., join_selection, enforce_sorting, topk_aggregation), but the only way to learn what each one does is to read the source or use EXPLAIN
VERBOSE.

i think what we really need is an end to end doc that describe all 21 optimizer both type , could you look into this ?

Thanks for the feedback @Adez017 The missing doc here is a reference page for the built-in rules, not another end-to-end optimizer guide. We already have docs for how the optimizer works and how to extend it.

This PR adds the missing reference: which built-in rules exist and what each one does, and links it from the existing optimizer guide.

Thanks for the feedback @Adez017 The missing doc here is a reference page for the built-in rules, not another end-to-end optimizer guide. We already have docs for how the optimizer works and how to extend it.

This PR adds the missing reference: which built-in rules exist and what each one does, and links it from the existing optimizer guide.

Thanks for clarification @kumarUjjawal

Thanks @comphead I agree, I think a fully hand written or as you said LLM generated page could also work, but in that case I would still want some kind of sync check to stop it drifting. Given that, a small generator felt like the safer option.

The reason I leaned toward generation here is that this page is meant to be reference docs, so the rule names, order, stages, and repeated physical passes should stay aligned with the actual default pipelines and with what users see in EXPLAIN VERBOSE.

I can rework this if doc generator is too much for this PR.

I can rework this if doc generator is too much for this PR.

making documentation in sync with optimizer rules makes a lot of sense, there is no guard right now.

Reg to generator, currently we have automated generation based proc macros for built in function and having another generator would be confusing IMO, however you trigger an awesome idea of having unified documentation framework that works for the entire core. Having that we would cover all aspects of core documentation + automatic md generation

currently we have automated generation based proc macros for built in function and having another generator would be confusing IMO

I see we do not want to end up with a bunch of one-off doc generators.

having unified documentation framework that works for the entire core. Having that we would cover all aspects of core documentation + automatic md generation

I think we should work toward this rather than having generator for each peices like this PR.

I will remove the optimizer-specific generator, maybe you can start the discussion around the shared docs generation framework.

I think we should work toward this rather than having generator for each peices like this PR.

We probably too overcomplicate things: generators are the last haven, for builtin functions it was made because the std tool like rustdoc was not enough. I feel we need to stick to rustdoc as much as possible, and this particular case should fit IMO.

For example refer to datafusion/core/src/lib.rs how it documents the Streaming Execution and here is the rendered version https://docs.rs/datafusion/latest/datafusion/#streaming-execution

For example refer to datafusion/core/src/lib.rs how it documents the Streaming Execution and here is the

Fair point, rustdoc with some sync guard would be better option. Thanks!

Here is a proposed followon to consolidate these docs into the main docs: #26899