Repository navigation
Conversation
schenksj
added a commit
that referenced
this pull request
Jun 22, 2026
schenksj
force-pushed
the
pr/delta-A7-docs
branch
2 times, most recently
from
June 22, 2026 01:57
51e7158 to
77f9032
Compare
schenksj
added a commit
that referenced
this pull request
Jun 22, 2026
…flaky re-run) + apache#4366 carveout links
schenksj
force-pushed
the
pr/delta-A6b-regression
branch
from
June 22, 2026 02:32
f0dcb24 to
c65b636
Compare
schenksj
force-pushed
the
pr/delta-A7-docs
branch
from
June 22, 2026 02:32
77f9032 to
59ff67c
Compare
schenksj
force-pushed
the
pr/delta-A6b-regression
branch
from
June 29, 2026 13:29
c65b636 to
6093760
Compare
schenksj
force-pushed
the
pr/delta-A7-docs
branch
from
June 29, 2026 13:29
59ff67c to
cf4f63a
Compare
schenksj
force-pushed
the
pr/delta-A6b-regression
branch
from
August 16, 2026 01:28
6093760 to
78b2999
Compare
schenksj
force-pushed
the
pr/delta-A7-docs
branch
from
August 16, 2026 01:28
cf4f63a to
2c8e3da
Compare
schenksj
force-pushed
the
pr/delta-A6b-regression
branch
from
August 18, 2026 15:16
78b2999 to
fff8e96
Compare
schenksj
force-pushed
the
pr/delta-A7-docs
branch
from
August 18, 2026 15:16
2c8e3da to
8e16e86
Compare
schenksj
force-pushed
the
pr/delta-A6b-regression
branch
from
August 18, 2026 18:09
fff8e96 to
b36ab22
Compare
schenksj
force-pushed
the
pr/delta-A7-docs
branch
from
August 18, 2026 18:09
8e16e86 to
548d9d2
Compare
schenksj
force-pushed
the
pr/delta-A6b-regression
branch
from
August 18, 2026 19:44
b36ab22 to
d15748f
Compare
schenksj
force-pushed
the
pr/delta-A7-docs
branch
from
August 18, 2026 19:44
548d9d2 to
ff178ea
Compare
schenksj
force-pushed
the
pr/delta-A6b-regression
branch
from
August 19, 2026 03:12
d15748f to
66bb165
Compare
schenksj
force-pushed
the
pr/delta-A7-docs
branch
from
August 19, 2026 03:12
ff178ea to
06de818
Compare
schenksj
force-pushed
the
pr/delta-A6b-regression
branch
from
August 19, 2026 10:59
66bb165 to
224909b
Compare
schenksj
force-pushed
the
pr/delta-A7-docs
branch
2 times, most recently
from
August 20, 2026 14:31
9f68442 to
ce403a4
Compare
schenksj
force-pushed
the
pr/delta-A6b-regression
branch
2 times, most recently
from
August 20, 2026 14:51
bf6d41e to
11860d4
Compare
schenksj
force-pushed
the
pr/delta-A7-docs
branch
2 times, most recently
from
August 20, 2026 16:58
0902791 to
f71afc3
Compare
schenksj
force-pushed
the
pr/delta-A6b-regression
branch
2 times, most recently
from
August 20, 2026 17:47
3d36966 to
fcf506d
Compare
schenksj
force-pushed
the
pr/delta-A7-docs
branch
2 times, most recently
from
August 20, 2026 20:01
8fd51b9 to
30beaa4
Compare
schenksj
force-pushed
the
pr/delta-A6b-regression
branch
from
August 20, 2026 20:01
fcf506d to
f0cce60
Compare
…rt 8] Adds the Delta contrib documentation: the user-facing guide plus the in-repo design docs. Docs only -- no code. User guide (docs/source/user-guide/latest/): - delta.md (new): how to build with -Pcontrib-delta + --features contrib-delta, the supported Spark/Delta/Scala matrix, usage, the four tuning configs (verified against DeltaConf.scala), supported features, and current limitations. - datasources.md, index.rst: link the new Delta guide into the data-sources page and the user-guide toctree (additive). Design docs (contrib/delta/docs/, 12 files): overview, planning, native execution, design decisions, build/deploy, fallback/ops, Spark 3.5 feasibility, known limitations, plus the iceberg-style kernel-read migration plan and its coherence/elimination audits. These are internal architecture/history docs linked from delta.md via GitHub URLs. Audited every config/class/path/proto reference and every user-facing claim against what actually landed (docs were authored for the integration branch, which this split reconstructs). Accuracy fixes: - delta.md storage: add Azure (abfs/abfss/wasb) and GCS (gs) -- both ship and work via object_store::parse_url (engine.rs); the line previously listed only local / HDFS / S3. - delta.md limitations: the residual S3-credential gap is explicit Hadoop credential-provider classes (AssumedRole/WebIdentity), NOT "per-bucket chains" (per-bucket static keys are handled); add the narrow far-future (~year 2262) INT96-timestamp overflow caveat (delta-kernel gap, A6). - delta.md: Java 17 is required for all Spark 4.x builds (4.0 and 4.1), not just 4.1; note Scala 2.12 is offered only for Spark 3.5. - delta.md usage: clarify the comet-spark jar must be the from-source -Pcontrib-delta build (the published Maven artifact carries no Delta support). - 12-elimination-evaluation.md: the proto kernel_read (field 25) row said "kept"; it is `reserved 25` and planner.rs no longer reads it -- corrected to "removed". - 05-build-and-deploy.md: `cargo build -p comet` -> `-p datafusion-comet`. The four tuning configs (hand-documented because contrib configs are not in the default GenerateDocs output), the version matrix, the native module list, the proto messages, and all inter-doc links verified accurate against HEAD. Docs 10/11/12 are explicitly framed as historical (doc 10 has a "Status: IMPLEMENTED and default" banner). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01BtErWgRQKCDRAg8Mk6qR4G Archived the three point-in-time engineering records (the iceberg-style kernel-read migration plan, its design-coherence audit, and the custom-code elimination evaluation) under contrib/delta/docs/archive/, leaving 01-08 + README as the living doc set. Incoming links (01/03/04/README) repointed to archive/; the README gains an "Archived design history" note. They document how the design was reasoned/pruned, not how the shipped integration works.
…[themeB, folded into A.7]
…markdown check] Co-Authored-By: Claude Fable 5 <[email protected]>
schenksj
force-pushed
the
pr/delta-A7-docs
branch
from
August 21, 2026 01:33
f433e7b to
728d5fc
Compare
schenksj
force-pushed
the
pr/delta-A6b-regression
branch
from
August 21, 2026 01:33
f0cce60 to
5fce6d8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 8 of the Delta Lake contrib PR breakup (stacked on part 7 / #11). Fork-local review draft. Docs only — no code.
What this adds
docs/source/user-guide/latest/delta.md(new) — the user-facing guide: build instructions (-Pcontrib-delta+--features contrib-delta), the supported Spark/Delta/Scala matrix, a usage example, the four tuning configs (verified againstDeltaConf.scala), supported features, and current limitations.datasources.md/index.rst— link the Delta guide into the data-sources page and the user-guide toctree (additive).contrib/delta/docs/(12 files) — internal architecture/history docs: overview, planning, native execution, design decisions, build/deploy, fallback/ops, Spark 3.5 feasibility, known limitations, and the iceberg-style kernel-read migration plan + its coherence/elimination audits.Audited against landed code (not just carried over)
The docs were authored for the integration branch (which this split reconstructs), so I cross-checked every config/class/path/proto reference and every user-facing claim against
HEAD. Accuracy fixes made:abfs/abfss/wasb) and GCS (gs) — both ship and work viaobject_store::parse_url(engine.rs); the line previously listed only local/HDFS/S3.fs.s3a.aws.credentials.provider, e.g. AssumedRole/WebIdentity) — not "per-bucket chains" (per-bucket static keys are handled).comet-sparkjar must be the from-source-Pcontrib-deltabuild (the published Maven artifact carries no Delta support).12-elimination-evaluation.md: the protokernel_read(field 25) row said "kept" — it isreserved 25andplanner.rsno longer reads it → corrected to "removed".05-build-and-deploy.md:cargo build -p comet→-p datafusion-comet.Verified accurate (no change needed)
The four tuning configs + defaults (hand-documented because contrib configs are not in the default
GenerateDocsoutput), the version matrix, the native module list, the proto messages, and all inter-doc links. Docs 10/11/12 are explicitly framed as historical (doc 10 carries a "Status: IMPLEMENTED and default" banner).🤖 This PR was prepared with the assistance of Claude (Anthropic). A human author reviewed and is responsible for the content.