Skip to content

docs: document DELTARS_* writer and vacuum environment variables - #4966

Merged
ion-elgreco merged 3 commits into
delta-io:mainfrom
Shahzeb-DataEngineer:docs/deltars-env-vars
Nov 8, 2026
Merged

ion-elgreco merged 3 commits into
delta-io:mainfrom
Shahzeb-DataEngineer:docs/deltars-env-vars

Conversation

@Shahzeb-DataEngineer

Copy link
Copy Markdown
Contributor

Description

Adds a section to the writing guide, next to the DELTARS_MAX_IN_FLIGHT_UPLOAD_BYTES one, documenting the other DELTARS_* variables that currently only exist in the source: what each bounds, its default, and that all but the upload-bytes cap are read once per process and cached (so setting one after the first write has no effect). It also spells out how the three concurrency knobs differ, since they bound different stages and do not substitute for each other.

Each default and behavior was verified against the current source (crates/core/src/operations/write/execution.rs, crates/core/src/datafile/writer/{partition,file,upload_budget}.rs, crates/core/src/operations/vacuum.rs).

Related Issue

Closes #4965

AI assistance disclosure

This change was prepared with AI assistance; the content was verified against the source files listed above before submission.

Adds a section to the writing guide next to the
DELTARS_MAX_IN_FLIGHT_UPLOAD_BYTES one, covering the other DELTARS_*
knobs: what each bounds, its default, and that all but the upload-bytes
cap are read once per process and cached. Defaults and behavior verified
against the current source.

Closes delta-io#4965

Signed-off-by: Shahzeb Ali <[email protected]>
Copilot AI balanced review requested due to automatic review settings October 27, 2026 04:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Oct 27, 2026
@ion-elgreco

Copy link
Copy Markdown
Collaborator

We might need a dedicate page to cover all the environment variables, aren't there more?

@Shahzeb-DataEngineer

Copy link
Copy Markdown
Contributor Author

Good question — I checked the source for every DELTARS_* variable, and these seven are the complete set: the six added here plus DELTARS_MAX_IN_FLIGHT_UPLOAD_BYTES, which already had its own section just above. So between the two sections, the writing guide now covers all of them. Happy to move them into a dedicated environment-variables page instead if you would prefer that, just say the word.

@ion-elgreco

Copy link
Copy Markdown
Collaborator

@Shahzeb-DataEngineer a separate page would fit better then nested somewhhere else

Move the writer concurrency table and the upload-bytes cap section
out of the writing guide into a new Environment variables page, and
link to it from the writing guide.

Signed-off-by: Shahzeb Ali <[email protected]>
@Shahzeb-DataEngineer

Copy link
Copy Markdown
Contributor Author

Done, moved them to a dedicated page in the latest commit. All seven DELTARS_* variables now live on a new Environment variables page under Usage (docs/usage/environment-variables.md), including the upload-bytes cap section that was already in the writing guide, and the writing guide now just links to it. Let me know if the placement looks right or if you would rather have the page elsewhere.

Copilot AI balanced review requested due to automatic review settings November 8, 2026 21:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@ion-elgreco

Copy link
Copy Markdown
Collaborator

@rtyler can you stamp it as well?

@ion-elgreco
ion-elgreco enabled auto-merge (squash) November 8, 2026 21:10
@codecov

codecov Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.35%. Comparing base (85df6d5) to head (94c3a49).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4966   +/-   ##
=======================================
  Coverage   82.35%   82.35%           
=======================================
  Files         210      210           
  Lines       78936    78936           
  Branches    78936    78936           
=======================================
  Hits        65011    65011           
  Misses      10193    10193           
  Partials     3732     3732           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ion-elgreco
ion-elgreco merged commit fee852e into delta-io:main Nov 8, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

[Feature]: Document the DELTARS_* environment variables the writer reads

4 participants