Repository navigation
docs: document DELTARS_* writer and vacuum environment variables - #4966
Conversation
Adds a section to the writing guide next to the DELTARS_MAX_IN_FLIGHT_UPLOAD_BYTES one, covering the other DELTARS_* knobs: what each bounds, its default, and that all but the upload-bytes cap are read once per process and cached. Defaults and behavior verified against the current source. Closes delta-io#4965 Signed-off-by: Shahzeb Ali <[email protected]>
|
We might need a dedicate page to cover all the environment variables, aren't there more? |
|
Good question — I checked the source for every |
|
@Shahzeb-DataEngineer a separate page would fit better then nested somewhhere else |
Move the writer concurrency table and the upload-bytes cap section out of the writing guide into a new Environment variables page, and link to it from the writing guide. Signed-off-by: Shahzeb Ali <[email protected]>
|
Done, moved them to a dedicated page in the latest commit. All seven |
Signed-off-by: Ion Koutsouris <[email protected]>
|
@rtyler can you stamp it as well? |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4966 +/- ##
=======================================
Coverage 82.35% 82.35%
=======================================
Files 210 210
Lines 78936 78936
Branches 78936 78936
=======================================
Hits 65011 65011
Misses 10193 10193
Partials 3732 3732 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Description
Adds a section to the writing guide, next to the
DELTARS_MAX_IN_FLIGHT_UPLOAD_BYTESone, documenting the otherDELTARS_*variables that currently only exist in the source: what each bounds, its default, and that all but the upload-bytes cap are read once per process and cached (so setting one after the first write has no effect). It also spells out how the three concurrency knobs differ, since they bound different stages and do not substitute for each other.Each default and behavior was verified against the current source (
crates/core/src/operations/write/execution.rs,crates/core/src/datafile/writer/{partition,file,upload_budget}.rs,crates/core/src/operations/vacuum.rs).Related Issue
Closes #4965
AI assistance disclosure
This change was prepared with AI assistance; the content was verified against the source files listed above before submission.