Skip to content

Commit 29f1a41

Browse files
viiryaviirya
committed
Document the new ExecutionOptions field in the upgrade guide
`enable_nlj_coordinated_fallback` is a new public field on the exhaustively constructible `ExecutionOptions`, so downstream code building that struct with a literal needs a source change. Adds the migration note to the 56.0.0 upgrade guide, following the shape of the existing `constructible_struct_adds_field` entry in 55.0.0: who is affected, and how to migrate either by setting the field explicitly or by filling the rest from `Default`. Also records what distributed engines need to know, since the upgrade guide is where they are most likely to see it: the coordination assumes all probe partitions run in one process, so an engine that runs each partition as an independent task should set the flag to `false` and keep the previous fail-fast behaviour. Co-authored-by: Claude Code
1 parent e3ee90e commit 29f1a41

1 file changed

Lines changed: 51 additions & 0 deletions

File tree

  • docs/source/library-user-guide/upgrading

‎docs/source/library-user-guide/upgrading/56.0.0.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,57 @@
2525
in this section pertains to features and changes that have already been merged
2626
to the main branch and are awaiting release in this version.
2727

28+
### `ExecutionOptions` has a new `enable_nlj_coordinated_fallback` field
29+
30+
`ExecutionOptions` gained a public `enable_nlj_coordinated_fallback: bool` field
31+
(default `true`). It controls whether the memory-limited `NestedLoopJoinExec`
32+
fallback shares per-chunk build-side state across probe partitions, which is what
33+
lets `LEFT`, `LEFT SEMI`, `LEFT ANTI`, `LEFT MARK` and `FULL` joins spill instead
34+
of failing with `ResourcesExhausted` when the right side has several partitions.
35+
36+
**Who is affected:**
37+
38+
- Users constructing `ExecutionOptions` (or `ConfigOptions`) with an exhaustive
39+
struct literal. Reading or `SET`-ing configuration is unaffected.
40+
- Distributed engines that run each output partition as an independent task.
41+
The coordination assumes all probe partitions run in one process; with one
42+
coordinator per task the shared probe-thread counter never reaches zero and the
43+
fallback would stall. Such engines should set the flag to `false`, which keeps
44+
the previous fail-fast behaviour for the affected join types.
45+
46+
**Migration guide:**
47+
48+
Set the new field, or fill it from `Default`:
49+
50+
```rust,ignore
51+
// Before
52+
ExecutionOptions {
53+
batch_size: 8192,
54+
// ... every other field ...
55+
}
56+
57+
// After: set it explicitly
58+
ExecutionOptions {
59+
batch_size: 8192,
60+
enable_nlj_coordinated_fallback: true,
61+
// ... every other field ...
62+
}
63+
64+
// After: or let the remaining fields come from Default, which also keeps
65+
// future field additions from breaking the literal
66+
ExecutionOptions {
67+
batch_size: 8192,
68+
..Default::default()
69+
}
70+
```
71+
72+
Distributed engines opting out:
73+
74+
```rust,ignore
75+
let mut config = SessionConfig::new();
76+
config.options_mut().execution.enable_nlj_coordinated_fallback = false;
77+
```
78+
2879
### `datafusion.optimizer.use_statistics_registry` is deprecated and ignored
2980

3081
The `datafusion.optimizer.use_statistics_registry` config flag is deprecated and

0 commit comments

Comments
 (0)