[{"content":"Introduction On the JVM, optimizing a hot kernel is not only about writing faster code. It also requires understanding the machine code HotSpot derives from the scalar implementation.\nUsing Parquet bit-unpacking as a concrete case, this experiment asks:\nWhen does an explicitly vectorized routine provide a real advantage over ordinary Java code optimized by HotSpot?\nA scalar Java loop is not a neutral baseline. HotSpot may recognize its structure and emit SIMD instructions, or leave it scalar when the access pattern is harder to prove. The measured speedup therefore depends not only on the vector implementation, but also on the scalar loop against which it is compared.\nThe results show when explicit vectorization is justified, how strongly its advantage depends on loop shape, and why a more specialized scalar routine is not necessarily faster.\nFrom bit-packed indices to integers Parquet dictionary encoding stores repeated values once and represents them in the data page through small integer indices. Those indices are bit-packed using only the number of bits the dictionary size requires: a dictionary of up to 8192 entries needs 13 bits per index. Reading the page therefore requires reconstructing an int[] of indices before the dictionary lookup can recover the original values.\nThis article isolates that reconstruction step:\nbit-packed dictionary indices \u2193 bit unpacking \u2193 int[] of dictionary indices To decode one value, a scalar decoder locates where it starts in the bit stream, loads the bytes around it, shifts away the preceding bits, applies a mask, and stores the resulting integer:\nlong bitPosition = 0; for (int i = 0; i < count; i++) { long bytePosition = bitPosition >>> 3; int bitOffset = (int) (bitPosition & 7); long word = source.get(LONG_LE, bytePosition); output[i] = (int) ((word >>> bitOffset) & mask); bitPosition += bitWidth; } The decoder processes one value at a time. At a width of 13 bits, the bit positions advance by a fixed increment, so the pattern is perfectly regular; but it is not a constant-stride, element-by-element array operation, and that is the distinction the compiler cares about. A more detailed introduction to Parquet dictionary encoding and bit unpacking is available separately, in From Dictionary Indices to Integers.\nHow Java reaches SIMD Scalar execution and SIMD A scalar loop processes one value after another:\nvalue 0 \u2192 shift \u2192 mask \u2192 store value 1 \u2192 shift \u2192 mask \u2192 store value 2 \u2192 shift \u2192 mask \u2192 store ... Modern processors can also apply one operation to several values at once. This is SIMD: Single Instruction, Multiple Data.\nFor example, on the machine used for this experiment, a 512-bit vector register can hold sixteen 32-bit integer lanes:\n\u250c\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2510 \u2502 v0 \u2502 v1 \u2502 v2 \u2502 v3 \u2502 .. \u2502v13 \u2502v14 \u2502v15 \u2502 \u2514\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2518 A vector shift can operate on all those lanes. The same applies to masking and other arithmetic operations.\nJava has two different ways to reach such instructions.\nTwo ways Java reaches vector instructions The first route is auto-vectorization.\nHotSpot contains an optimizing just-in-time compiler called C2. When a method becomes sufficiently hot, C2 compiles its bytecode into optimized machine code.\nOne of its optimization passes, called SuperWord, looks for isomorphic, independent scalar operations that can be packed into SIMD operations, provided their memory dependencies and access patterns can be proven safe.\nOrdinary scalar-looking Java code can therefore become vector machine code.\nFor example:\nfor (int i = 0; i < count; i++) { output[i] = a[i] + b[i]; } The programmer has written a scalar loop. Each source-level iteration adds a single pair of elements.\nC2 may nevertheless recognize that the iterations are independent and access memory in a regular stride, and emit machine code that processes several elements at once:\nordinary scalar Java loop \u2193 C2 and SuperWord analyze the loop \u2193 SIMD machine instructions, if recognition succeeds The second route is the Vector API.\nWith the Vector API, the programmer expresses the vector computation directly. On supported hardware, and for operations the target can map efficiently, HotSpot lowers it to the matching SIMD instructions; otherwise the API stays functional but may fall back to a less efficient form:\nVector API operations \u2193 HotSpot lowers the explicit vector computation \u2193 SIMD instructions when supported efficiently The Vector API did not introduce SIMD to Java. HotSpot had already been capable of generating vector instructions from suitable scalar loops for many years.\nThe difference is where the vector structure comes from. With auto-vectorization, C2 must infer that structure from scalar code. With the Vector API, the structure is explicit in the program.\nThis distinction leads to the central question:\nWhen can C2 derive efficient machine code from ordinary scalar Java, and when does the vector structure need to be expressed explicitly through the Vector API?\nWhy loop shape matters C2 does not vectorize every loop. It must be able to prove that the transformation is safe and recognize a pattern that it knows how to combine.\nA byte-aligned conversion is comparatively simple:\nfor (int i = 0; i < count; i++) { output[i] = source[i] & 0xFF; } Every iteration reads the next byte, applies the same mask, and writes the next integer:\nsource[0] \u2192 output[0] source[1] \u2192 output[1] source[2] \u2192 output[2] The memory accesses have a fixed stride. The iterations are independent. The operation is identical for every element.\nThis regular shape allows SuperWord to recognize independent parallel work and construct load and store packs. Whether those packs can be connected into a vectorized computation depends on the operations between them. Here that operation is an implicit unsigned widening, and on the examined JDK it lies outside the conversions SuperWord can currently vectorize, so the loop stays scalar.\nBit unpacking at 13 bits has a different shape.\nSuccessive values begin at changing positions:\nvalue 0 starts at bit 0 value 1 starts at bit 13 value 2 starts at bit 26 value 3 starts at bit 39 The scalar decoder calculates a byte position and bit offset for every value:\nlong bytePosition = bitPosition >>> 3; int bitOffset = (int) (bitPosition & 7); long word = source.get(LONG_LE, bytePosition); output[i] = (int) ((word >>> bitOffset) & mask); bitPosition += bitWidth; From the compiler’s perspective, the loop contains:\ninput positions derived from a changing bit position; shift distances that vary between iterations; values that may cross byte boundaries; relationships between neighboring bytes that are not expressed as direct element-wise operations. A programmer can see that several values could still be extracted in parallel. The compiler, however, would have to reconstruct that higher-level structure from the scalar operations.\nThe explicit Vector API kernel avoids that recognition problem by expressing the parallel structure directly.\nThe explicit vector kernel The vector implementation processes a block of packed values in several stages:\nload packed bytes \u2193 rearrange bytes into vector lanes \u2193 shift each lane by its required offset \u2193 apply the bit mask \u2193 store decoded integers In simplified Vector API form:\nByteVector.fromMemorySegment( BYTE_SPECIES, source, baseByte, LITTLE_ENDIAN) .rearrange(shuffle) .reinterpretAsInts() .lanewise(LSHR, shiftVector) .lanewise(AND, maskVector) .intoArray(output, outputOffset); Three values depend on the bit width:\nthe byte shuffle; the shift applied to each lane; the mask. They are computed for the selected width and reused.\nThe sequence of operations does not otherwise change for widths up to 25 bits:\nload \u2192 rearrange \u2192 shift \u2192 mask \u2192 store Across widths up to 25 bits, the parameters change, but the kernel still performs the same classes of operation.\nIts computational shape is therefore largely independent of the width.\nThat is the structural reason the vector cost stays nearly flat across widths, which the measurements confirm later.\nWhy the baseline matters The term “scalar baseline” describes source code, not necessarily the quality or even the execution shape of the machine code produced from it.\nTwo loops written as ordinary scalar Java may lead HotSpot to generate substantially different code. A regular loop may compile into a compact and efficient scalar form, or in some cases into SIMD instructions. A more irregular loop may retain address calculations, variable shifts, and other per-value work that the compiler cannot reorganize effectively.\nComparing the Vector API only with the latter can make explicit SIMD appear overwhelmingly superior. Comparing it with the strongest machine code HotSpot can derive from a relevant scalar form shows how much advantage remains against a genuinely competitive baseline.\nThe experiment therefore uses several scalar baselines. It does not assume that a compiler-friendly source loop is successfully auto-vectorized; the generated machine code is itself part of the evidence.\nPositioning the experiment Relation to existing work Vectorized integer unpacking is established work.\nLemire and Boytsov demonstrated that blocks of compressed integers can be decoded at very high throughput using explicit SIMD implementations. Their work helped establish a family of techniques based on fixed-width blocks, width-specific decoding strategies, vector rearrangement, shifts, and masks.\nPARQUET-2159 brought this line of optimization directly into Parquet Java. Its benchmarks compared the existing Parquet unpacking implementation with routines written using the Java Vector API. The resulting work became the basis of Parquet Java’s optional vector implementation.\nWhat this experiment examines differently This article does not introduce a new unpacking technique. Instead, it examines how the measured advantage of explicit vectorization changes with the scalar baseline against which it is evaluated.\nPrevious work has established that explicit SIMD unpacking can outperform scalar implementations. In this experiment, however, the baseline itself is treated as part of the result. The comparison includes regular scalar loops whose performance is materially affected by the machine code HotSpot can derive from them, as well as irregular loops for which SuperWord provides no measurable benefit. This makes it possible to distinguish how much of the observed SIMD advantage comes from the explicitly vectorized kernel and how much comes from the particular scalar loop used as its baseline.\nMachine-code and compiler-trace inspection add a further result: a regular scalar baseline can remain highly competitive without being auto-vectorized. For the byte-to-int widening loop, SuperWord constructs implementable load and store packs but rejects their def-use relationship because their vector element sizes differ and the implicit widening is not one of its recognized conversion idioms.\nThe experiment also evaluates Parquet Java’s specialized scalar routines separately from its vectorized path. The results show that scalar specialization does not necessarily produce a stronger baseline, particularly for wider and more irregular bit widths.\nExperimental setup Benchmark environment The measurements come from a controlled, single-threaded microbenchmark on one machine:\nAMD Ryzen 9 7950X3D Zen 4 x86-64 Linux, little-endian AVX-512 with VBMI\/VBMI2 OpenJDK 25.0.3 Vector API, tenth incubator 512-bit preferred vector species This configuration satisfies the platform and CPU-feature checks applied by ParquetReadRouter in the Parquet Java version examined here.\nScope and limitations The benchmark measures in-cache decoding kernels rather than the complete cost of reading a Parquet file.\nA real Parquet scan may also include:\nstorage access; page reads; decompression; the dictionary lookup (gather into the dictionary page); null handling; filtering; memory allocation; scheduling; downstream execution; memory-bandwidth contention. In a complete query, these costs may dominate bit unpacking.\nThe purpose of the microbenchmark is not to predict total query speed. It is to isolate the encoded-to-decoded boundary and make its computational shape measurable. The benchmark stops once it has produced the int[] of decoded indices; it does not include the subsequent dictionary lookup or the materialization of the original column values.\nThe results are also specific to one processor and one JDK.\nA different machine or compiler version could change:\nthe absolute throughput; the machine code C2 produces for the scalar baselines, including whether it auto-vectorizes them; the effectiveness of the specialized scalar routines; the magnitude or nature of the width-26 transition; the behavior of SuperWord on the widening loop; the cost of the vector kernel’s own operations, such as byte rearrangement and other cross-lane steps, whose lowering depends on the available instruction set. The role of MemorySegment Before comparing baselines, one possible confound must be removed. The experiment reads its input through a MemorySegment.\nA MemorySegment is a bounded region of memory, on the heap or off-heap, introduced by the Foreign Function & Memory API finalized in JDK 22.\nIt would be easy to assume that MemorySegment is responsible for a large part of the measured speedup.\nIt is not.\nIn the benchmark, a scalar decoder reading from a heap byte[] and an equivalent decoder reading from an off-heap MemorySegment produced almost identical throughput.\nThis result is unsurprising. The two implementations use the same extraction algorithm, perform the same bit arithmetic, and operate on data that fits in cache. Only the source of the load changes.\nMemorySegment is therefore not the performance result; its role is architectural. It supplies a bounded memory abstraction from which the Vector API can load directly, without requiring an intermediate Java array.\nIt is the substrate of the vector implementation, not its explanation.\nInput padding and boundary handling The decoders use fixed-width loads that may read beyond the last byte logically required for the values being decoded. The scalar decoders load one 64-bit word per value, while the main vector kernel loads a 64-byte ByteVector per block; values outside complete vector blocks are handled by a scalar fallback.\nTo keep these loads within the allocated memory region without adding an end-of-buffer branch to the measured kernels, the benchmark appends 64 zero-filled bytes to the encoded data. The padding is sized for the widest load: the vector block.\nAll decoders operate under the same padded-input contract. The measured advantage therefore cannot be attributed to a scalar- or vector-specific end-of-buffer path.\nMeasurement strategy Throughput is the primary performance result, but it can vary with processor frequency, core placement, and other run-time conditions. Hardware counters provide complementary evidence by showing whether a throughput difference is accompanied by a substantial difference in the dynamic work performed by the implementations.\nFor this reason, the analysis also considers retired instructions per decoded value: the number of retired machine instructions attributed to the measured kernel, divided by the number of values decoded. It is not a count of source-level Vector API operations.\nThis metric is especially useful when its pattern remains stable across runs and bit widths. It indicates how much dynamic instruction work an implementation executes for every decoded value.\nInstruction counts must not be interpreted as a direct prediction of throughput. Machine instructions differ in latency, throughput, and execution-resource requirements, and one vector instruction may perform a wider or more complex operation than one scalar instruction. Operations such as byte rearrangement and per-lane shifting may therefore retire relatively few instructions while still carrying a nontrivial execution cost.\nThe counters are consequently used as explanatory evidence rather than as a substitute for throughput measurements. Machine-code inspection identifies the instructions emitted by C2, while perfasm indicates where sampled cycles accumulate within the resulting hot loops.\nExperimental provenance The throughput measurements, hardware-counter data, and machine-code evidence described above were collected in separate runs.\nThe reported throughput measurements come from one primary benchmark run using:\n3 forks 5 \u00d7 3-second warmup iterations 10 \u00d7 1-second measurement iterations The processor governor was fixed to performance. The benchmark did not pin its forks to specific cores: thread placement was left to the operating-system scheduler, so on this multi-CCD processor individual measurements can include core-placement effects. A separate verification run with all forks pinned to a single CCD reproduced the reported ratios within normal run-to-run variability, so the comparisons are not an artifact of core placement.\nThe hardware counters, including retired instructions per decoded value, were collected separately on the same host using shorter perfnorm passes:\n1 fork 2 \u00d7 1-second warmup iterations 3 \u00d7 1-second measurement iterations The generated C2 machine code was inspected for the width-8 and width-16 widening loops with SuperWord enabled and disabled. Perfasm profiling was then used to attribute sampled cycles to instructions within the hot loops. This makes it possible to distinguish static code-shape differences from their dynamic cost. Cycle attribution remains approximate on out-of-order hardware.\nThe benchmark source, raw results, verification scripts, and extracted machine code are available in the accompanying repository. For all four widening configurations, the repository includes the decoder method across all compilation tiers, with the C2 hot loop identified. It also provides a script for building a JDK 25-compatible hsdis plugin and the exact command used to regenerate the complete disassembly.\nThe disassembly, SuperWord trace, and perfasm attribution come from separate follow-up runs on the same host and JDK. They explain the primary throughput measurements but are not part of the primary benchmark run.\nThe publication profile and verification report can be reproduced with:\n.\/run_lab.sh --publication Results The experiment uses three baselines to answer distinct but related questions. The first measures the advantage over a generic scalar decoder for which SuperWord provides no measurable benefit. The second gives the compiler a regular, byte-aligned loop and then inspects what HotSpot actually produces from it. The third compares the vector kernel with Parquet Java’s width-specialized scalar implementation.\nBaseline 1: a generic scalar decoder The first baseline is the generic scalar loop shown earlier.\nIt loads a word from a variable byte position, shifts it by a variable offset, applies a mask, and stores one decoded value.\nThe hardware counters show approximately:\n15.2 instructions per decoded value This value remains nearly constant across the tested widths.\nDisabling SuperWord does not materially change the measurements. On the tested system, the loop therefore serves as a baseline for which SuperWord provides no measurable benefit.\nAcross the reported irregular widths, the explicit vector kernel reaches approximately 4.7 times the throughput of this generic decoder.\nThat result is valid, but it answers a limited question:\nIs explicit SIMD faster than a scalar loop for which SuperWord provides no measurable benefit?\nThe answer is yes, but this comparison captures the case in which the generic scalar decoder retains substantial per-value work and receives no measurable benefit from SuperWord.\nThe comparison does not yet show how much advantage remains when the scalar code has a shape that the compiler can optimize more effectively.\nBaseline 2: a regular widening loop At a byte-aligned width of 8 bits, decoding can be expressed as a simple widening conversion:\nfor (int i = 0; i < count; i++) { output[i] = source[i] & 0xFF; } Its fixed-stride memory accesses and independent iterations make it a highly regular and competitive scalar baseline. It is not, however, a positive example of SuperWord auto-vectorization. The loop contains an implicit unsigned widening from byte-sized loads to integer-sized stores, a relationship that the SuperWord implementation in the examined JDK does not recognize as a supported conversion idiom.\nThe purpose of this baseline is therefore to measure how much advantage explicit vectorization retains against efficient scalar machine code derived from a regular decoding loop. A separate element-width-preserving addition loop is used later as a positive control to verify that SuperWord can successfully vectorize suitable code on the same host and JDK.\nWith the default compiler configuration, the loop processed approximately 5,961 values per microsecond. The explicit vector implementation was approximately 1.65 times faster, a meaningful gain, but substantially smaller than the 4.7-fold advantage measured against the generic scalar decoder.\nDisabling SuperWord produced an unexpected result: throughput increased to approximately 7,495 values per microsecond. Against this faster scalar result, the advantage of the explicit vector implementation fell to approximately 1.3 times.\nWhy enabling SuperWord makes the scalar loop slower Machine-code inspection explains the regression. Neither configuration emits SIMD code for the widening operation. With SuperWord disabled, C2 produces a compact loop unrolled eight ways, keeping each decoded value in a general-purpose register until it is stored. With SuperWord enabled, C2 produces a larger loop unrolled sixteen ways, with more simultaneously live scalar values. Several of them are routed through XMM registers as temporary storage and then moved back to general-purpose registers before being stored.\nSuperWord OFF (loop unrolled 8\u00d7): source[i] \u2192 GPR \u2192 output[i] SuperWord ON (loop unrolled 16\u00d7): source[i] \u2192 GPR \u2192 XMM \u2192 GPR \u2192 output[i] \\_____________\/ scalar round trip through XMM These transfers perform no SIMD computation. They add dynamic instruction work and are consistent with the lower throughput. At width 8, approximately 31% of the sampled cycles reported by perfasm within the SuperWord-enabled hot loop are associated with these scalar GPR\u2013XMM round trips, which are absent from the disabled configuration. They are therefore a substantial source of additional dynamic work in the slower loop.\nWhy the widening loop is not vectorized A SuperWord diagnostic trace shows why the widening operation remains scalar. C2 recognizes that the loop iterations are independent and constructs a 16-lane LoadUB pack and a 16-lane StoreI pack. Both packs are individually implementable on the target.\nThe problem is the connection between them:\nLoadUB: 16 \u00d7 1-byte elements \u2193 widening StoreI: 16 \u00d7 4-byte elements SuperWord normally requires a packed definition and its use to have compatible vector element sizes. It supports size-changing relationships only for a limited set of recognized widening idioms. Here, the widening is implicit in LoadUB rather than represented by one of those idioms, so the load and store packs cannot be connected into an accepted vector def-use chain.\nThis limitation is recognized upstream. The OpenJDK enhancement JDK-8375502, C2 SuperWord: implement unsigned casts, describes the same situation: SuperWord finds LoadUB and StoreI packs that are directly connected without an explicit cast, so the widening is implicit, and proposes inserting the missing cast during auto-vectorization. The enhancement remains open. On the JDK 25 examined here, the widening loop therefore remains scalar.\nBy contrast, an element-width-preserving loop such as:\nfor (int i = 0; i < count; i++) { output[i] = a[i] + b[i]; } has a homogeneous def-use chain:\nLoadI \u2192 AddI \u2192 StoreI 4 B 4 B 4 B SuperWord can connect these packs and vectorize the loop.\nA positive control rules out a general limitation of the host or the SuperWord pipeline: on the same machine and JDK, the addition loop above vectorizes while the byte-to-int widening loop does not:\nSuperWord stage homogeneous a[i] + b[i] widening src[i] & 0xFF recognizes parallel work yes yes builds load and store packs yes yes packs implementable on target yes yes def-use element sizes compatible yes (4 B = 4 B) no (1 B vs 4 B) outcome vectorized scalar This does not imply that SuperWord is generally harmful. For this loop, host, and JDK, however, enabling it produces the less efficient of two scalar forms.\nThe supported conclusion is:\nOn this host and JDK, neither of the two inspected scalar configurations emits SIMD for the widening loop. The default configuration therefore represents the normal HotSpot baseline, while the SuperWord-disabled configuration provides the fastest scalar compilation observed in this experiment and thus the more demanding comparison for the explicit vector kernel.\nBaseline 3: Parquet Java’s specialized scalar fallback Parquet Java provides scalar unpacking routines specialized for individual bit widths. These routines are part of the core parquet-encoding module and form the commonly available fallback when the separate optional vector module is absent or not selected.\nInstead of calculating the extraction position generically for every value, each routine encodes the required loads, shifts, masks, and combinations directly for one particular bit width.\nThis removes some runtime calculations, but it can also expand the decoding operation into a longer sequence of scalar instructions. At wider, irregular bit widths, a value may have to be assembled from several parts of the packed input, requiring additional loads, shifts, and bitwise combinations.\nSpecialization and vectorization are therefore separate optimizations. Specialization fixes the extraction pattern in advance; vectorization applies operations to several values in parallel. If C2 cannot reduce the expanded extraction sequence to sufficiently efficient machine code, the result may simply be a longer scalar instruction sequence than the generic decoder executes.\nThat is what the measurements indicate at several wide, irregular widths on the tested system. Parquet’s specialized scalar unpacker retires approximately 8 instructions per value at smaller widths, but up to approximately 23 at wider widths. Disabling SuperWord leaves the count essentially unchanged, providing no measurable evidence that SuperWord improves these routines on this system. These counts establish that the specialized routines perform substantially more dynamic work at the wider widths. Unlike the width-8 loop, this experiment does not attribute that cost at the disassembly level. The source-level structure is nevertheless consistent with the result: at these widths, the specialized extraction sequences contain additional loads, shifts, and combinations relative to the generic word-based decoder.\nAt widths 17 and 25, the specialized routines therefore retire more instructions and achieve lower throughput than the generic scalar decoder:\nwidth 17: Parquet specialized scalar: approximately 1,277 values\/\u00b5s generic scalar: approximately 1,992 values\/\u00b5s and:\nwidth 25: Parquet specialized scalar: approximately 1,004 values\/\u00b5s generic scalar: approximately 1,992 values\/\u00b5s Figure 1: throughput of the generic and specialized Parquet scalar decoders. At widths 17 and 25, the specialized implementation retires more instructions and falls below the generic loop.\nThe result is narrow. It does not mean that Parquet is generally slow, nor that per-width specialization is inherently harmful: the comparison concerns only Parquet Java’s scalar fallback, not its separate optional vector implementation (selected only on supported platform and processor configurations), and the effectiveness of specialization depends on the implementation, compiler, processor, and bit width.\nThe supported conclusion is:\nOn this host, Parquet’s per-width scalar specialization did not improve performance monotonically. At some wide, irregular widths, it was slower than the generic scalar loop.\nThe main performance result Across the tested widths up to 25 bits, including the irregular widths, the computational shape of the explicit vector kernel remains materially unchanged. It performs the same load \u2192 rearrange \u2192 shift \u2192 mask \u2192 store sequence, while only the shuffle, shift vector, and mask vary.\nThe measurements reflect this stability. In the counter measurements, the vector kernel retires between 1.566 and 1.570 instructions per decoded value across these widths. The scalar results are less uniform: the generic decoder remains nearly flat at a much higher cost, while Parquet’s specialized scalar cost increases with the width.\nFigure 2: instructions per decoded value. The vector kernel stays nearly flat through width 25. The generic decoder remains at approximately 15.2 instructions per value, while Parquet’s specialized scalar cost increases with the width.\nThe resulting speedups describe distinct compiler situations rather than a single continuous trend.\nAt width 8, the regular widening loop gives HotSpot a highly favorable scalar shape. Depending on whether the default or fastest observed scalar configuration is used, the explicit vector kernel is approximately 1.3 to 1.65 times faster.\nAt width 16, scalar reconstruction requires more work even though the input remains byte-aligned. The explicit vector kernel is approximately 2.4 to 2.65 times faster.\nAt irregular widths such as 13, 17, and 25, SuperWord provides no measurable benefit to the generic scalar decoder, and the explicit vector kernel reaches approximately 4.7 times its throughput.\nThe larger speedups at the irregular widths do not result from the vector kernel becoming faster. Its execution cost remains nearly constant through width 25; what changes is the amount and shape of the work that remains in the scalar implementations.\nFigure 3: throughput by compiler situation. The vector-to-scalar ratios differ because each comparison uses a scalar implementation with a different operation and machine-code shape.\nThe width-26 transition The stable vector result has a clear boundary.\nTo extract a value, the kernel needs a working window large enough to contain both the value and its possible intra-byte offset. In the worst case, this requires bitWidth + 7 bits. Up to width 25, the required window fits within a 32-bit lane:\n25 + 7 = 32 bits On the benchmark host, a 512-bit vector can therefore hold sixteen such lanes:\n512 bits \/ 32 bits = 16 lanes At width 26, the working window grows to 33 bits and no longer fits:\n26 + 7 = 33 bits The kernel consequently switches to 64-bit lanes. A 512-bit vector then holds only eight lanes:\n512 bits \/ 64 bits = 8 lanes This halves the number of values processed per vector block and requires the decoded values to be narrowed back to Java int values before storage.\nThe measured instruction cost rises from approximately 1.57 to 2.88 instructions per value, while throughput falls by a factor of approximately 1.4.\nThe two ratios are not expected to match. Wider lanes reduce lane-level parallelism and introduce a narrowing step, but these operations have different execution costs and may overlap within the processor. Retired-instruction count reveals the change in computational shape; it does not predict throughput proportionally.\nUnlike the width-dependent behavior of the scalar implementations, this transition follows directly from the vector kernel’s lane-width requirement. Its exact performance impact remains platform-dependent, because it depends on the preferred vector species and the available lowering path.\nConclusion The value of explicit vectorization cannot be separated from the scalar implementation used to evaluate it.\nThe reported speedups illustrate this directly. At the reported irregular widths, the explicit vector kernel reaches approximately 4.7 times the throughput of the generic scalar decoder. Against the regular width-8 widening loop, the advantage falls to approximately 1.3 to 1.65 times. These ratios answer different questions because they compare the same vector kernel with scalar implementations that have substantially different execution costs. The larger ratio remains a valid measure of improvement over the generic implementation, but it is not an intrinsic measure of the advantage of explicit SIMD.\nTwo methodological consequences follow. First, a vector implementation should be compared with the strongest relevant scalar baselines, including compact, regular formulations that the JIT may compile to efficient machine code. A regular scalar baseline can remain highly competitive even when its specific widening idiom is not supported by SuperWord. Second, scalar specialization and vectorization should be evaluated independently. Removing runtime calculations through width-specific code does not by itself guarantee fewer instructions, higher throughput, or parallel machine code.\nThe Vector API is most compelling when the scalar formulation leaves substantial work on the hot path and prevents HotSpot from deriving sufficiently efficient machine code, not merely when auto-vectorization fails. The explicit kernel examined here retains a stable computational shape across widths up to 25, while the cost of the scalar implementations depends strongly on their extraction logic and generated machine code.\nThe widening experiment also shows that enabling SuperWord is not evidence that SIMD instructions were emitted. Compiler configuration, generated machine code, retired-instruction counts, and sampled-cycle profiles describe different aspects of execution and must be interpreted together.\nParquet bit unpacking is the concrete case examined here, but the same methodology applies to JVM kernels used in parsing, compression, encoding, hashing, and numerical processing.\nReferences Daniel Lemire and Leonid Boytsov. “Decoding billions of integers per second through vectorization.” Software: Practice & Experience, 45(1), pp. 1\u201329, 2015. DOI: 10.1002\/spe.2203. Apache Parquet. “PARQUET-2159: Parquet bit-packing de\/encode optimization.” Apache Software Foundation issue tracker, https:\/\/issues.apache.org\/jira\/browse\/PARQUET-2159. The optional vector path ships as a separate parquet-encoding-vector module; the version examined here is parquet-java 1.16.0. OpenJDK. “JEP 338: Vector API (Incubator).” Initial Vector API incubation in JDK 16. https:\/\/openjdk.org\/jeps\/338 OpenJDK. “JEP 508: Vector API (Tenth Incubator).” Vector API incubation used in JDK 25. https:\/\/openjdk.org\/jeps\/508 OpenJDK. “JEP 454: Foreign Function & Memory API.” Finalized in JDK 22. https:\/\/openjdk.org\/jeps\/454 Daniel Lemire, Leonid Boytsov, and Nathan Kurz. “SIMD Compression and the Intersection of Sorted Integers.” Software: Practice & Experience, 46(6), pp. 723\u2013749, 2016. DOI: 10.1002\/spe.2326. Christian Del Monte. decode-shape: benchmark, measurement, and verification harness for the experiments in this article. https:\/\/github.com\/cdelmonte-zg\/decode-shape OpenJDK. “JDK-8375502: C2 SuperWord: implement unsigned casts.” HotSpot\/C2 enhancement, open and unresolved at the time of writing (reported 2026-01-16). https:\/\/bugs.openjdk.org\/browse\/JDK-8375502 ","permalink":"https:\/\/cdelmonte.dev\/deep-dives\/unpacking-parquet-explicit-simd-and-scalar-baselines\/","summary":"On the JVM, optimizing a hot kernel is not only about writing faster code: it is also about understanding how much the result depends on the machine code HotSpot derives from the scalar loop. Using Parquet bit-unpacking as a concrete case, the piece shows that a SIMD speedup depends on which scalar baseline C2 is handed, when explicit vectorization is actually justified, and why a more specialized scalar routine is not necessarily faster.","title":"Unpacking Parquet: Explicit SIMD, Scalar Baselines, and What HotSpot Makes of Them"},{"content":"The most consequential decisions in an execution engine are not always the ones that appear in the plan. Some live in the shape of the data.\nSpark evolved into a row-oriented execution engine deeply shaped by the JVM runtime, and that choice defined what Spark became. The analytical ecosystem is now reorganizing itself around something different, and the trace of that reorganization is visible in the physical plan of a typical Spark query that reads Parquet.\nRun df.explain on an ordinary Parquet scan, and you will often find one operator sitting just above the scan that has nothing to do with the query itself. It does not filter, join, aggregate, or sort. It is called ColumnarToRow, and its only job is to reorganize bytes from a columnar in-memory layout into a row stream. Why does an analytical engine that reads columnar data need this operator at all?\nThe data starts columnar A Parquet file is not a sequence of records waiting to be materialized as Java objects. It is a physical layout built around a specific bet: analytical workloads benefit when the engine can avoid reading and decoding irrelevant columns.\nValues are grouped by column, encoded by type (dictionary, run-length, delta, bit-packing), and compressed more effectively than row-oriented alternatives because adjacent values in a column tend to have low entropy relative to each other. A query that touches three columns out of forty reads three column chunks and skips the rest. Predicate pushdown can eliminate entire row groups before decoding begins. File-level and row-group statistics enable data skipping at the metadata layer.\nIn the dominant lakehouse path, this assumption is everywhere. The table is logical; the physical data is usually stored in columnar files, most often Parquet.\nSpark SQL was not designed as a purely columnar execution engine.\nN.B. — Not every scan shows the boundary This does not mean that every Spark query starts from Parquet, or even from a columnar source. Spark also reads CSV, JSON, Avro, JDBC, Kafka, and other inputs, and many of these paths do not expose a ColumnarToRow operator after the scan. That absence is not a counterexample. It simply means that the columnar-to-row boundary is not visible there.\nThe deeper point is that Spark’s classic SQL execution path still converges on InternalRow and, in its optimized runtime form, on UnsafeRow. ColumnarToRow is the visible trace of this commitment only when the input representation disagrees with the execution representation.\nSpark crosses into rows Spark’s internal execution model has a different center of gravity. The key abstraction is not the Parquet column chunk. It is the internal row, specifically InternalRow and its dominant runtime implementation, UnsafeRow.\nInternalRow is an abstract class that defines a uniform API for accessing fields by ordinal: getInt(i), getUTF8String(i), isNullAt(i). It says nothing about how the data is stored. UnsafeRow is the answer to that question.\nUnsafeRow does not use the JVM object model to represent data. A row is encoded as a contiguous region of bytes, a single byte[] on the JVM heap (or a raw memory address off-heap), read and written through low-level unsafe memory access with pointer arithmetic. Inside the row payload: no object headers, no boxed values, no pointer chasing between fields. The engine addresses the row as a compact binary region.\nTo make this concrete, consider a row from a simple query:\nSELECT user_id, name, age, score, city FROM users Schema: [INT, STRING, INT, DOUBLE, STRING], with city null.\nThe binary layout of the corresponding UnsafeRow looks like this:\n=== UnsafeRow Binary Layout === Logical payload: 57 bytes | Aligned UnsafeRow size: 64 bytes | Fields: 5 --- NULL BITSET --- [0000-0007] 0x0000000000000010 nulls: field[4] --- FIXED-LENGTH REGION (8 bytes\/field) --- field[0] [0008-0015] 0x000000000000002A \u2190 42 field[1] [0016-0023] 0x0000003000000009 \u2190 offset=48, len=9 field[2] [0024-0031] 0x0000000000000026 \u2190 38 field[3] [0032-0039] 0x4058600000000000 \u2190 97.5 (IEEE 754) field[4] [0040-0047] 0x0000000000000000 \u2190 NULL --- VARIABLE-LENGTH REGION --- [0048-0056] 43 68 72 69 73 74 69 61 6E ASCII: Christian [0057-0063] 00 00 00 00 00 00 00 \u2190 8-byte alignment padding The layout above is simplified for exposition, but it follows the structure Spark uses for UnsafeRow: null bitset, fixed-width slots, and variable-length data.\nThe layout has three regions. A null bitset where each bit maps to a field: reading nullability is a single getLong plus a bit mask. A fixed-length region with one 8-byte slot per field: primitives are stored inline, while variable-length types pack an offset and a byte length into a single 8-byte word, with the upper 4 bytes encoding the offset (measured from the start of the row) where that field’s variable-length bytes begin and the lower 4 bytes encoding the length. A variable-length region where strings live as UTF-8 byte sequences, while arrays, maps, and nested structs are embedded as Spark’s own unsafe binary structures.\nReading user_id means computing the slot address and loading the primitive value directly from the binary region. No allocation. No virtual dispatch. Pointer arithmetic dressed as Java.\nThat entire row occupies 64 bytes after alignment. On a typical 64-bit JVM with compressed ordinary object pointers, the same logical row represented as an Object[] with boxed primitives and String instances can easily occupy several times more memory across multiple heap objects. At scale, the difference is not only memory footprint. It is the number of objects the JVM has to track, scan, and eventually collect.\nThe exact byte layout matters less than what the layout enables. By moving execution data out of the ordinary JVM object graph and into a compact binary segment, Spark made it possible to relocate, sort, and ship rows without ever decoding them back into objects. UnsafeRow is still a row, but it behaves more like a memory segment than like a Java object.\nThe format is not confined to in-memory operator evaluation. UnsafeRow also underlies Spark’s serialized shuffle path, handled by UnsafeShuffleWriter: because rows are already contiguous, length-prefixed byte regions, the shuffle writer can treat them as relocatable byte buffers and sort them by manipulating pointers rather than decoding payloads. UnsafeRow is not just an executor-local row format. It is part of how data moves through the entire execution pipeline.\nThis is the representation Spark’s classic execution path is built around once the plan crosses the boundary from columnar batches.\nThe boundary has a name When Spark reads Parquet through its vectorized reader (ParquetFileFormat with spark.sql.parquet.enableVectorizedReader = true, which is the default), the data arrives as ColumnarBatch, a structure that holds column vectors, one per schema field. This is still columnar. The data has not yet changed shape.\nBut when the next operator in the physical plan expects rows, which most traditional Spark SQL operators do, the planner inserts ColumnarToRowExec. This operator converts the columnar batch into the row stream expected by the downstream row-based operator.\nYou can see it in a physical plan:\n== Physical Plan == *(1) Filter (isnotnull(amount) AND (amount > 100.0)) +- *(1) ColumnarToRow +- FileScan parquet [id, name, amount] ... ColumnarToRow sits between the scan and the first operator that touches the data. It appears only in the physical plan, inserted by ApplyColumnarRulesAndInsertTransitions, the physical planning rule that detects mismatches between operators producing columnar output and operators consuming rows.\nFrom the logical perspective, the query is: scan, filter, project. From the physical perspective: scan columnar, change representation, filter rows, project rows.\ncolumnar (at rest \u2192 in memory) row-oriented execution \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2554\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2557 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 Parquet \u2502\u2500\u2500\u25b6\u2502 ColumnarBatch \u2502\u2500\u2500\u25b6\u2551 ColumnarToRow \u2551\u2500\u2500\u25b6\u2502 UnsafeRow \u2502 \u2502 files \u2502 \u2502 column vectors\u2502 \u2551 boundary \u2551 \u2502 stream \u2192 ops\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u255a\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u255d \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 physical plan: FileScan parquet \u2192 *(1) ColumnarToRow \u2192 *(1) Filter \u2192 *(1) Project Figure 1: A lakehouse scan starts columnar and stays columnar through ColumnarBatch; classic Spark SQL crosses into row execution at ColumnarToRow, the one operator whose only job is to change the data’s shape.\nConceptually, the values have to be re-expressed from a columnar in-memory layout as a row stream compatible with Spark’s internal row representation. Nullability is mapped to row-level null bits, fixed-width values become row slots, variable-length data has to be made accessible through row-oriented offsets.\nDeep dive — The conversion cost The cost scales with the number of rows and the width of the schema. For narrow schemas and selective predicates it may be negligible. For wide schemas with many string columns scanned in full, it is not. In most workloads the conversion cost remains secondary to shuffle and I\/O, but in CPU-bound pipelines it becomes harder to ignore. On the write path, the picture is slightly different. The row-to-columnar encoding never appears as an operator in the plan. The write command consumes a stream of InternalRows (WriteFilesExec runs its child as an RDD[InternalRow]), and ParquetWriteSupport writes each row field by field into Parquet’s record consumer; the columnar on-disk encoding is performed downstream by the Parquet column writers, not by an in-memory row-to-columnar conversion.\nDeep dive — The write-side ColumnarToRow There is a transition in the other direction too, and it is asymmetric with the read path. ApplyColumnarRulesAndInsertTransitions special-cases the classic V1 file write (a plain df.write to Parquet or ORC) under planned writes: if the child is already columnar it leaves it columnar, instead of inserting a ColumnarToRow, because the write consumes the rows through executeWrite either way. The DataSource V2 write path has no such case, so a columnar input feeding a V2 write does get a ColumnarToRow. The round trip is real either way: columnar storage to row execution and back to columnar storage. The transition in the plan is the observable symptom. What produces it is what the rest of this article is about.\nWhy UnsafeRow made sense This shape change is sometimes read as a critique of Spark’s design, as if a more modern engine would not need it. It is more accurate to read UnsafeRow and the row-oriented execution model as a coherent answer to a specific set of constraints. Constraints that were binding when Spark’s execution engine was built, and that have shifted since.\nThe first constraint was the JVM itself. In the early 2010s, building a distributed analytical engine meant building it on the JVM. The ecosystem was JVM: Scala for productivity, Hadoop and Hive for storage and SQL, HDFS clients in Java, Kafka in Scala, YARN as the cluster substrate. Distributed coordination, network IO, fault tolerance, and ecosystem integration were the dominant engineering problems, and the JVM solved them well. Bytecode portability meant the same artifact ran across heterogeneous hardware. A mature concurrency model and rich serialization libraries shortened the path from research code to production system. The JVM was not chosen against alternatives. It was the substrate that made the system possible.\nBut the JVM imposes costs on data-intensive execution that are easy to underestimate until they dominate. Spark’s early execution engine allocated millions of small objects per query: one per field, one per row, one per intermediate result. Large heaps and object-heavy execution made garbage collection a first-order performance concern. Object headers, pointer indirection, and poor cache locality made the CPU spend more time chasing references than computing results.\nProject Tungsten was the response. It did not reject the JVM as the runtime. It pushed against the JVM’s own data model from inside it. UnsafeRow was the materialization of that idea: a format that moved data out of the JVM object model into binary buffers managed by Spark’s own memory manager. The engine took back control of allocation, layout, and access patterns while keeping the data on (or off) the JVM heap. The remarkable property of UnsafeRow is not that it is row-oriented. It is that it is almost not-JVM: manual memory management, pointer arithmetic, off-heap buffers, implemented in a language whose runtime was designed to make exactly these operations unnecessary.\nWhole-stage codegen extended the same logic. Instead of interpreting a tree of operator objects at runtime, where each operator called next() on its child, received an InternalRow, processed it, and passed it up, Spark began generating Java source code that fused an entire pipeline stage into a single function. Two distinct gains came from this. The first, often described, is that the generated function processes rows through a tight scalar loop in which the JIT can inline field access, eliminate bounds checks, and keep hot variables in CPU registers. The second, structurally larger, is what disappears: the per-row virtual dispatch through the operator tree, and the boxing of InternalRow at operator boundaries. Codegen turns a tree walk over polymorphic objects into a straight-line scalar function. The JIT is the consumer; eliminating polymorphism at the operator level is what makes the JIT’s job tractable.\nThe choice extended further than execution. UnsafeRow became central to Spark SQL’s shuffle path. Once a row was already a contiguous, length-prefixed byte region, the network layer could move it between executors without re-serializing the rows. The same representation that operators consumed in memory was the representation that flowed through the system between stages. A decision about how to lay out a row in memory had become a decision about how data moved across the cluster.\nEach layer reinforced the next, until after several years of development the JVM was no longer just the runtime under Spark. It had become the shape of the engine’s thinking.\nThe choice of row-oriented execution was not a limitation Spark failed to overcome. It was a trade-off that extracted a decade of production value from the JVM execution model.\nThat optimization had a cost, but it was not visible until the alternative became practicable.\nThe cascade The choice of representation sets the unit of work.\nA row engine pushes each row through a chain of operators; a columnar engine applies one operation across an entire column. The difference is not cosmetic. In a row-oriented engine, a filter that evaluates amount > 100.0 on a million rows executes a loop: load the row, extract the field at the right offset, compare, branch, move on. The JVM JIT compiles this into tight scalar code. It is fast, but it is fundamentally one-value-at-a-time.\nIn a columnar engine, the same filter operates on a contiguous array of doubles. The engine can process the array in batches using SIMD instructions, comparing multiple values with a single instruction. For fixed-width primitive columns there is no row object to decode and no per-row field offset to compute; nullability is handled through a separate bitmap. The values are already adjacent in memory, the CPU prefetcher works efficiently, and cache lines are fully utilized.\nFrom this single choice of representation, the rest of the engine follows as a cascade.\nChoosing row at the representation layer makes row-at-a-time operators natural. Row-at-a-time operators make scalar codegen natural. Scalar codegen makes the JVM JIT a natural compiler target. The JVM as compiler target makes manual memory management like UnsafeRow defensible as the way out.\nChoosing columnar at the representation layer makes vectorized operators natural. Vectorized operators make native execution natural. Native execution gives the developer the SIMD intrinsics, the memory layout control, and the cache-line discipline that columnar processing rewards.\nEach layer constrains the next. Representation is not an implementation detail. It is the constraint that is logically prior to every other choice in the execution model. The shift between row-oriented and columnar execution is not, at its core, about which language hosts the kernel. It is about how the engine organizes its data plane, and everything else cascades from there.\nsame logical query \u2571 \u2572 \u25bc \u25bc \u2500\u2500 ROW PATH \u2500\u2500 \u2500\u2500 COLUMNAR \/ NATIVE PATH \u2500\u2500 Row representation Columnar representation \u2502 \u2502 \u25bc \u25bc Row-at-a-time operators Vectorized operators \u2502 \u2502 \u25bc \u25bc Whole-stage codegen Native kernels (C++ \/ Rust) \u2502 \u2502 \u25bc \u25bc JVM JIT as compiler target SIMD + cache locality \u2502 \u2502 \u25bc \u25bc Off-heap manual memory Arrow \/ Velox \/ Photon Representation is not an implementation detail. It is the constraint that precedes every other choice. Figure 2: The representation choice cascades through the stack. Rows pull scalar operators, whole-stage codegen, the JVM JIT, and manual memory behind them; columnar pulls vectorized operators, native kernels, and SIMD.\nSpark chose one path coherently. The analytical execution ecosystem is now organizing itself around the other.\nMoving the boundary Several projects now attempt to keep data columnar through execution rather than converting it to rows.\nDatabricks Photon moves supported parts of SQL and DataFrame execution into a native vectorized engine written in C++ that processes data in columnar batches. At the API layer, nothing changes: the same SQL, the same DataFrame interface, the same Delta tables. Operationally, the picture is more nuanced. Photon is proprietary to Databricks, carries a DBU multiplier, supports a set of operators that changes between releases, and falls back to Spark’s JVM execution for operators it does not cover.\nGluten and Velox approach the same problem from outside Databricks. Gluten is a Spark plugin that intercepts the physical plan, translates it into Substrait (a cross-engine intermediate representation for query plans), and delegates execution to a native backend, typically Velox, Meta’s C++ execution library. The data stays in columnar batches through JNI calls to native code and returns as ColumnarBatch through Spark’s columnar execution API. Velox uses its own columnar vector model, similar to Apache Arrow and interoperable with it but not identical to Arrow’s in-memory layout for every type. Arrow itself has emerged as the lingua franca for columnar in-memory data interchange across analytical engines.\nAcross these systems the same dynamic applies. For operators the native path covers, the row transition disappears from the hot path. For operators it does not, control returns to Spark’s JVM execution and the ColumnarToRow boundary reappears at the fallback edge.\nWhat these projects share is not a preference for C++ over Java. It is a specific architectural choice: align the in-memory representation with the storage representation, so that the boundary between columnar at rest and columnar in execution disappears for the operators where compute density is high.\nThe portability has relocated The response these systems share is larger than any of them individually. Read together, Photon, Gluten, and Velox describe a single architectural movement that goes beyond Spark.\nThe JVM was the right portability layer for the problem Spark was originally solving. It provided a common runtime across heterogeneous machines, integrated naturally with Hadoop, Hive, Kafka, and the broader Scala\/Java ecosystem, and made distributed analytics operationally portable before the lakehouse stack existed.\nThe bottleneck mix has shifted. Distributing tasks across nodes is still hard, shuffle still dominates many workloads, skew has not been eliminated. What has changed is that, for an increasing share of analytical compute, the inner loop matters as much as the coordination layer. SIMD width, cache line discipline, vectorized kernel specialization, zero-copy interchange with GPU memory and with other analytical engines: these are the layers where the modern native ecosystems have accumulated the deepest tooling and library coverage. The Vector API, introduced by JEP 338 and refined through many incubation rounds, gives the JVM a credible path into the same space, but it remains an incubating API, and the practical center of gravity for new analytical engine development has settled in C++ and Rust.\nThis helps explain why systems such as DuckDB, Velox, and ClickHouse are implemented in C++, why DataFusion and Polars are implemented in Rust, and why Photon follows the same native-engine direction. The convergence reflects both a technical fit and a set of historical and ecosystem choices: mature columnar libraries, embeddable deployment models, and direct integration with Python and the broader scientific computing stack are easier to access in those ecosystems today. The shift is not nostalgia for compiled languages. It is a practical reorganization toward the stack where new analytical workloads are most directly served.\nBut portability does not disappear. It relocates.\nThe old portability layer was the runtime. The new portability layer is no longer a single layer at all: it is distributed across table metadata, storage formats, in-memory representations, and query-plan representations.\nThe portability that used to live in JVM bytecode now lives, in stratified form, across distinct layers of the stack. Apache Arrow provides portability of the in-memory columnar representation, enabling zero-copy interchange between engines that agree on how vectors are laid out in RAM. Apache Parquet (and ORC) provide portability of the on-disk format, so the same files are readable by Spark, Trino, DuckDB, ClickHouse, and Polars. Substrait provides portability of the query plan, a cross-engine representation of the operators and functions a backend supports, allowing a planner in one engine to delegate execution to a backend in another. Delta Lake, Iceberg, and Hudi provide portability of the table across engines (snapshots, schema evolution, transactional metadata exposed independently of the execution engine that touches them), each within its own format.\nBEFORE: one center of portability \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 JVM bytecode \u2502 \u2502 the common runtime \u2502 \u2502 across machines \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 Spark execution \u00b7 Scala\/Java ecosystem \u00b7 Hadoop\/Hive\/Kafka \u2551 relocation \u25bc AFTER: portability stratified across layers Table semantics \u2192 Delta \u00b7 Iceberg \u00b7 Hudi Storage format \u2192 Parquet \u00b7 ORC In-memory format \u2192 Apache Arrow Query plan \u2192 Substrait (supported fragments) \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Execution \u2192 Photon \u00b7 Velox \u00b7 DataFusion (specializes) Figure 3: Portability relocates. The four layers above the line carry the portability the JVM once held alone; the planner, catalog, API, and governance stay stable, and the execution engine underneath is free to specialize.\nEach of these layers carries part of the portability burden that JVM bytecode once carried more centrally. The result is a system in which the planner, the catalog, the API surface, and the governance layer can remain stable, while the execution engine can be specialized: compiled for a specific hardware target, written in a language that exposes SIMD and memory layout, optimized for the workload’s actual bottleneck.\nThis is the division Spark is increasingly being adapted to, from inside through Photon and from outside through Gluten and Velox. The strongest evidence that the portability layer has moved is that even Spark, the system most associated with the old layer, is increasingly surrounded by mechanisms that move execution elsewhere.\nConclusions ColumnarToRow in a Spark physical plan is the visible trace of a system that lives on the boundary between the two layers. It is the point where a query designed inside the new portability model (columnar at rest, with a logical plan that in principle any engine could execute) meets an execution engine built coherently for the old one. The transition is not a flaw. It is the seam where two architectural eras meet.\nThe next time ColumnarToRow appears in a physical plan, read it as a record of where Spark stands in the longer movement of analytical systems toward a different locus of portability.\n","permalink":"https:\/\/cdelmonte.dev\/essays\/where-spark-changes-shape\/","summary":"Run df.explain on a Parquet scan and you find ColumnarToRow, an operator whose only job is to change the data’s shape. It is the seam where two architectural eras meet, and a record of how portability in analytical systems has relocated from the JVM runtime to the data format and the query plan.","title":"Where Spark Changes Shape: UnsafeRow, the JVM, and the Relocation of Portability"},{"content":"Spark exposes its own compiler as a surface for intervention\nA Language Without a Name Inside Catalyst1 there is a language for manipulating logical trees. It has a grammar, a composition mechanism, and execution semantics.\nIt is, in fact, an embedded DSL, but it isn’t documented as such and doesn’t have a name. Yet it’s the mechanism by which all of Spark’s analysis and optimization rules are written, from name resolution all the way to predicate pushdown. And it’s the same mechanism that Delta Lake, Iceberg, and other external projects use to extend Catalyst’s pipeline.\nThis article looks at Catalyst from that angle: not only as an optimizer, but as a programmable rewriting system.\nThe DSL Catalyst’s pipeline is a sequence of rule batches, managed by RuleExecutor. Each batch has an execution strategy: Once or FixedPoint(n). A Once batch applies its rules a single time; a FixedPoint batch applies them all repeatedly until the plan stops changing or the maximum number of iterations is reached. The “stops changing” criterion is structural (fastEquals on the tree), not semantic: it’s a syntactic fixed point of the composed transformation function, not a guarantee of normal form, nor of confluence. What must stabilize is the composition of the batch’s rules, not each individual rule: per-rule idempotence is recommended engineering discipline (it prevents oscillation and rules producing structurally different but equivalent plans), not a formal API requirement.\nThe demo below applies a batch of rewrite rules to a small logical-plan tree until it stabilizes:\nFixedPoint batch \u2014 step through the algorithm Ready Current plan   Rules in batch (FixedPoint(100)) Iteration 0 Step 0 Reset Step \u2192 Auto The demo uses a simplified top-down traversal. The point isn’t the traversal order itself, but the fixed-point execution of a batch of tree rewrites: what stabilizes is the composition of the rules, not any single rule on its own.\nTo give a concrete idea, two typical batches, reduced to their main rules:2\nBatch "Operator Optimization before Inferring Filters" (FixedPoint) \u251c\u2500\u2500 PredicatePushdown \u251c\u2500\u2500 ConstantFolding \u251c\u2500\u2500 ColumnPruning \u2514\u2500\u2500 SimplifyConditionals Batch "Finish Analysis" (FixedPoint(1)) \u251c\u2500\u2500 EliminateSubqueryAliases \u2514\u2500\u2500 ReplaceExpressions Each element of a batch is a Rule[LogicalPlan]: a function LogicalPlan => LogicalPlan that receives a tree, rewrites it, and returns it. It’s the unit of composition of the DSL.\nInside the Rule, the actual work is done by transform, a method of TreeNode[T], the abstract class from which all logical plans in Spark derive. In current Spark transform delegates to transformDown (preorder, parent before children), but the API explicitly warns rule authors not to rely on transform for directionality: when traversal order matters, use transformDown or transformUp directly. The rule applies a partial function where the pattern matches, and returns the transformed node. Where it doesn’t match, the node is left unchanged. The Rule[LogicalPlan] signature makes the transformation an endomorphism over LogicalPlan: it preserves Catalyst’s syntactic universe. Semantic invariants (resolution, types, exprId coherence) remain the rule author’s responsibility, not a framework guarantee.\nIn the following example the pattern captures a SubqueryAlias (the node that marks where a named subquery begins) and replaces it with the child subtree. After name resolution the alias is no longer needed, and this rule removes it from the tree. It’s one of the rules in the Finish Analysis batch shown above:\nobject EliminateSubqueryAliases extends Rule[LogicalPlan] { def apply(plan: LogicalPlan): LogicalPlan = plan transform { case SubqueryAlias(_, child) => child } } This is the language: case classes for the nodes, partial functions for the pattern matching, Rule for composition, RuleExecutor for execution. It isn’t complex, but it’s the computational structure on which the entire relational layer of Spark is built.3\nHow Spark uses the DSL internally Spark uses this DSL uniformly across the logical-planning pipeline. Analyzer and Optimizer do different things, but they share the same machinery: both are built on RuleExecutor[LogicalPlan] and run batches of Rule[LogicalPlan].\nLogicalPlan (unresolved) \u2502 \u25bc Analyzer = RuleExecutor \u251c\u2500\u2500 Batch "Substitution" (FixedPoint) \u251c\u2500\u2500 Batch "Hints" (FixedPoint) \u2514\u2500\u2500 Batch "Resolution" (FixedPoint) \u2502 \u25bc LogicalPlan (analyzed) \u2502 \u25bc Optimizer = RuleExecutor \u251c\u2500\u2500 Batch "Finish Analysis" (FixedPoint(1)) \u251c\u2500\u2500 Batch "Operator Optimization" (FixedPoint) \u2514\u2500\u2500 Batch "Infer Filters" (Once) \u2502 \u25bc LogicalPlan (optimized) \u2502 \u25bc [physical planning, codegen \u2014 out of scope] The operator-optimization phase contains rules such as predicate pushdown, column pruning, constant folding, boolean simplification, and filter combination.\nThe Analyzer performs name resolution, type coercion\/checking, and attribute binding:\nUnresolvedAttribute("age") \u2192 AttributeReference("age", IntegerType, exprId=42) This transformation introduces exprId, a stable identity for resolved attributes. It disambiguates homonymous references produced by self-joins and aliasing. The analogy isn’t a classical symbol table name \u2192 declaration; it’s closer to the compiler technique of giving bound variables an internal identity, as GHC does with Unique on Var.\nWhat matters here is less the what than the how: the transformation happens through rules over logical-plan trees. The same machinery that resolves UnresolvedAttribute is also used later by the optimization rules listed above.\nThe Optimizer applies these optimization rules. They too are expressed as Rule[LogicalPlan]:4\nobject PredicatePushdown extends Rule[LogicalPlan] { def apply(plan: LogicalPlan): LogicalPlan = plan transform { case Filter(condition, Join(left, right, joinType, joinCond, hint)) if canPushToLeft(condition, left) => Join(Filter(condition, left), right, joinType, joinCond, hint) } } Rule-based vs cost-based Readers familiar with optimizer architecture can skip this section without losing the main thread.\nThis choice (rewriting by rules, rather than searching over a space of equivalent plans) isn’t a given. The main alternative, in relational optimizer architecture, is memo-based cost optimization5: these optimizers build a space of equivalent plans and use a cost model to choose a low-cost implementation. In that world, a rule is not merely an improving transformation. It can also express an equivalence, allowing the optimizer to explore combinations that no single rule would know how to produce directly.\nThis is more powerful in principle. In practice, it requires high-quality statistics, a memo structure to avoid combinatorial explosion, and a significantly more complex architecture. The central structural difference from Catalyst is the memo. Memo-based optimizers enumerate equivalent expressions in a shared structure and use cost to choose among them. Catalyst has no such memo:6 it applies directive transformations directly to the current tree.\nCatalyst’s rule-based rewriting largely gives up global exploration in exchange for simplicity: a fixed sequence of local transformations, no general enumeration of alternatives, no global cost function selecting among them. The operational advantage is that every rule is readable and testable in isolation, and the set is extendable from outside without rewriting the optimizer: exactly the property Delta, Iceberg, and similar projects need. The disadvantage is fragility with respect to ordering: adding a rule can change the output plan in unexpected ways if it interacts with those that follow, and plan quality depends on what the active rules can recognize, not on the global optimum.\nSpark’s cost-based join reorder is itself implemented as a Rule[LogicalPlan]: the mechanism remains rule-based; the difference is that the rule consumes statistics. The practical consequence is that performance in Spark often depends on what the optimizer can “see” structurally in the tree, not on the exploration of alternative plans. For common cases this matters little: Catalyst has a good set of normalization rules (CombineFilters merges adjacent filters, CollapseProject fuses consecutive projections, BooleanSimplification rewrites logical expressions) that make superficially different forms converge on the same optimized plan.\nThe difference emerges in complex plans: multi-way joins when cost-based join reordering is disabled or statistics are missing; physical-strategy selection in JoinSelection, which selects the first applicable physical strategy according to its priority order rather than keeping several alternatives alive for a later global cost decision; rule interactions blocked by correlated subqueries, CTEs, or UDFs that prevent local rewriting. The last is probably the most frequent case in practice. In all these cases two logically equivalent DataFrame transformation sequences can produce different physical plans: not because the optimizer is wrong, but because the shape of the input tree activates different rules.\nA concrete example on Spark 4.1.1 (AQE and CBO disabled to isolate the rule-based pipeline): three datasets, A large, B very small (broadcastable), C medium (above the broadcast threshold). Two logically equivalent orderings:\nval plan1 = A.join(B, "k").join(C, "k") \/\/ (A \u22c8 B) \u22c8 C val plan2 = A.join(C, "k").join(B, "k") \/\/ (A \u22c8 C) \u22c8 B Same final result (495 million rows in both cases). But not the same physical plan:\nForm 1: Form 2: SortMergeJoin BroadcastHashJoin \u251c\u2500\u2500 BroadcastHashJoin(A, B) \u251c\u2500\u2500 SortMergeJoin(A, C) \u2502 \u251c\u2500\u2500 A \u2502 \u251c\u2500\u2500 Exchange + Sort(A) \u2502 \u2514\u2500\u2500 BroadcastExchange(B) \u2502 \u2514\u2500\u2500 Exchange + Sort(C) \u2514\u2500\u2500 Exchange + Sort(C) \u2514\u2500\u2500 BroadcastExchange(B) Form 1 broadcasts B first (a local operation, no shuffle of A) and pays the shuffle only for the sort-merge with C. Form 2 pays the full shuffle of A upfront for the sort-merge with C, and only afterwards applies the cheap broadcast of B. Same result, very different runtime cost. Spark doesn’t reorder because (without CBO and statistics) the JoinSelection strategy is applied to the two Join nodes in the order they appear in the tree: the first Join is mapped to the best of broadcast\/sort-merge, and the second is evaluated on the output of the first, with its broadcast\/sort-merge decision depending on the available size estimates for the resulting subtree. The alternatives (for instance: moving the broadcast of B to the top of the tree) aren’t even generated.\nThe examples\/join_order_demo.scala script reproduces this comparison on Spark 4.1.1; the full output of the two explain calls shows every BroadcastExchange, Exchange, and Sort at the points predicted by the two orderings.\nThe example shows the rule-based pipeline’s sensitivity to logical tree shape; it doesn’t claim Spark can’t choose good joins in general: AQE can convert SortMergeJoin to BroadcastHashJoin at runtime, and explicit hints plus high-quality statistics with CBO enabled cover other cases. The point is isolated: with AQE and CBO disabled, what remains is the rule-based pipeline.\nHow external projects use the DSL Spark exposes an API, SparkSessionExtensions,7 that allows external projects to inject custom rules into the Catalyst pipeline. It is a documented Developer API (marked @Experimental\/@Unstable in Spark’s ScalaDoc), but it is the official extension mechanism Spark exposes for injecting parser, analyzer, optimizer, and planner customizations. The extension surface covers most of the planning pipeline, from parsing and analysis to optimization and physical planning:\ninjectParser: extend or replace the SQL parser injectResolutionRule: add rules to the Analyzer injectPostHocResolutionRule: post-analysis resolution rules injectCheckRule: plan validation after analysis injectOptimizerRule: add rules to the Optimizer injectPlannerStrategy: physical-planning strategies In current Spark versions, injected optimizer rules are integrated into the optimizer’s rule batches, including the operator-optimization phase. The exact placement is an implementation detail rather than a semantic contract: extensions should not depend on fine-grained neighboring order between external and native rules across Spark versions.\nDelta Lake and Iceberg both use this mechanism, with different emphases: Delta intervenes mainly in parsing and analysis, Iceberg mainly in optimization, but there’s overlap: Iceberg also injects resolution rules, and Delta also injects optimizer rules.\nDelta Lake injects a parser, analyzer rules, and check rules. It extends the analysis and validation portion of the pipeline:\n\/\/ From DeltaSparkSessionExtension (simplified) override def apply(extensions: SparkSessionExtensions): Unit = { extensions.injectParser { (session, parser) => new DeltaSqlParser(parser) } extensions.injectResolutionRule { session => new DeltaAnalysis(session) } extensions.injectCheckRule { session => new DeltaUnsupportedOperationsCheck(session) } extensions.injectPostHocResolutionRule { session => new PreprocessTableMerge(session.sessionState.conf) } } Iceberg injects rules into the optimizer. It extends the optimization portion:\n\/\/ From IcebergSparkSessionExtensions (simplified) override def apply(extensions: SparkSessionExtensions): Unit = { extensions.injectParser { case (_, parser) => new IcebergSparkSqlExtensionsParser(parser) } extensions.injectResolutionRule { spark => ResolveProcedures(spark) } extensions.injectOptimizerRule { spark => RewriteDelete(spark) } extensions.injectOptimizerRule { _ => OptimizeConditionsInRowLevelOperations } } The point of these snippets8 isn’t which rules they inject: it’s that external rules operate in the same pipeline, on the same types, with the same mechanism as native rules. A Rule[LogicalPlan] injected by Iceberg is structurally indistinguishable from PredicatePushdown.\nA Static Analyzer on Catalyst’s IR If projects like Delta Lake and Iceberg extend Catalyst for their own operations, the same mechanism is available to anyone. A concrete use case: an extension that intercepts dangerous patterns in the plan and signals, or blocks, them before execution.\nA caveat: the code that follows is a sketch, not a production-ready library. The complete version lives at github.com\/cdelmonte-zg\/planguard.9\nFor an extension that inspects the plan and can reject queries, without transforming it, the correct hook is injectCheckRule. Unlike transformational rules, which are endomorphisms LogicalPlan => LogicalPlan, a check rule is a validator LogicalPlan => Unit: it inspects, it may throw, but it does not rewrite. It is invoked in the analysis check phase (not in the optimizer’s FixedPoint batches), so it isn’t multiplied by iterations the way an optimizer rule would be. It’s the documented point for analysis-level errors.\nclass PlanGuardExtension extends (SparkSessionExtensions => Unit) { \/\/ Entry point: Spark instantiates this class based on \/\/ spark.sql.extensions and calls apply once per SparkSession. override def apply(extensions: SparkSessionExtensions): Unit = { extensions.injectCheckRule { session => DetectDangerousPatterns(session) } } } case class DetectDangerousPatterns(session: SparkSession) extends (LogicalPlan => Unit) with Logging { override def apply(plan: LogicalPlan): Unit = { plan.foreach { \/\/ 1. Inner join without a condition: the classic forgotten ON. \/\/ df.crossJoin(other) is explicit user intent and is NOT \/\/ intercepted (joinType = Cross, not Inner). case Join(_, _, Inner, None, _) => throw new SparkException( "Join without condition detected: possible cartesian product" ) \/\/ 2. Join with a condition but no equi-join key: the planner \/\/ will fall back to BroadcastNestedLoopJoin \/ CartesianProduct. \/\/ ExtractEquiJoinKeys is the same helper Catalyst uses to \/\/ decide between hash and nested-loop. case j @ Join(_, _, _, Some(_), _) if !isEquiJoin(j) => logWarning( "Non-equi join detected: will execute as nested-loop join" ) \/\/ 3. Python UDF inside a Filter: an opaque node to Catalyst, \/\/ blocks predicate pushdown and any rewrite on the predicate. case Filter(condition, _) if containsPythonUdf(condition) => logWarning( "Python UDF in filter predicate: pushdown and optimization blocked" ) \/\/ 4. collect_list \/ collect_set \/ collect_top_k in an aggregation: \/\/ dangerous memory profile, non-spillable in-memory buffer. \/\/ Note: in Spark 4.x Aggregate has 4 fields (groupingExpressions, \/\/ aggregateExpressions, child, hint: Option[AggregateHint]); \/\/ in 3.x there are 3. case Aggregate(_, aggregations, _, _) if aggregations.exists(containsCollect) => logWarning( "collect_* in aggregation: may create large in-memory buffers" ) case _ => } \/\/ inspection-only: no return value, no plan transformation. } private def isEquiJoin(j: Join): Boolean = ExtractEquiJoinKeys.unapply(j).exists { case (_, leftKeys, _, _, _, _, _, _) => leftKeys.nonEmpty } private def containsPythonUdf(expr: Expression): Boolean = expr.exists { case _: PythonUDF => true; case _ => false } \/\/ Collect[_] is the abstract parent class of CollectList, CollectSet, \/\/ and CollectTopK (all in org.apache.spark.sql.catalyst.expressions.aggregate). \/\/ CollectList\/CollectSet are unbounded per group; CollectTopK is bounded by k \/\/ but can still be dangerous with large k or many groups. private def containsCollect(expr: Expression): Boolean = expr.exists { case AggregateExpression(_: Collect[_], _, _, _, _) => true case _ => false } } Registration:\nspark.sql.extensions=dev.cdelmonte.planguard.PlanGuardExtension Deployment doesn’t require modifying Spark: the extension is compiled into a JAR, included in the cluster’s classpath or passed via --jars at submit time, and activated with spark.sql.extensions=.... It’s the same procedure used, among others, by io.delta.sql.DeltaSparkSessionExtension in the delta-spark JAR.\nspark.sql.extensions is additive: multiple values are comma-separated, for example io.delta.sql.DeltaSparkSessionExtension,dev.cdelmonte.planguard.PlanGuardExtension. In clusters with multiple extensions the value must be concatenated, not overwritten.\nThe same hook works “on the fly” from a programmatically constructed SparkSession, for example in a notebook or in PySpark:\nspark = ( SparkSession.builder .config("spark.sql.extensions", "dev.cdelmonte.planguard.PlanGuardExtension") .getOrCreate() ) An operational detail worth noting: the JAR must be on the classpath when the JVM starts, not when the SparkSession is built. Concretely, it must be passed with --jars to spark-submit, or, from “pure” Python, via PYSPARK_SUBMIT_ARGS set before the pyspark import:\nimport os os.environ["PYSPARK_SUBMIT_ARGS"] = ( "--jars \/abs\/path\/to\/extension.jar pyspark-shell" ) from pyspark.sql import SparkSession # only from here on This holds for any SparkSessionExtensions: it’s a property of the JVM bootstrap, not specific to this extension. In managed environments (Databricks attached cluster, EMR, Glue, Dataproc) the driver JVM is already running by the time getOrCreate is called: the classpath must be injected from the cluster configuration (init scripts, bootstrap actions, properties), not from code.\nWhat the rule intercepts, and why at the logical-plan level The cartesian join is intercepted at the logical-plan level, before the physical-strategy choice. AQE may demote a SortMergeJoin to SHJ\/BHJ based on shuffle statistics, but it doesn’t convert a cartesian into something non-cartesian: the problem is structural, not resolvable at runtime. The rule distinguishes the intentional case (explicit crossJoin, joinType = Cross) from the accidental one (Inner with empty condition, the forgotten ON): only the latter is blocked.\nThe non-equi join is the cartesian’s less conspicuous cousin. The condition exists, syntactically it looks like a normal join, but no equality key can become a hash key: Catalyst cannot extract hash keys from it, so planning falls back to nested-loop or cartesian-like strategies such as BroadcastNestedLoopJoin or CartesianProductExec. The rule doesn’t block it (there are legitimate cases: range joins, semi-joins with < to select top-N); it simply makes it visible. For the check it uses the same ExtractEquiJoinKeys the planner consults to decide the physical strategy.\nThe Python UDF in a filter is an opaque node that blocks predicate pushdown. Catalyst can’t inspect its contents, so no rewrite rule can cross it. The rule doesn’t solve the problem: it makes it visible before the query runs on a cluster with real data.\ncollect_list in an aggregation isn’t a logical error: it’s an operational risk. No native Catalyst rule flags it. A traditional optimizer doesn’t worry about it; in a framework for distributed dataflow, the operational risk belongs precisely to the level where the plan is still inspectable.\nPlanGuard is a static analyzer over Catalyst’s IR (a linter that enforces local operational invariants before execution), not an optimizer: it doesn’t pick a better plan; it rejects or flags plans that embody dangerous patterns.\nExtending Observability The same architectural position (a rule running at plan-time before execution) also works as a signal-emission point, not just a blocking one. Instead of just logging warnings or throwing exceptions, the Rule can emit OpenTelemetry spans or events, with metadata about the detected pattern, the plan involved, the problematic predicate. Cartesian joins and UDFs in filters are not only intercepted at plan-time: they’re tracked as events in the cluster’s monitoring system, through any OTel-compatible backend. Catalyst’s pipeline becomes an emission point for diagnostic signals, not just for plan transformation.\nThe hook is mechanical. You initialize a Tracer once. The OTLP exporter reads endpoint and service name from standard environment variables (OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_SERVICE_NAME):\n\/\/ lazy val: the SDK starts on first detection, not at class-load. \/\/ Standard OTel env vars decide where and how to export. private lazy val tracer: Tracer = { val provider = SdkTracerProvider.builder() .addSpanProcessor( BatchSpanProcessor.builder(OtlpGrpcSpanExporter.builder().build()).build()) .build() \/\/ Without a shutdown hook BatchSpanProcessor loses queued spans \/\/ when the JVM terminates: the worst case for blocking events. Runtime.getRuntime.addShutdownHook(new Thread(() => provider.close())) OpenTelemetrySdk.builder().setTracerProvider(provider).build() .getTracer("dev.cdelmonte.planguard") } Each detection becomes a span with attributes10 about the pattern and the node. The span is opened and closed in-place inside the case branch: the plan isn’t touched, only observed. It’s a zero-duration span: used as an event (a counter source via spanmetrics), not as a unit of time. The canonical OTel path for a pure counter is the Meter API; the span-as-event is a conscious alternative, because a single emission yields both the per-query detail and the aggregated metric without duplicating the source. On Tempo or Jaeger it will appear disconnected from the Spark query trace, which is exactly what we want.\ncase Filter(condition, _) if containsPythonUdf(condition) => tracer.spanBuilder("catalyst.plan_guard.detect") .setAllAttributes(Attributes.of( AttributeKey.stringKey("planguard.pattern"), "python_udf_in_filter", AttributeKey.stringKey("planguard.node"), "Filter")) .startSpan().end() logWarning("Python UDF in filter predicate: pushdown blocked") From here on the pipeline is the standard OTel one: the span travels in OTLP gRPC to the collector, can be exported to any backend, and (through the spanmetrics connector) is automatically converted into a Prometheus counter. A single emission produces both the detail for diagnosis (the individual query, the plan, the attributes) and the aggregated metric for alerting. Catalyst’s DSL doesn’t just rewrite the plan: it’s also the point at which the plan becomes visible.\nConclusions Catalyst isn’t just Spark’s internal optimizer. It’s a framework with an embedded DSL for manipulating logical trees, and that DSL is one of the system’s main programmable surfaces at the query-planning level: the one in which the dataflow is still represented as a transformable structure, before it becomes physical code.\nLogical-plan analysis and optimization are built on a single mechanism: rules that pattern-match on tree nodes and rewrite them. The same mechanism is exposed externally through SparkSessionExtensions, and it’s the same mechanism by which Delta Lake extends the analyzer and Iceberg rewrites operations in the optimizer. Anyone can inspect, transform, or reject a dataflow before it is executed.\nMost Spark users will never see this DSL. But it is the level at which Spark analyzes, optimizes, and prepares every query for physical planning. The actual code generation to JVM bytecode happens downstream, in WholeStageCodeGen. Once you’ve seen this level, Spark no longer appears as just an engine that runs distributed transformations, but as a system that exposes its own compiler as a surface for intervention.\nThe operational pitfalls one encounters writing real rules are the subject of the article Anti-patterns in Catalyst rules, the operational counterpart to this one.\nNotes Armbrust et al., Spark SQL: Relational Data Processing in Spark, SIGMOD 2015: the foundational Catalyst paper, with the architectural motivation for rule-based rewriting. Source code lives in org.apache.spark.sql.catalyst.{trees, rules, plans}: TreeNode, Rule, RuleExecutor, LogicalPlan and the cited rules (EliminateSubqueryAliases, PushDownPredicates, referred to as PredicatePushdown in the text, BooleanSimplification). ↩︎\nA note in the spirit of philological honesty: in the Spark source Finish Analysis is a batch with a single meta-rule (FinishAnalysis in Optimizer.scala) that internally foldLeft-applies a set of sub-rules, including EliminateSubqueryAliases and ReplaceExpressions. The diagram above is didactic. ↩︎\nFor those familiar with functional compilers: this structure (iterative rewriting over term trees until stabilization) resembles the way GHC organizes its simplifier passes over Core (Santos & Peyton Jones, “A Transformation-Based Optimiser for Haskell”, Science of Computer Programming 32, 1998). Catalyst splits the rewriting across two ADTs (LogicalPlan for relational nodes, Expression for predicates and functions), while GHC operates on a single Core ADT. Same mechanism, different domain. ↩︎\nThe snippet is intentionally simplified. The real rule (PushDownPredicates) splits conjunctive predicates (Filter(a AND b) is broken apart and each conjunct is pushed independently), and also accounts for join type, aliases, determinism, and many other constraints. ↩︎\nMemo-based optimizers populate a shared structure with equivalent expressions and use cost estimation to select the cheapest physical implementation. The canonical pedigree is the Volcano Optimizer Generator (Graefe & McKenna, ICDE 1993) and its successor Cascades (Graefe, 1995), alongside IBM’s Starburst (Lohman, Pirahesh et al., late ’80s), with evolutions in SQL Server (Cascades-derived) and Postgres (System-R \/ dynamic-programming lineage). ↩︎\n“Memo” here means a Cascades-style shared structure of equivalent sub-plans during search. Spark’s CacheManager is a separate mechanism that caches query results (data) keyed by structural plan equivalence; it isn’t a memo of equivalent sub-plans available for cost-based selection during planning. ↩︎\nSparkSessionExtensions ScalaDoc: the documented Developer API for the extension point. ↩︎\nThe snippets above are simplified and version-dependent: the exact set of injected rules changes across Delta and Iceberg releases. ↩︎\nThe operational pitfalls (placement, idempotence, FixedPoint multiplication, PySpark bootstrap) are covered in Anti-patterns in Catalyst rules, the operational counterpart to this article. ↩︎\nAttributes must stay low-cardinality. Embedding the full plan, raw predicates, or a per-query query_id will blow up the cardinality of spanmetrics-derived metrics; a plan fingerprint (a hash) is safer. ↩︎\n","permalink":"https:\/\/cdelmonte.dev\/deep-dives\/the-hidden-dsl-in-catalyst\/","summary":"How Spark’s internal rewriting framework, Catalyst, exposes an embedded DSL with a public extension surface, the same one Delta Lake and Iceberg use to plug into the optimizer pipeline.","title":"The Hidden DSL in Catalyst"},{"content":"Updated July 6, 2026\nThe problem For scan-heavy analytical workloads, the dominant cost lever is whether files can be eliminated from the scan set before they are opened. This depends on how the layout interacts with the predicate. Partition pruning and data skipping can eliminate unnecessary file reads before any engine touches the data, using metadata from the transaction log. In library terms: partition pruning tells you which branch of the library to visit; data skipping tells you which shelves cannot possibly contain the book.\nThree things break this: partitions that do not match the predicate, statistics that are absent, and data mixed enough that min\/max ranges cannot rule anything out. None of them throws an error. There is no warning. The query runs, returns the right answer, and you quietly scan more and pay more.\nWith small datasets the problem is easy to ignore. Volumes grow, and the same misalignment that costs nothing on 20 files compounds at 20,000. At that point you need to understand what’s happening, and the options are limited. You can investigate by reading the execution plan or the engine’s profile, but that path is both engine-specific and execution-bound: it requires running the query against a particular engine. There is no widely available, engine-independent tool that attributes pruning from Delta log metadata alone.\nI wrote delta-explain to fill this gap.\nWhat delta-explain is for delta-explain is a metadata-only diagnostic for Delta Lake file pruning. Given a table snapshot and a WHERE predicate, it shows which files can be eliminated by partition pruning and data skipping before any query engine reads the data.\nUse it when you need to make pruning behavior visible, measurable, or enforceable.\nBefore a layout change. If you are about to repartition a large table, change its clustering, or rewrite it into a staging copy, run delta-explain with the production predicates against both layouts. The report shows, in file counts, whether the new layout actually improves pruning before the migration touches production.\nAfter a rewrite or suspected regression. If a dashboard got slower after last week’s table rewrite, --at-version lets you run the same analysis against an earlier version of the table:\ndelta-explain s3:\/\/warehouse\/events -w "date = '2026-07-01'" --at-version 41 # before the rewrite delta-explain s3:\/\/warehouse\/events -w "date = '2026-07-01'" # after Two reports, same predicate, one variable: the layout. A suspicion becomes a measured difference.\nWhen a predicate does not prune. delta-explain splits the predicate into partition-safe, stats-safe, and unsplittable fragments, then labels the run as exact, conservative, or incomplete. Supported fragments are lowered into the pruning analysis; unsupported constructs degrade with diagnostics instead of failing the run. With --verbose, the report lists every kept and dropped file, including partition values and min\/max statistics, so you can see which files survived and why. And when the counts say that a predicate did not prune but not why, --explain-why names the reason from the same metadata the report already read: whether the filter misses the partition columns, the statistics are absent, or the min\/max ranges are simply too wide to exclude anything. It is a deterministic reading of the report, not a prediction, so it is as trustworthy as the numbers it explains.\nWhen writer behavior changes. A pipeline upgrade or configuration change can silently stop emitting useful per-file statistics. Nothing fails, but future data skipping degrades. A post-ETL --assert-stats run turns that silent regression into an exit code the same day it happens.\nWhen pruning is part of the contract. For known tables and known predicates, --min-pruning turns an expectation like “this query should never scan more than a tenth of the table” into a CI gate: an exit code for shell pipelines, or a machine-readable JSON report, formally specified and versioned, whose assertions carry stable names a calling system can branch on.\nWhen you inherit a table you do not fully know. Features like deletion vectors, column mapping, and liquid clustering change what the numbers mean. delta-explain detects them and declares them in a table_features block with warnings, rather than reporting figures that assume every table has the same semantics.\nGetting started There are three ways to use delta-explain: from the shell, from Python, and as a gate in CI. All three run the same binary and produce the same report.\nFrom the shell delta-explain is a single binary. Rust users get it straight from crates.io with cargo install delta-explain; everyone else picks a channel: Homebrew, Scoop, a Debian package, prebuilt binaries on GitHub Releases, or the Docker image on GHCR. The documentation site collects the install commands.\nJust one required argument (the table, a local path or an object-store URI) and a predicate are needed:\ndelta-explain .\/users -w "country = 'DE' AND age > 40" The report it prints is the subject of the worked examples below.\nFrom Python pip install delta-explain ships the same CLI binary inside the wheel and adds a thin wrapper that runs it and returns the JSON report as a Python object:\nimport delta_explain report = delta_explain.explain(".\/users", where="country = 'DE' AND age > 40", min_pruning=60, assert_stats=True) report.total_files # 6 report.final_files # 1 report.total_pruning_pct # 83.33... report.passed # True: both gates passed report["analysis"]["confidence"] # "conservative" The table and predicate are the same as in the worked example below, which also explains where that 83% comes from. The wrapper is deliberately not a second API: it shells out to the same binary and hands back the same versioned JSON document, so everything the next subsection says about assertions applies unchanged. The two assertion gates (min_pruning, assert_stats) travel as keyword arguments, report.passed collapses them into one line of a pytest suite or an Airflow task, and the object is also dict-like: every field of the document (assertions, analysis, statistics coverage, table features) is one subscript away.\nIn CI The same analysis can run as a contract, and nothing has to live in your repository: delta-explain is on the GitHub Marketplace as a composite action that downloads a pinned release binary, so the gate is one step, with pruning-pct, final-files, and result exposed as outputs for later steps:\n- uses: cdelmonte-zg\/delta-explain@v0.6.0 with: table: s3:\/\/warehouse\/events where: "date = '2026-07-01'" min-pruning: "90" assert-stats: "true" env-creds: "true" On any other CI system, install the binary in the job (pip install delta-explain is the quickest path, a prebuilt wheel) or run the Docker image, and use the flags directly.\n--min-pruning enforces a floor on the total pruning percentage:\ndelta-explain s3:\/\/warehouse\/events \\ --env-creds \\ -w "date = '${PARTITION_DATE}'" \\ --min-pruning 90 --env-creds reads cloud credentials from the standard environment variables (AWS_*, AZURE_*, GOOGLE_*). The command exits 1 if total pruning is below 90%, with stderr:\nASSERTION FAILED: total pruning 33.3% is below threshold 90.0% --assert-stats is the second gate. It fails (exit code 1) if any file in the snapshot is missing per-file statistics in the transaction log. The intended use is a “writer broken” alarm: if your ETL stops emitting stats, future pruning will degrade, and you want to catch this before it reaches production. Long-lived tables do not produce false alarms: statistics are resolved through the kernel’s checkpoint-aware log replay, so a file is flagged only when its add action genuinely carries none (the semantics reference documents the mechanism).\n--format json produces structured output for downstream parsing. Flags can be combined:\ndelta-explain s3:\/\/warehouse\/events \\ --env-creds \\ -w "date = '${PARTITION_DATE}'" \\ --min-pruning 90 \\ --assert-stats \\ --format json Choosing the threshold, and what the JSON document guarantees to whatever consumes it, have their own section after the worked examples.\ndelta-explain in practice An example Consider a Delta table partitioned by country, with 6 files distributed across 3 partitions (DE, IT, US). Each file has min\/max statistics on the age column. The table is intentionally tiny, so every surviving file can be inspected on screen. The percentages that follow are not a benchmark; they expose the mechanism that decides whether a production scan touches tens of files or thousands.\nWe want to understand what would happen if an engine ran a query with:\nWHERE country = 'DE' AND age > 40 Specifically: how many files remain in the candidate scan set, and how much pruning occurred. Without running a query through an engine, delta-explain can answer this:\ndelta-explain .\/users -w "country = 'DE' AND age > 40" The output:\nDelta table: .\/users Version: 5 Predicate: country = 'DE' AND age > 40 Predicate Analysis: partition-safe: country = 'DE' stats-safe: age > 40 unsplittable: - confidence: conservative Files in snapshot: 6 Phase 1: Partition pruning [exact] predicate: country = 'DE' files remaining: 2 (-4, 67% pruned) Phase 2: Data skipping (min\/max statistics) [conservative] predicate: age > 40 files remaining: 1 (-1, 50% pruned) Total reduction: 6 -> 1 files (83% pruned) The Predicate Analysis block names the split: country = 'DE' is assigned to partition pruning, while age > 40 is assigned to data skipping. The run is reported as conservative overall because Phase 2 uses file-level statistics; partition pruning alone would have been exact.\nThe phases are attribution measurements, not a query plan. delta-explain is not saying that Spark, Trino, or any other engine must execute the scan in this order. It is asking a narrower metadata question: over this Delta snapshot, how many files can be eliminated by partition values, and how many more can be eliminated by per-file statistics?\nPartition pruning is exact in this model: a file’s partition values either satisfy country = 'DE' or they do not. Data skipping is conservative: min\/max statistics can prove that a file cannot match, but they cannot prove that every surviving file actually contains a matching row. Take age = 50 against a file whose statistics say age: 40..60: the range overlaps, so the file is kept, yet it may not contain a single row where age is exactly 50. A file is kept whenever its range still overlaps the predicate.\nThat is the main safety rule behind the report: delta-explain may keep too many files, but it must not drop a file that could contain a matching row.\nSix files reduce to one, 83% pruned before any Parquet data file is opened.\nWith that model in mind, --verbose shows the same attribution file by file:\ndelta-explain .\/users -w "country = 'DE' AND age > 40" --verbose Phase 1: Partition pruning [exact] predicate: country = 'DE' files remaining: 2 (-4, 67% pruned) [DROPPED] part-00000-48368dae.parquet (1.1 KB 3 records) partition(country=IT) stats(age: 41..65) [DROPPED] part-00000-fcf95aac.parquet (1.1 KB 5 records) partition(country=IT) stats(age: 22..38) [DROPPED] part-00000-eee5a3ec.parquet (1.1 KB 3 records) partition(country=US) stats(age: 31..55) [DROPPED] part-00000-de2ffaef.parquet (1.1 KB 4 records) partition(country=US) stats(age: 18..29) [KEPT ] part-00000-a35083c1.parquet (1.1 KB 4 records) partition(country=DE) stats(age: 40..60) [KEPT ] part-00000-c34f1417.parquet (1.1 KB 5 records) partition(country=DE) stats(age: 20..35) Phase 2: Data skipping (min\/max statistics) [conservative] predicate: age > 40 files remaining: 1 (-1, 50% pruned) [KEPT ] part-00000-a35083c1.parquet (1.1 KB 4 records) partition(country=DE) stats(age: 40..60) [DROPPED] part-00000-c34f1417.parquet (1.1 KB 5 records) partition(country=DE) stats(age: 20..35) (In the verbose excerpts throughout this article, file names and per-file statistics are abbreviated for legibility; the real output shows full file names and the statistics of every indexed column.)\nThe verbose output makes the conservative property concrete. In Phase 2, the file with age: 20..35 is eliminated because its maximum (35) doesn’t satisfy age > 40. No row in that file can match. The remaining file has age: 40..60: its maximum (60) exceeds 40, so the file cannot be ruled out. The lower bound plays no role in the exclusion test here; the maximum alone is enough to show the file may contain values greater than 40. Data skipping is conservative: it keeps any file whose range overlaps the predicate, even if most of its rows ultimately fall below the threshold.\nThe interactive diagram below steps through the same attribution model.\nComparative scans on .\/users Partitioned Flat Full scan \u00b7 done Files (6 total, partitioned by country) DE file 1 age: 40..60 DE file 2 age: 20..35 IT file 3 age: 41..65 IT file 4 age: 22..38 US file 5 age: 31..55 US file 6 age: 18..29 Predicate Analysis partition-safecountry = 'DE' stats-safeage > 40 unsplittable- confidenceconservative Predicate applied country = 'DE' AND age > 40 Files in snapshot6 Phase 1 (partition pruning)6 \u2192 2 (-4, 67% pruned) Phase 2 (data skipping)2 \u2192 1 (-1, 50% pruned) Surviving1 \/ 6 Step 3 \/ 3 Reset Step \u2192 Auto A counter-example The same data, the same predicate. Only the layout changes: files now contain records from multiple countries mixed together, with no partitioning at all.\ndelta-explain .\/users-flat -w "country = 'DE' AND age > 40" Delta table: .\/users-flat Version: 0 Predicate: country = 'DE' AND age > 40 Predicate Analysis: partition-safe: - stats-safe: country = 'DE' AND age > 40 unsplittable: - confidence: conservative Files in snapshot: 6 Phase 1: Data skipping (min\/max statistics) [conservative] predicate: country = 'DE' AND age > 40 files remaining: 4 (-2, 33% pruned) There is no partition pruning: country is not a partition column. The entire predicate falls into stats-safe; where the relevant statistics are present, as they are here, it is evaluated by data skipping alone, which decides through file-level statistics. For equality and range predicates like these, that mainly means min\/max.\nOn string columns, min\/max is still well-defined: lexicographic order is compatible with equality and range predicates. The issue is physical interleaving: when values are mixed across files, every file’s country range spans most of the alphabet, so country = 'DE' cannot be excluded from any of them. Per-file min\/max collapses the mixed values into a wide range, and that range is too broad to exclude the file. In library terms: when every shelf label spans most of the alphabet, no shelf can be ruled out.\nThe detail with --verbose:\ndelta-explain .\/users-flat -w "country = 'DE' AND age > 40" --verbose Phase 1: Data skipping (min\/max statistics) [conservative] predicate: country = 'DE' AND age > 40 files remaining: 4 (-2, 33% pruned) [DROPPED] part-00001.snappy.parquet (1.1 KB 4 records) stats(age: 18..29, country: AL..US) [DROPPED] part-00002.snappy.parquet (1.1 KB 5 records) stats(age: 20..35, country: AT..US) [KEPT ] part-00003.snappy.parquet (1.1 KB 4 records) stats(age: 30..55, country: DE..US) [KEPT ] part-00004.snappy.parquet (1.1 KB 3 records) stats(age: 22..50, country: AT..IT) [KEPT ] part-00005.snappy.parquet (1.1 KB 5 records) stats(age: 40..65, country: CH..US) [KEPT ] part-00006.snappy.parquet (1.1 KB 3 records) stats(age: 25..45, country: AT..DE) In the verbose output, country prunes nothing: every displayed country range still includes DE under lexicographic min\/max reasoning. File 6 (country: AT..DE, age: 25..45) is the cleanest illustration: even though DE happens to coincide with the upper bound of its country range, the age range satisfies age > 40 and the file stays.\nThe overall result: pruning drops from 83% to 33%. On 6 files the difference is negligible, but at production scale the gap between 17% and 67% retention (four times as many candidate files) becomes a metadata and scheduling problem in its own right: each surviving file may imply object-store requests, footer reads, task scheduling, and reader setup, even before the cost of decoding the data itself.\nN.B. — Layout is workload-specific Pruning quality is query-pattern-dependent: a partition layout tuned for one workload can be the wrong choice for another. The point is not that the table is “wrong” in an absolute sense; it is that the data layout is not aligned to the query pattern. delta-explain makes this misalignment visible. When part of the predicate cannot help The problem section said that pruning fails silently. Here is what that failure looks like when it finally has a voice. A function call over a column, UPPER(country) = 'DE', sits outside the pruning language delta-explain vouches for: there is no per-file min\/max for the computed value of UPPER(country) to compare against. Put it in the predicate and watch what the tool does with it:\ndelta-explain .\/users -w "UPPER(country) = 'DE' AND age > 40" Delta table: .\/users Version: 5 Predicate: UPPER(country) = 'DE' AND age > 40 Predicate Analysis: partition-safe: - stats-safe: age > 40 unsplittable: UPPER(country) = 'DE' confidence: incomplete Files in snapshot: 6 Phase 1: Data skipping (min\/max statistics) [incomplete] predicate: age > 40 (+1 unsupported fragment, keeps all files) files remaining: 3 (-3, 50% pruned) Warnings! [UNSUPPORTED_EXPRESSION]: Unsupported expression: UPPER(country); the fragment 'UPPER(country) = 'DE'' cannot contribute to pruning and is applied conservatively (keeps all files) The run does not fail. The supported conjunct still prunes half the table, the warning names exactly the fragment that could not contribute, and the confidence drops to incomplete so nothing downstream mistakes this for a fully attributed result. The phase line is honest about it too: rather than printing the whole predicate as if it all pruned, it shows age > 40 (+1 unsupported fragment, keeps all files), so the number and the predicate it is credited to always agree.\nN.B. — The line moves when the rewrite is provable The boundary of the pruning language is not fixed; it moves outward whenever a construct can be rewritten into the language soundly. A prefix pattern like country LIKE 'D%' is the clearest case: on a string column it is exactly the lexicographic range country >= 'D' AND country < 'E', so delta-explain now normalizes it into that range and prunes on it instead of degrading: delta-explain .\/users -w "country LIKE 'D%' AND age > 40" prunes 6 to 1, exactly as its rewritten form does. What stays outside the language is what cannot be rewritten this way: an arbitrary function like UPPER(...), or a column-to-column comparison, whose truth cannot be bounded by any single file’s metadata. The discipline is the same in both directions: a fragment is admitted only when the rewrite is provably equivalent, and reported as unsupported otherwise. Thresholds and the JSON contract The gates from the getting-started section leave two practical questions open: what threshold to set, and what the JSON output guarantees to whatever consumes it.\nThe recommended gate is not a fixed cutoff at all. Real ETL pipelines have non-uniform file counts across partitions, so a fixed threshold produces false positives when the selected partition is unusually large or when late-arriving backfills distort the baseline. Measure drift against a recent baseline instead, and when the assertion fires, rerun with --verbose --format json to identify which files lacked stats or which fragments landed in unsplittable, and --at-version to compare against the layout that used to pass.\nThe Python wrapper turns that advice into a few lines. Instead of hard-coding a percentage, compare today’s report against a version that was known to be healthy:\nimport delta_explain TABLE = "s3:\/\/warehouse\/events" PREDICATE = "date = '2026-07-01'" baseline = delta_explain.explain(TABLE, where=PREDICATE, at_version=41, env_creds=True) current = delta_explain.explain(TABLE, where=PREDICATE, env_creds=True) drift = baseline.total_pruning_pct - current.total_pruning_pct assert drift < 10, f"pruning dropped {drift:.1f} points against version 41" The same report supports guards that a percentage cannot express. analysis.confidence degrades to incomplete when part of the predicate could no longer be attributed (an unsupported construct, a mixed OR), which is often the first sign that a query changed shape upstream; and stats.mode moving from exact to partial catches a writer that started skipping statistics on some files before the pruning numbers visibly suffer:\nif current["analysis"]["confidence"] == "incomplete": raise RuntimeError("part of the predicate no longer contributes to pruning") if current["stats"]["mode"] != "exact": raise RuntimeError("some files are missing statistics") All of the above assumes a baseline exists. For a brand-new gate, the layout gives you a first floor: with N equal-sized partitions and an equality predicate that selects one of them, partition pruning eliminates (N-1)\/N of the files, so 365 daily partitions give 364\/365 \u2248 99.7% and four region partitions give 75%. With uniformly distributed file counts this is the exact figure; real workloads deviate in both directions, a sparse selected partition pruning more, a hot one less. Set the initial threshold safely below that floor, then switch to drift once a few healthy runs have accumulated.\nThat leaves the second question, and it holds for any consumer of the report, not just CI: what does the JSON document guarantee? Three things.\nFirst, the document identifies itself. It declares a schema_version (currently "0.4.0"), changes to it follow SemVer, and every report validates against a JSON Schema shipped in the repository and enforced in CI.\nSecond, the verdict is machine-stable. Every entry in assertions[] carries a stable name (min_pruning, stats_complete) and a result: pass | fail, mirrored by a top-level result; the stable name is what a calling system branches on.\nThird, stdout is all-or-nothing. An assertion failure prints the complete report (with result: "fail") and exits 1; a runtime error prints nothing to stdout, puts the message on stderr, and exits 1; a usage error exits 2. A failed run never emits a partial document, so delta-explain ... | jq cannot parse garbage.\nClosing delta-explain reasons at the protocol layer: the Delta transaction log, its actions, partition values, and file-level statistics. That is not the whole performance story. Large files with many row groups, Parquet reader pushdown, dynamic partition pruning, runtime filters, caching, and engine-specific planning all sit beyond this layer.\nBut the protocol layer answers an earlier question: before any engine opens a Parquet file, how many files can the metadata already rule out?\nThat question is often enough to expose the main problem. If a predicate still leaves thousands of files alive, the issue is usually visible in the log: wrong partitioning, missing statistics, or a layout that does not match the query pattern. If the same predicate leaves one file out of a hundred, later row-group pruning may still help, but the largest reduction has already happened upstream.\nThe report is therefore an opportunity, not a physical plan. delta-explain shows the file-level pruning opportunity available from the table metadata and the supplied predicate. It does not reproduce Spark’s, Trino’s, or Databricks’ execution plan, and it does not claim to explain all of performance.\nIts purpose is narrower: to make an invisible physical assumption visible, measurable, and assertable. Query engines explain what they plan to execute. delta-explain asks what the table metadata already proves before execution starts.\nThe exact guarantees behind that claim are defined in docs\/semantics.md. The companion piece, A Small Compiler for Explaining Delta Lake Pruning, explains how those numbers are produced, and why the design behind them looks the way it does.\n","permalink":"https:\/\/cdelmonte.dev\/projects\/delta-explain-making-delta-pruning-visible\/","summary":"Partition pruning and data skipping are invisible by default. delta-explain reads the Delta log directly and shows, step by step, how a WHERE predicate narrows down candidate files, with no engine required.","title":"delta-explain: Making Delta Lake Pruning Visible"},{"content":"What to know before writing an extension that enters Spark’s pipeline\nForeword In the previous article I described Catalyst as an embedded DSL: rules that pattern-match on tree nodes, batches executed by a RuleExecutor, an extension surface exposed via SparkSessionExtensions. The final example (a rule that intercepts cartesian joins, Python UDFs, and collect_list) was intentionally a sketch.\nWhen the sketch becomes real code running inside a cluster’s pipeline, things that looked like details turn into concrete problems. This article is the operational counterpart to that first one: a collection of anti-patterns I’ve encountered (some of which I’ve made myself) building PlanGuard in its operational version, with the goal of making the why behind each one explicit.\nThe temptation to put the throw inside an optimizer rule In the previous article PlanGuard is registered with injectCheckRule. The choice isn’t obvious. The most immediate reflex, reading the SparkSessionExtensions documentation,1 is to use injectOptimizerRule with a Rule[LogicalPlan] that throws an exception:\nobject BlockCartesian extends Rule[LogicalPlan] { def apply(plan: LogicalPlan): LogicalPlan = plan transform { case Join(_, _, Inner, None, _) => throw new SparkException("cartesian product detected") } } \u2026and register it with injectOptimizerRule. The rule enters the optimizer batch, pattern-matches on the Inner join without condition, throws. It works, at least the day you write it.\nWhat you don’t see, and what justifies the different choice in the main article, is where the throw ends up. An optimizer rule is invoked by RuleExecutor.execute inside a Batch(FixedPoint, ...), which is itself called by QueryExecution.optimizedPlan. Above sits assertAnalyzed, and above that the caller where the user wrote df.collect() or df.explain(). The throw crosses all these levels, and any operation that materializes executedPlan: df.collect(), df.show(), df.write..., even df.explain(), which is supposed to be there precisely to see the plan and diagnose it. Operations that stop at analysis (df.queryExecution.analyzed, df.printSchema which uses analyzed.schema) don’t trigger it: the optimizer is strictly lazy.\nThe right hook is injectCheckRule. A check rule is a function LogicalPlan => Unit that Spark invokes inside checkAnalysis, at the end of Analyzer.execute, so after analysis, before optimization. It can throw an exception to reject the plan. It’s documented by Spark for exactly this purpose: “this query must not be executed”.\nA useful mechanical detail: analysis is memoized via QueryExecution.analyzed (lazy val), so on a DataFrame that has already been analyzed explain() does not re-fire the check rule. But if the plan hasn’t been analyzed yet, explain() does activate it: the end of analysis is the single point at which the check rule runs.\nThere’s also a less obvious temporal consequence. With injectOptimizerRule, the throw reaches the caller of df.collect() (the optimizer is lazy: rewriting only happens when an action requires the physical plan). With injectCheckRule, analysis is run inside Dataset.ofRows, which is called by every method that returns a new DataFrame: df.filter, df.join, df.select. The query “blows up” at the DataFrame’s construction, not at collect.\nval joined = left.join(right) \/\/ \u2190 with CheckRule, SparkException is thrown here joined.collect() \/\/ \u2190 with OptimizerRule, it would have been here From the user’s perspective the check rule is better: the failure is attached to analysis, not to optimization or execution. In Spark classic this usually means the exception is raised while constructing the next DataFrame, close to the offending transformation, rather than being deferred to collect\/show\/write. But it’s a detail that changes the shape of the tests: the try\/catch must wrap the DataFrame construction, not just the action. It’s exactly the shift that separates an “article-sketch rule” from a “pipeline rule”.\nIn short, choose the hook based on intent:\nsemantic invariant of the plan (rejecting queries that must not run) \u2192 injectCheckRule2 plan rewriting (transformation that produces a new LogicalPlan) \u2192 injectOptimizerRule per-query metrics\/observability (observing without modifying) \u2192 QueryExecutionListener Side-effects and FixedPoint multiplication The other reason injectCheckRule is preferred isn’t the throw: it’s the number of times the rule is called. Even when you write a legitimate optimizer rule (one that actually rewrites the plan, because that’s what optimizer rules are supposed to do) and you add “innocent” side-effects like logging or metric emission, there’s a less obvious consequence. Spark’s optimizer organizes its rules into batches; many batches have a FixedPoint strategy, which means: “execute all the batch’s rules repeatedly until the plan stops changing”.\nBatch "Operator Optimization" (FixedPoint) \u251c\u2500\u2500 PredicatePushdown \u251c\u2500\u2500 ConstantFolding \u251c\u2500\u2500 ColumnPruning \u251c\u2500\u2500 DetectDangerousPatterns \u2190 ours \u2514\u2500\u2500 SimplifyConditionals If our rule is a no-op (returns the plan unchanged), it doesn’t contribute to termination. But if other rules in the batch keep rewriting (ConstantFolding simplifies an expression, PredicatePushdown moves a filter, ColumnPruning trims a projection), every batch iteration re-applies ours too (in the absence of pruning declared via nodePatterns\/containsAnyPattern). Typically this is a few iterations for a simple plan; with subqueries, CTEs, or complex predicates the number can rise significantly.\nThe result is concretely observable, and reproducible. The script examples\/fixedpoint_multiplication_demo.scala builds a custom RuleExecutor whose single batch is FixedPoint(100) containing a counting probe alongside ConstantFolding, BooleanSimplification, and CombineFilters, three real Catalyst rules whose rewrites keep the batch iterating until convergence. The probe is observational (returns the plan unchanged); the other three are what makes the batch loop. Run on a query with three adjacent Filter nodes (Spark 4.1.1, local[1]):\n[demo] Filters in analyzed plan (N = expected count): 3 [demo] CountingProbe invocations under FixedPoint: 6 [demo] CountingProbe invocations as a check rule (one-shot): 3 The probe fires 6 times for 3 actual nodes; the same logic applied as a check rule, outside any FixedPoint, fires exactly 3. The 2\u00d7 ratio is artifact-specific (it depends on which rules are in the batch and how quickly they converge), but the structural point is unchanged: every FixedPoint iteration re-applies the rule, and any side-effect (Prometheus counter increment, log warning, audit event) is duplicated accordingly. In real Spark optimizer batches with subqueries, CTEs, or complex predicates, the multiplier rises.\nFaced with this problem, the practicable routes are three, in order of preference:\nMove the rule into a check rule (as in the previous point). Check rules run once after analysis: no FixedPoint, no multiplication.\nMemoize per plan-id (treat as anti-pattern, not solution). The natural temptation is to track the System.identityHashCode(plan) of the input and skip if already seen. It doesn’t work: Catalyst is immutable, every transform\/copy produces a new instance, so at iteration N+1 the input plan has a different identityHashCode from iteration N even if structurally identical. The cache never hits between iterations. To deduplicate across iterations the key must be structural (plan.canonicalized or a hash of treeString), not referential, but at that point the cost of structural comparison exceeds the cost of the duplication you were trying to avoid. The truly correct pattern, when the need is “one metric per query”, is to observe outside the optimizer: QueryExecutionListener.onSuccess is the documented point for per-query emissions. Caveat: it only sees queries that reach execution. Queries rejected in analysis don’t trigger it, so it isn’t a substitute for a check rule when enforcement is required.\nUse transformDownWithPruning with nodePatterns. It doesn’t resolve the multiplication of emissions across iterations (the rule still gets revisited), but it reduces the cost of a single traversal by skipping entire subtrees that don’t contain the patterns of interest. It’s a cost mitigation, not an architectural fix.\nThe rule many people think they’ve written (“one emission per detection”) corresponds to the check rule. The one written as an optimizer rule is “one emission per detection per batch iteration”, a different thing.\nUnprotected mutable state The architectural observation comes before the code: if a Rule needs mutable state to be correct, it’s probably in the wrong place. A check rule that runs once doesn’t have this problem; neither does a QueryExecutionListener for per-query metrics. If the temptation comes anyway, it’s worth seeing why the “obvious” mitigations (var, ThreadLocal, ConcurrentHashMap) are traps.\nThe memoization from the previous point (“track the plans already processed”) looks innocuous. It’s written like this:\nprivate var lastSeen: LogicalPlan = _ def apply(plan: LogicalPlan): LogicalPlan = { if (plan eq lastSeen) return plan lastSeen = plan \/\/ ... emit, log, etc. plan } A var inside a Rule. It works as long as there’s one query at a time on a single rule instance. Spark by default doesn’t parallelize the optimizer across different queries, but several scenarios break this quiet: SparkSession.newSession() allows multiple sessions on the same JVM; notebooks execute cells sequentially, but not always; Spark Connect (SparkConnectService) keeps the same JVM alive with N clients; AQE (AdaptiveSparkPlanExec) re-applies the optimizer on subplans during execution, on threads different from the main one. How the rule is instantiated depends on how you register it. injectOptimizerRule(_ => MyRule) with MyRule as a singleton object means state shared globally across every session in the JVM; injectOptimizerRule(session => new MyRule(session)) gives at least one instance per session, but that same instance can still be used by multiple concurrent queries within the same session.\nThreadLocal isn’t a reliable way out: AQE crosses threads (sub-plans are optimized in scala.concurrent.Future with a dedicated executor), and reused thread pools can leak values across queries. ConcurrentHashMap keyed on identityHashCode is the FixedPoint multiplication problem amplified: a LogicalPlan’s referential identity changes with every transform\/copy, so the cache doesn’t hit either across FixedPoint iterations or across queries. To really deduplicate across plans the key must be structural (canonicalized), and at that point you’ve written a mini-optimizer inside a rule.\nConclusion: mutable state in a Rule is a time bomb, and the mitigations one would find writing the fix alone are almost all wrong. If the need is “don’t emit the same warning more than once in 5 minutes”, that rate-limit belongs to the observability system, not to the Spark rule.\nforeach when transformDownWithPruning would have done A typical inspection rule does something like:\ndef apply(plan: LogicalPlan): LogicalPlan = { plan.foreach { case Filter(condition, _) if isDangerous(condition) => logWarning("...") case _ => } plan } plan.foreach visits every node of the tree, whatever the pattern of interest. On a large plan (nested subqueries, views, CTEs, broadcast hints) that means many visits per rule, per iteration. It isn’t a correctness problem (foreach doesn’t rewrite the plan, so it doesn’t violate idempotency); it’s a cost problem. Catalyst has an API that does the same thing with structural pruning: transformDownWithPruning, paired with a containsAnyPattern(...) test that skips entire subtrees if they don’t contain any of the indicated patterns.\nimport org.apache.spark.sql.catalyst.trees.TreePattern._ def apply(plan: LogicalPlan): LogicalPlan = { plan.transformDownWithPruning(_.containsAnyPattern(JOIN, FILTER, AGGREGATE)) { case f @ Filter(condition, _) if isDangerous(condition) => logWarning("...") f case other => other } } The mechanism is the same one Spark’s native rules use internally: nodePatterns and TreePatternBits were introduced in Spark 3.2 precisely to reduce the cost of iteration in the FixedPoint. It’s an accelerator for the rewriting engine: subtrees without the indicated patterns are skipped entirely, so on plans with few relevant nodes one visits a fraction of the total nodes. It must be kept distinct from idempotency and termination: containsAnyPattern doesn’t help convergence, it only accelerates it.3\nIdempotency An optimizer rule should be idempotent, or at least participate in a batch whose composed transformation reaches a fixed point.4 Catalyst does not check idempotency rule by rule: it compares the plan produced by one batch iteration with that of the previous iteration, and stops when they are structurally equal (fastEquals). In practice idempotency holds as operational discipline: without it, the batch may diverge or saturate the iteration limit. Semantically equivalent plans can have different normal forms, and the order of rules in the batch determines which one is obtained: that’s why order matters.\n\/\/ Anti-pattern: the rule keeps "improving" the plan and never terminates. case Filter(c, child) => Filter(And(c, Literal(true)), child) This example is obvious. There are less obvious ones, and they tend to appear when the rule introduces syntactically fresh nodes on each application. For example:\n\/\/ Anti-pattern: each application creates new Aliases with new ExprIds. case Project(projectList, child) => Project(projectList.map(e => Alias(e, e.name)()), child) The plan stays semantically identical, but it changes structurally on every iteration: every Alias() without an explicit ExprId generates a new one, fastEquals fails, and the FixedPoint doesn’t terminate.\nSpark has a quiet way to protect itself from non-idempotent rules: FixedPoint has a maximum iteration limit (spark.sql.optimizer.maxIterations, default 100), beyond which it stops and logs a warning. It works, but the warning is easily ignored in production, and a rule that converges at “100 iterations” is a rule doing useless work hundreds of times per query.\nThe minimum viable test, when writing a rule, is: apply it, take the output, apply it again, verify output1.fastEquals(output2). It catches the most frequent class of non-idempotency: the one shown above, where the rule introduces syntactically fresh nodes (Alias() without explicit ExprId, freshly allocated case classes) that defeat structural equality on every application. For subtler equivalences (reordering of conjuncts in an And, reassociation of filters), fastEquals is too strict; the upgrade path is plan.canonicalized.fastEquals(other.canonicalized), which normalizes the tree before comparing.\nTesting rules the way Spark tests its own Idempotency is the minimum bar. Spark also exposes the scaffolding it uses to test its own rules, and an extension that adopts the same scaffolding gets much closer to the testing discipline Spark uses for native rules. PlanChangeLogger (spark.sql.planChangeLog.level=WARN) logs rule applications that actually change the plan, together with batch-level summaries and metrics. It’s the same instrument Spark uses to inspect optimizer rewrites, and it works equally well for user-injected rules.\nRuleExecutor exposes two override hooks, validatePlanChanges and validatePlanChangesLightweight, that run plan-integrity checks after every rule application; they’re gated by spark.sql.planChangeValidation and spark.sql.lightweightPlanChangeValidation. The actual checker is LogicalPlanIntegrity (org.apache.spark.sql.catalyst.plans.logical), with validateOptimizedPlan, validateExprIdUniqueness, and validateSchemaOutput covering the most common breakage modes: unresolved attributes after analysis, broken exprId references, schema drift between input and output. For unit testing, PlanTest and its derivatives (AnalysisTest, PlanTestBase) are the fixtures the Catalyst test suite uses: they run rules in isolation, compare logical plans with normalized exprIds, and assert on plan structure rather than on rendered strings. The practical consequence: a test that fails after a Spark version bump because the plan’s treeString formatting changed is a test you wrote yourself; a test that fails because the structure of the plan changed is the same kind of signal Spark’s own CI catches.\nJVM bootstrap in PySpark This last one isn’t strictly a rule-writing anti-pattern, but it’s the trap one falls into deploying an extension: the JAR doesn’t load and Spark logs a seemingly inexplicable ClassNotFoundException. The main article touches on it as an operational detail; here what matters is the diagnosis and the scenarios the basic pattern doesn’t cover.\nspark = ( SparkSession.builder .config("spark.jars", "\/path\/to\/extension.jar") # \u2190 often too late .config("spark.sql.extensions", "com.example.MyExtension") .getOrCreate() ) The anti-pattern isn’t spark.jars itself. The Spark docs are clear: spark.jars is a comma-separated list of JARs included on the driver and executor classpaths. The anti-pattern is setting it too late, once the driver JVM\/SparkContext is already up: the builder option cannot retroactively change the already-fixed classpath.\nThe order of events is this: PySpark, when getOrCreate() is invoked for the first time, launches a new JVM via python\/pyspark\/java_gateway.py (the function is called launch_gateway). The JVM receives the driver classpath at launch, built by spark-submit from PYSPARK_SUBMIT_ARGS (in particular --jars). Once up, the JVM reads spark.sql.extensions and looks up the class on the driver classpath, which is already fixed. In notebooks and managed environments where the driver is already running before user code starts, .config("spark.jars", ...) in the builder is simply too late. Hence the ClassNotFoundException.\nThe correct forms depend on the environment:\n# 1. via spark-submit, --jars is read at JVM launch spark-submit --jars \/path\/to\/extension.jar your_script.py # 2. from "pure" Python, set PYSPARK_SUBMIT_ARGS before import import os os.environ["PYSPARK_SUBMIT_ARGS"] = ( "--jars \/path\/to\/extension.jar pyspark-shell" ) from pyspark.sql import SparkSession # \u2190 now the JVM starts with the JAR spark = SparkSession.builder \\ .config("spark.sql.extensions", "com.example.MyExtension") \\ .getOrCreate() For dependencies resolved as Maven coordinates, the equivalent is spark.jars.packages (also passed via PYSPARK_SUBMIT_ARGS or --packages to spark-submit): same temporal constraint, must be set before the JVM launches.\nIn managed environments the os.environ pattern doesn’t work at all: the driver JVM is launched by the infrastructure, before user code runs. Quick references:\nEnvironment Mechanism spark-submit \/ local PySpark --jars or PYSPARK_SUBMIT_ARGS as above Databricks (attached cluster) cluster-level init script that installs the JAR in \/databricks\/jars\/, or library install from the UI\/API EMR Serverless application-configuration with sparkSubmitParameters that include --jars Dataproc Serverless gcloud dataproc batches submit with --properties spark.jars=... AWS Glue --extra-jars in job parameters; sealed classpath, no runtime override A separate wrinkle for self-managed Spark with --deploy-mode cluster (YARN, Kubernetes, Standalone): spark-submit --jars still works (the cluster manager handles distribution to the remote driver), but the os.environ["PYSPARK_SUBMIT_ARGS"] trick presupposes that the local Python process launches the driver JVM, which isn’t the case when the driver runs remotely. Cluster mode goes through spark-submit proper, not through PySpark’s local gateway.\nThe same constraint applies to Delta Lake and Iceberg, and to anything based on spark.sql.extensions. It’s not a property of the extension: it’s a property of the JVM bootstrap in PySpark. If you don’t know it, you only learn it after losing an afternoon.\nConclusions Catalyst makes it easy to write a rule: literally, a LogicalPlan => LogicalPlan function with a pattern match inside. It also makes you believe, at first, that this is enough to solve real problems. The distance between “the rule matches the right pattern” and “the rule behaves well inside the pipeline” is short to write but long to walk: choosing between optimizer rule and check rule, understanding FixedPoint’s effect on emissions, avoiding mutable state, leveraging pruning, ensuring idempotency, testing the rule the way Spark tests its own, and handling JAR bootstrap.\nNone of the above is esoteric: it’s in Spark’s sources, in the project’s issues, in various Data+AI Summit talks. But it’s not collected in one place, and for someone writing their first extension it’s almost all invisible until it manifests as duplicates in spans, failed classloading in production, or a FixedPoint that doesn’t terminate. The DSL is simple; the framework around the DSL is where the real complexity hides.\nNotes SparkSessionExtensions ScalaDoc: the documented Developer API for the extension point. The corresponding source resides in org.apache.spark.sql.catalyst.{trees, rules, plans}: RuleExecutor, Rule, TreePattern, and the analysis check phase (checkAnalysis). ↩︎\nThe check rule fires once per analyzed plan. If the same query goes through CacheManager\/Dataset.persist and is re-executed on a cache hit, the check rule is not re-applied: the plan has already “passed” analysis. ↩︎\nTwo practical notes. Readability: transformDownWithPruning has the same signature as transform, and for an inspection rule that returns the node unchanged the visual pattern is identical; the difference is only in the pruning predicate, which is one line. Compatibility: TreePattern is Spark’s internal API (org.apache.spark.sql.catalyst.trees), and its members can change between minor versions (3.2 \u2192 3.3 \u2192 3.5 \u2192 4.0); for extensions distributed cross-version it’s worth verifying against the target version before relying on a specific enum name. ↩︎\nIn term-rewriting theory the standard termination criterion is the existence of a reduction ordering (well-founded, monotonic, stable on contexts and substitutions); Knuth\u2013Bendix completion presupposes one to orient equations toward confluence. Per-rule idempotency is sufficient but not necessary: mutually recursive rules can reach a structural fixed point without any one being idempotent. The non-uniqueness of normal forms is the related lack of confluence (Church\u2013Rosser): both theoretical conditions are useful discipline for Catalyst rule authors, but neither is enforced by the framework. ↩︎\n","permalink":"https:\/\/cdelmonte.dev\/deep-dives\/anti-patterns-in-catalyst-rules\/","summary":"Six concrete anti-patterns I encountered building a real Catalyst extension: from the wrong rule type for throws, to mutable state under AQE and Spark Connect, to JVM bootstrap traps in PySpark.","title":"Anti-patterns in Catalyst rules"},{"content":"This analysis reflects the state of the ecosystem as of early 2026.\nFor the better part of a decade, the standard way to build a reliable data platform on object storage was: use Spark, add Delta, deploy on Databricks. It worked. It still works. But underneath that answer lies an architectural assumption that is quietly being dismantled: the assumption that the protocol, the data representation, the query engine, and the runtime should live in the same monolithic stack.\nThis article traces the disaggregation of that monolith (bottom-up, layer by layer) and maps what is replacing it.\nThe Monolith and Its Problems Delta Lake as it was designed Delta Lake was open-sourced by Databricks in 2019, but it was not designed in the abstract. It had been incubated internally before the open-source release: designed for Spark, written in Scala, tightly coupled to SparkSession, SparkContext, and Catalyst internals. The DeltaLog reads from a Spark-accessible filesystem. The OptimisticTransaction commits through Spark-configured storage. Schema evolution, partition pruning, and time travel were all surfaced through Spark-integrated APIs and planner internals.\nThis was not negligence. It was pragmatism. The problem Delta was solving (ACID semantics on object storage, schema enforcement, time travel) needed to be solved now, for Spark, where the actual workloads were running. Tight coupling was the fastest path to production.\nAs the ecosystem expanded, the cost of that coupling became harder to ignore.\nThe connector proliferation problem Engines and access layers outside Spark (Trino, Flink, Hive, Presto, Redshift Spectrum, and a growing set of Python-native libraries) all needed to read and write Delta tables. But the authoritative implementation of Delta’s protocol semantics was tightly coupled to Spark. Every other engine had to reimplement the transaction log interpretation on its own.\nThe consequences were unsurprising. New protocol features would ship in the Spark connector and take months to appear in other engines. Subtle differences in how each connector interpreted the transaction log led to correctness bugs difficult to diagnose. And every connector team bore the maintenance cost of a parallel implementation of the same protocol logic.\nThis is not an accident of Delta’s design. It is a predictable consequence of embedding protocol semantics inside an engine-coupled implementation. The question was how to separate that logic from the engine.\nThe Architectural Response Extracting the protocol The answer was not to build better connectors one by one. It was to separate what the protocol says from how the data is processed.\nDelta Kernel is the initiative that emerged from this insight: a set of libraries, developed under the delta-io umbrella, whose goal is to extract protocol logic into a shared layer independent of any engine. The division of responsibility is precise: snapshot construction, log interpretation, feature handling, and schema semantics live in the kernel; file reading, expression evaluation, and execution (including distributed execution where relevant) remain the responsibility of the host engine. The kernel exists in two implementation lines: a Java kernel for JVM-based engines, and a Rust kernel for non-JVM and cross-language environments, with C FFI bindings for embedding.1\nBefore Delta Kernel: \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 Delta Scala \u2502 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u2502 \u2502 Protocol logic (Delta) \u2502 \u2502 \u2502 \u2502 + Spark runtime coupling \u2502 \u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 After Delta Kernel: \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 Spark \/ Flink \/ Trino \/ \u2502 \u2190 engine-specific execution \u2502 DataFusion \/ custom engine \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 Engine interface \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 Delta Kernel \u2502 \u2190 protocol logic only \u2502 \u2022 transaction log parsing \u2502 \u2502 \u2022 snapshot resolution \u2502 \u2502 \u2022 partition pruning metadata \u2502 \u2502 \u2022 column mapping \u2502 \u2502 \u2022 schema enforcement \u2502 \u2502 \u2022 deletion vector application \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 delta-rs and delta-kernel-rs delta-rs started in 2020 as a full Rust implementation of Delta Lake, with DataFusion integration and Python bindings. Today, while delta-rs retains everything operational (DML, conflict resolution, retry logic), delta-kernel-rs, developed as part of the Delta Kernel initiative starting in 2023, is emerging as the shared protocol foundation: snapshot resolution, log interpretation, schema enforcement, and metadata semantics. Its write path is still experimental (currently limited to blind appends), but the read-side protocol semantics are already delegated from delta-rs.\nThe kernel is engine-agnostic. It knows what needs to happen (which log entries to read, which predicates to evaluate, which files belong to a snapshot) but not how. The how is defined by the caller through the Engine interface (in Rust, an interface is called a trait): how to access storage, how to evaluate expressions, how to parse JSON and Parquet.2\npub trait Engine: AsAny { fn evaluation_handler(&self) -> Arc<dyn EvaluationHandler>; fn storage_handler(&self) -> Arc<dyn StorageHandler>; fn json_handler(&self) -> Arc<dyn JsonHandler>; fn parquet_handler(&self) -> Arc<dyn ParquetHandler>; } This is the seam where the kernel stops and the host engine begins.3 The EngineData abstraction is nominally format-agnostic, but in practice the shipped reference path is Arrow-based, making Arrow the de facto data exchange format in the default implementation path today.\nThe Disaggregation The Engine trait of delta-kernel-rs reveals something larger than API design. It is not only the engine being decoupled from protocol semantics: the entire monolithic stack is coming apart into independent layers with explicit boundaries:\nProtocol: Delta Kernel, shared semantics, embeddable in any engine In-memory format: Apache Arrow, identical columnar layout across languages4 Query planning and execution: pluggable. DataFusion (planning + execution) in the Rust ecosystem, Calcite (planning framework) in the JVM ecosystem (Flink, Drill, and others), Catalyst in Spark, Trino’s own cost-based optimizer5 Each layer is now an independent library, replaceable without disturbing its neighbors. This is what disaggregation means in this context: not the disappearance of the monolith, but the modularization of what used to be inseparable.\nConvergence, not parity In practice, the clean separation described above holds only partially. And that is exactly the point.\nDelta Kernel is an explicit attempt to create a shared protocol layer, but the ecosystem remains uneven. The Flink connector still relies on Delta Standalone (the pre-kernel Java library, no longer the strategic foundation for new connector work), and a kernel-based Flink 2.0 connector has only been proposed, not built.6 On the Rust side, delta-rs and delta-kernel-rs are advancing quickly, but public issue trackers still show open gaps around several newer table features: liquid clustering, row tracking, and v2 checkpoints among them.789\nWhat is converging is not feature parity. It is the boundary. The old world was one in which connectors reimplemented Delta protocol behavior in parallel, with divergence accumulating silently. The emerging world is one in which engines may still differ in maturity and operational surface, but are no longer forced, by design, to diverge at the level of protocol interpretation.\nThe pattern is moving upward Apache Comet is the clearest signal that the disaggregation is moving beyond protocol and format into the engine itself. Comet operates at the Spark execution layer and is format-agnostic: it works equally with Delta, Iceberg, or plain Parquet. It keeps Spark’s distributed runtime and Catalyst’s logical planning intact, but replaces physical execution operators with Rust implementations built on DataFusion and Arrow. Plans cross the JVM\/native boundary via Protobuf; data crosses via Arrow FFI: zero-copy columnar exchange across the language boundary.10 Comet’s operator coverage is expanding, though not all operators are covered, and unsupported ones fall back to JVM execution transparently. Meta’s Velox pursues a similar pattern with a C++ execution engine and its own columnar format, interoperable with Arrow but not Arrow-native.11 Together, these projects demonstrate that the separation of protocol from execution, which Delta Kernel introduced at the storage layer, is now being applied one level up: between planning and execution within the query engine itself.\nThe disaggregation is moving upward through the stack, but it stops at the distributed runtime. The next two sections examine why.\nWhat the Architecture Does Not (Yet) Cover The layered picture described above is architecturally coherent, but several areas remain outside its reach.\nStreaming: the protocol layer provides primitives for exactly-once sinks and change data feeds, but the orchestration layer (micro-batch triggering, offset tracking, watermark management) remains tightly coupled to Spark Structured Streaming or Flink. The Rust ecosystem does not yet offer a streaming equivalent with comparable production maturity.\nCatalog integration: the disaggregated stack has no portable, vendor-neutral catalog. Hive Metastore and AWS Glue integrations exist at the engine level (Spark, Trino) but are not part of the shared protocol layer, and Unity Catalog was open-sourced in mid-2024 but remains operationally centered on the Databricks ecosystem.\nOperational complexity: the monolithic stack handles maintenance and governance implicitly; a disaggregated deployment requires building that infrastructure explicitly. In practice, this means teams must own VACUUM scheduling, access control policies, and incident response across heterogeneous engines, for a stack whose failure modes no longer fit a single vendor’s documentation. Dependency coordination across independently released components is a concrete cost: version skew is a real operational burden that the monolithic stack did not expose as visibly. These are solvable problems, but the engineering effort is real.\nThese are real gaps, but they are gaps of coverage. The distributed runtime is a different kind of problem.\nThe Missing Layer The distributed runtime is the layer most structurally resistant to disaggregation.\nWhy this layer is different The distributed runtime layer is harder to disaggregate for a structural reason: its components are deeply entangled with each other. The shuffle service, memory manager, task scheduler, and DAG scheduler are tightly interdependent. Cluster state, partial failure handling, scheduling under resource pressure: these are not independent concerns that can be extracted one at a time. Spark has spent more than a decade solving them together.\nThis is why you cannot replace Spark with DataFusion + Calcite + delta-kernel and get equivalent behavior at scale. The single-node path is real: the Embucket team demonstrated DataFusion completing all 22 TPC-H queries at SF1000 (~1 TB) on a single node.12 But once your data outgrows a single machine (or your workload requires stateful fault tolerance, as streaming does) you need a distributed runtime.13\nThis is also why newer systems (Trino, Flink SQL, Ray Data, Dask) still end up rebuilding substantial portions of scheduling, shuffle, and fault-tolerance infrastructure.\nThe pressure building on this layer The distributed runtime is harder to disaggregate, but it is not immune to the same forces. The pressure builds more slowly and comes from a different direction.\nManaged shuffle is the first crack. LinkedIn’s Magnet (a push-based shuffle optimization developed at LinkedIn for Spark), Apache Celeborn (external shuffle service), and AWS Glue’s cloud shuffle storage plugin all extract or externalize shuffle (the most expensive and most Spark-specific component of the distributed runtime) into a standalone service.\nRay attacks the problem from the opposite direction: provide a general-purpose distributed computing framework on which data systems can be built. But Ray was designed with different priorities: fine-grained actor and task parallelism, not coarse-grained stage-based SQL execution with optimized shuffle. Ray Data’s analytical path is evolving, but as of early 2026 its shuffle and fault-tolerance model have not yet reached the same level of SQL-oriented maturity and production hardening that Spark provides.\nConclusion What began as a monolith is separating into layers with explicit boundaries. The pattern is consistent: each layer, once extracted behind a stable interface, becomes independently replaceable. The bottom layers (protocol and in-memory representation) are already disaggregated behind stable interfaces. Execution is now the layer where disaggregation is most visibly advancing. The distributed runtime remains the last intact monolithic layer, and the most structurally resistant to disaggregation, which is why Spark is still the center of Databricks’ commercial offering even as the layers around it come apart.\nThe common framing (“will DataFusion replace Spark?”, “is DuckDB the new Spark?”) misses the structural point. DataFusion and DuckDB are execution engines. They compete with Spark’s execution operators, not with Spark’s distributed runtime. Comet makes this explicit by embedding DataFusion inside Spark’s runtime rather than replacing it. As the protocol and in-memory layers become commoditized, the actual competition is between distributed runtimes (Spark, Flink, Ray, and their respective ecosystems), and that is where the next chapter of this story will be written.\nFor engineers building on this stack, the practical consequence is composability that would have been difficult to achieve even a few years ago: mixing engines against shared Delta tables, validating protocol correctness locally with pip install deltalake instead of standing up a cluster, embedding native execution inside a JVM runtime.14\nThe lakehouse stack is not converging on a new dominant engine. It is converging on a layered architecture.\nNotes The delta-kernel-rs FFI layer (ffi\/ crate) exposes a C API using extern "C" functions and repr(C) types. Iterators are modeled as C-compatible structs with function pointers (EngineIterator), and all kernel handles use a Handle type that manages ownership across the FFI boundary. This enables embedding in C, C++, Python (via ctypes\/cffi), and JVM (via JNI) without requiring Rust on the consumer side. ↩︎\n“Snapshot resolution” is the process of computing the current state of a Delta table at a given version: reading the transaction log (JSON commit files and checkpoint Parquet files), replaying add\/remove actions, and producing the set of data files that constitute the table at that point in time. This is the most fundamental operation the kernel performs: everything else (scans, schema reads, statistics) depends on having a resolved snapshot. ↩︎\nThe kernel ships a DefaultEngine (behind feature flags default-engine-native-tls \/ default-engine-rustls) that bundles Arrow, Parquet, cloud object store support, and async I\/O via tokio, lighter than a full query engine but not trivial. delta-rs uses DefaultEngine for all protocol operations. Apache DataFusion is an optional integration layer (behind a datafusion feature flag, though always enabled in the Python distribution) that provides SQL and DataFrame query capabilities on top. ↩︎\nApache Arrow defines a columnar in-memory format with an identical memory layout across languages: Rust, Python, Java, C++ all represent Arrow buffers the same way. Data crosses component boundaries (kernel to engine, engine to connector, connector to caller) with little or no reshaping. Delta Lake stores data on disk as Parquet, which is also columnar. Parquet and Arrow share the same logical data model (typed columns, stored contiguously), so reading Parquet into Arrow is a columnar decode rather than a format conversion, substantially cheaper than materializing row-oriented structures. The kernel’s built-in DefaultEngine uses arrow-rs and parquet-rs directly; DataFusion also operates natively on Arrow RecordBatch structures. ↩︎\nApache Calcite provides SQL parsing, relational algebra, and an optimizer framework used by Flink, Drill, Phoenix, and Beam. The analogy with DataFusion is architectural, not literal: Calcite is primarily a planning framework that delegates execution to external engines, while DataFusion combines planning and execution in a single library. The relevance to the disaggregation thesis is that Flink’s Delta connector runs through Calcite’s planning layer, though as of early 2026, it still uses Delta Standalone rather than the JVM kernel for protocol semantics. ↩︎\nSee Feature Request: Flink 2.0 connector built on Delta Kernel. The existing Flink connector depends on Delta Standalone, which is no longer released in the 4.x line and receives only critical fixes in 3.x. A design document for a kernel-based Flink connector exists, but no implementation work has started as of early 2026. ↩︎\nSee delta-rs issue #3249: Features, Support needed, which tracks liquid clustering, row tracking, and v2 checkpoints as features still requiring implementation or completion. The distinction between delta-kernel-rs (protocol-level read support) and delta-rs (full operational support including writes) matters here: the kernel may recognize a protocol feature for reading purposes while delta-rs has not yet implemented the corresponding write-path operations. ↩︎\ndelta-rs implements DML operations in crates\/core\/src\/operations\/: merge\/, update\/, delete.rs, optimize.rs (including Z-order), vacuum.rs, load_cdf.rs, constraints.rs, restore.rs. Each uses DataFusion for query planning and execution, and the kernel for protocol-level commit. The kernel provides the primitives (add files, remove files, update deletion vectors); delta-rs provides the semantics (which rows to delete, how to merge, when to compact). The full list of delta-rs operations above the kernel boundary includes MERGE, UPDATE, DELETE, OPTIMIZE (including Z-order), VACUUM, Change Data Feed reads, CHECK constraints, additive schema evolution, and table restore, all gated behind the datafusion feature flag and not available through delta-kernel-rs alone. ↩︎\nThe kernel’s TableChanges implementation requires identical schemas across the queried version range: compatible but non-identical schemas are not yet supported. Column mapping (both name and id modes) is supported for CDF reads as of v0.20. ↩︎\nComet serializes Spark physical plan nodes to Protobuf (defined in native\/proto\/src\/proto\/operator.proto). The serialized plan crosses the JNI bridge via CometExecIterator \u2192 Native.createPlan(). On the Rust side, the Protobuf is converted to DataFusion physical plan nodes via PhysicalPlanner. Data flows via the Arrow C Data Interface: raw memory pointers to Arrow arrays and schemas cross JNI, enabling zero-copy data sharing between JVM and native memory. Unsupported operators fall back to the original Spark JVM operator transparently (CometExecRule.scala). ↩︎\nMeta’s Velox is a C++ execution engine that operates on its own columnar in-memory format (RowVector\/BaseVector), not directly on Arrow. It provides Arrow import\/export via the Arrow C Data Interface, meaning data crosses the Velox boundary through a (typically zero-copy) conversion step. Velox’s relationship to Arrow is thus more like “interoperable peer” than “native consumer,” unlike DataFusion which operates directly on Arrow RecordBatch structures. Velox was created by Meta. ↩︎\nThe Embucket team demonstrated DataFusion completing all 22 TPC-H queries at SF1000 (\u22481 TB) on a single node in November 2025, illustrating the performance envelope of the Rust\/Arrow path on hardware-optimized single-node configurations. The DataFusion benchmarks repository (apache\/datafusion-benchmarks) provides tooling to reproduce TPC-H derived benchmarks at various scale factors. These numbers should be understood as directional indicators, not as controlled head-to-head benchmarks. ↩︎\nThe threshold is workload-dependent rather than a fixed data size. Scan-heavy analytics can run on a single node well into the terabyte range (as the Embucket SF1000 result demonstrates), but shuffle-heavy workloads (large joins, high-cardinality aggregations) exhaust single-node capacity much sooner. The commonly cited 500 GB to 1 TB range is typical for in-memory-dominant execution, not a hard wall. DuckDB spills aggressively to disk and can push beyond it, while DataFusion’s spill support is less mature. The point is not a number but a shape. ↩︎\nIn practice, the gap extends beyond expression evaluation. Concurrent write resolution (the behavior when two writers conflict) differs substantially: Delta Spark and delta-rs both implement conflict resolution with isolation levels (Serializable and WriteSerializable at the protocol level, with SnapshotIsolation as an internal conflict-checking mode), while delta-kernel-rs reports conflicts without resolving them. Table maintenance behavior (statistics quality after OPTIMIZE, file sizing, Z-order effectiveness) also varies. For a local-development workflow, these differences are unlikely to surface. For integration testing against production-like concurrency patterns, they matter. ↩︎\n","permalink":"https:\/\/cdelmonte.dev\/essays\/the-disaggregation-of-the-lakehouse-stack\/","summary":"How Delta Kernel, Arrow, and pluggable execution are disaggregating the lakehouse stack. The lakehouse stack is not converging on a new dominant engine. It is converging on a layered architecture in which protocol, data representation, and query execution are increasingly isolated behind stable interfaces.","title":"The Disaggregation of the Lakehouse Stack"},{"content":"There is a class of production incident that resists conventional diagnosis. The system did exactly what it was supposed to do. The API was used correctly. The documentation, consulted afterward, turns out to have been accurate all along. And yet the outcome violated every expectation: a job that rewrites ten times more data than expected, a container that provides less isolation than its interface suggests, a join that degrades catastrophically at scale.\nBeyond Leaky Abstractions The standard diagnosis is “leaky abstraction,” a term Joel Spolsky popularized in 2002.1 All abstractions leak, he argued. The implementation details eventually surface. This is true, but it explains very little. It doesn’t tell you which abstractions leak, where they leak, or why the leaks are so consistently surprising. It names the phenomenon without explaining its structure.\nI want to propose a more precise diagnosis, one that draws on semiotics2 rather than software engineering, and specifically on the work of Charles Sanders Peirce and Umberto Eco. The claim is this: many of the most persistent and surprising performance pathologies in modern data systems are semiotic failures: predictable, recurring, and invisible to the user the interface presupposes but does not fully equip.\nA Fifty-Year-Old Promise The contract embedded in these interfaces is not new. Edgar F. Codd articulated it in 1970, in the paper that founded the relational model.3 One of its central principles was data independence: the idea that queries should be written against a logical model, entirely insulated from the physical layout of the data. The user thinks in relations. The system handles storage, access paths, and execution strategy.\nIt was a radical and largely successful idea. The cost-based optimizer that Selinger and her colleagues built for IBM’s System R in 1979 was its practical realization:4 given a declarative SQL query, the system would enumerate possible physical execution plans and choose the cheapest one. The user never needed to know which plan was chosen.\nBut the System R team soon recognized a tension they could not quite resolve. If the physical layer is fully hidden, users cannot understand why queries are slow. They cannot diagnose anomalies. They cannot make informed decisions about schema design or query structure. The solution (EXPLAIN, and later EXPLAIN ANALYZE) was already an admission: pure logical transparency is not enough. At some point, the physical layer has to become visible.\nSpark, Delta Lake, and container runtimes have re-enacted this same drama at a different scale and with a fundamentally different physical architecture. The promise is Codd’s promise, restated fifty years later. The tension is the same tension. What has changed is the distance between the logical interface and the physical reality. And that distance, in distributed systems, is vast.\nThe Interface as Sign Consider df.join(other, on="key"). The syntax denotes a logical operation: a relational join with clear set-theoretic semantics. But what the syntax produces in the mind of an experienced engineer is something richer: an expectation of predictable cost, transparent execution, and uniform scaling. In Peirce’s triadic model of the sign, where every sign consists of a representamen (the sign-vehicle itself), an object (what the sign stands for), and an interpretant (the meaning the sign produces in its interpreter), this accumulated expectation is the interpretant: not just the formal definition of the operation but the practical assumptions carried forward from prior systems. The practitioner brings to the interface not just its definition but the full weight of past experience with databases and dataframe APIs that behaved consistently.\nThe interpretant is legitimate: the natural product of a sign that activates certain encyclopedic competencies and leaves others dormant. The problem lies in the reader the interface presupposes: a reader whose competencies cover the ordinary case, and who is systematically unequipped for the cases that matter most.\nThe Model Reader Eco’s concept of the Model Reader gives this observation its theoretical form.5 Every text, Eco argued, presupposes a model reader: a specific set of competencies, knowledge, and interpretive habits. The text inscribes a particular reader into its structure. The model reader is not a real person but a textual strategy: the set of competencies the text assumes in order to function as intended.\nSoftware interfaces do the same thing. The designers of Spark’s DataFrame API presupposed a model user: someone who thinks in terms of logical transformations, who trusts the optimizer to make good decisions. That model user is well-served by the API in most cases.\nBut data systems at scale require a different reader: someone who understands file layout, partition statistics, shuffle cost, and optimizer visibility. That reader was not presupposed by the interface. They had to become that reader themselves, through failures and post-mortems and the gradual accumulation of knowledge that no documentation quite captures.\nEncyclopedia, Dictionary, Breakdown What determines the competencies the model reader carries? Eco called it the encyclopedia:6 the open, rhizomatic web of knowledge, assumptions, and inferential pathways through which a reader interprets any sign. Meaning, for Eco, is never self-contained: it lives in a network of connections that the reader activates. When that network is simplified, its connections pruned, its ambiguities resolved into univocal definitions, the encyclopedia degenerates into a dictionary.\nAn API is a dictionary. It takes the encyclopedic complexity of a system (its physical behaviors, its edge cases, its failure modes) and compresses it into a closed set of operations with precise, minimal definitions. But the model reader presupposed by that interface still carries an encyclopedia, whose rhizomatic branches the dictionary has anaesthetized but never eliminated.\nThe failures described here arise at those branches. Consider SELECT * FROM orders WHERE status = 'pending'. The dictionary entry is precise: filter rows by predicate, return all columns. But the reader’s encyclopedia fills in unstated assumptions: that an index exists, that the scan will be efficient, that “all columns” is cheap. Whether the query triggers an index lookup or a full table scan, whether it reads from cache or from disk, whether “all columns” means five fields or fifty including serialized blobs, none of this is encoded in the syntax. The representamen is identical across all these cases. The physical cost varies by orders of magnitude. This is not a documentation failure but a semiotic one: the interface presupposes a reader whose encyclopedia includes the ordinary case but not its pathological edges.\nThe dictionary\/encyclopedia distinction explains the structure of the gap. But it does not yet name the moment when the gap becomes visible to the practitioner. Terry Winograd and Fernando Flores, drawing on Heidegger’s analysis of tools and transparency, gave that moment a precise name: breakdown.7 A system functions smoothly as long as the user and the system share an implicit interpretive horizon. When that horizon fractures, the user is forced to confront the machinery they had been trusting invisibly. The breakdown is not a system failure. It is a failure of the shared interpretive frame.\nThis is exactly what happens when a Spark job runs for six hours instead of twenty minutes, or when a MERGE rewrites a terabyte of data to update a thousand rows. The system has not failed. The interpretive frame has. The engineer must reconstruct, from query plans and executor logs, a physical model the interface was designed to hide.\nThe gap between the reader the interface assumes and the reader the system actually requires is a central source of performance pathology in modern data engineering.\nFour Fracture Points Each of the following cases shares a structure: an interface that activates the wrong expectation, a physical reality that diverges from it, and a failure mode that becomes predictable once the divergence is named.\nSpark and the compiler you do not see The phrase “Spark is lazy” is the dictionary entry. The encyclopedic reading it activates is: execution is deferred, but the cost of execution is proportional to what you wrote. In reality, Spark functions as a dataflow compiler. Between your code and physical execution sits Catalyst, which rewrites and optimizes the logical plan in ways that are invisible to the caller. The model reader presupposed by “lazy evaluation” is someone who expects deferred but proportional execution. That reader is not wrong, but they are unequipped for the moment when the optimizer’s decisions depend on information that may or may not be available at plan time.8\nDelta Lake MERGE and the upsert that rewrites everything MERGE INTO looks like a row-level operation. The dictionary entry says: match rows, update or insert as appropriate. The encyclopedic reading is: only affected rows are touched. Partition pruning and data skipping can narrow the candidate set substantially, but the real constraint is that the storage layer operates at file granularity: once a file contains a modified row, the entire file must be rewritten. The result is that MERGE often rewrites far more data than the logical operation requires. The representamen evokes surgical precision. The object is a sledgehammer.\nThe skew that statistics cannot see The standard model of data skew (a small number of keys that appear disproportionately often) is encoded in every explanation of the problem and in the catalog statistics that optimizers use to detect it. But skew can also arise from keys that appear exactly once but carry payloads of radically different sizes: payload skew, as opposed to key frequency skew. This form of skew is invisible to cardinality statistics. No interface signals the difference, and no optimizer fully compensates for it: adaptive query execution can detect oversized partitions after the shuffle, but the interface provides no signal in advance, and the underlying statistics remain blind to the cause.\nThis case has the same structure as the others, but displaced one level. The dictionary here is not the operation itself but the statistics that describe the data: cardinality counts, partition sizes, histogram buckets. That dictionary presents itself as a complete account of distribution. It has anaesthetized the encyclopedic branch that matters most: payload variance. The reader whose encyclopedia includes only the dictionary’s entries sees sufficient information where there is a critical gap. Eco defined semiotics as the discipline that studies “everything which can be used in order to lie.”9 The statistics do not merely omit: they signify completeness where completeness does not hold. In this precise sense, the dictionary lies.10\nContainers and the isolation that is not there “Container” is the most semiotic term in modern infrastructure. The word activates a powerful interpretant of strong isolation, reinforced not just by the metaphor (the shipping container, the sealed box) but by the operational interface: docker run produces what looks like a separate machine, with its own filesystem, process tree, and network stack. The physical reality is simpler: Linux containers share a kernel. They are namespace and cgroup boundaries, not trust boundaries. A container escape exploits a kernel vulnerability that exists precisely because the isolation the interface signifies is not the isolation the kernel provides. The interface presupposes a model reader who is building a deployment pipeline. It does not presuppose the model reader who is running untrusted code. This is not a failure of containers. It is a semiotic mismatch between the metaphor the interface evokes and the guarantees the kernel actually provides.\nWhy Interfaces Are Designed This Way The logical transparency that Spark, Delta, and container runtimes offer is genuinely valuable: the same value Codd identified in 1970. Data independence allows practitioners to reason about complex systems without holding the entire physical implementation in their heads. The model reader these interfaces presuppose is not wrong; it is simply incomplete. And that incompleteness concentrates at the boundaries of the ordinary case: at scale, under skew, in adversarial environments. The interface is most misleading precisely where the stakes are highest.\nIn classical relational systems, the logical-physical distance was relatively contained (a handful of join algorithms, a single node, stable statistics), and failure modes tended to be gradual. In modern distributed systems, that distance is vastly larger: between a logical operation and its execution sit optimizers, adaptive planners, distributed schedulers, shuffle mechanisms, storage layouts, and incomplete statistics. The abstraction does not degrade smoothly. It holds, until it doesn’t. A Spark job runs reliably for months, then becomes ten times slower because the data distribution crossed a threshold the planner had not anticipated. The competencies required to diagnose this (the conditions under which AQE intervenes, the difference between logical and physical partitioning, the cost structure of shuffles) are not encoded in any interface. They are learned through production incidents and the slow reading of query plans.\nThe semiotic reading does not make these problems easier to fix. An experienced systems engineer will recognize every failure mode described here without needing Peirce to explain them.\nWhat the framework provides is not diagnosis but taxonomy: a structural account of why the same class of failure recurs across systems that share nothing except the decision to separate the logical interface from the physical reality. That is a modest claim, and I think a correct one.\nPostscript I studied semiotics under Umberto Eco at the University of Bologna. I work on distributed data systems. The connection between these two bodies of work did not occur to me for a long time. When it did, it seemed obvious, not as a metaphor, but as a structural analogy. The problems share the same architecture. The vocabulary is different.\nIf you have found the semiotic framing useful, the primary texts are Eco’s A Theory of Semiotics (1976) and The Role of the Reader (1979). Peirce’s collected papers are harder going, but the triadic sign model is accessible through secondary literature. Winograd and Flores’s Understanding Computers and Cognition (1986) is the most direct bridge between this tradition and software systems. It remains underread in engineering circles and does not deserve to be.\nFor the historical background on data independence and the optimizer, Codd\u2019s 1970 paper is short and still worth reading directly. Selinger et al.\u2019s 1979 paper on System R is where the logical\/physical tension becomes fully visible as an engineering problem. The connection to compiler-style optimization becomes particularly clear in these database papers, where equivalent logical expressions are explicitly mapped to alternative physical execution plans.\nJoel Spolsky, “The Law of Leaky Abstractions,” 2002. ↩︎\n“Leaky abstraction” names the eventual surfacing of hidden implementation details. The semiotic account proposed here is narrower: it explains why certain interfaces systematically produce the wrong practical expectations even when their formal semantics are correct. ↩︎\nEdgar F. Codd, “A Relational Model of Data for Large Shared Data Banks,” Communications of the ACM 13, no. 6 (1970). ↩︎\nPatricia Selinger et al., “Access Path Selection in a Relational Database Management System,” Proceedings of the 1979 ACM SIGMOD International Conference on Management of Data. ↩︎\nUmberto Eco, The Role of the Reader: Explorations in the Semiotics of Texts (Bloomington: Indiana University Press, 1979). ↩︎\nUmberto Eco, A Theory of Semiotics (Bloomington: Indiana University Press, 1976). The dictionary\/encyclopedia distinction is developed further in Semiotics and the Philosophy of Language (1984). ↩︎\nTerry Winograd and Fernando Flores, Understanding Computers and Cognition: A New Foundation for Design (Norwood, NJ: Ablex, 1986). The concept of breakdown draws on Heidegger\u2019s distinction between ready-to-hand (Zuhandenheit) and present-at-hand (Vorhandenheit) in Being and Time. ↩︎\nFor the full technical analysis, see Spark Is Not Just Lazy. Spark Compiles Dataflow. ↩︎\nUmberto Eco, A Theory of Semiotics (1976), \u00a70.1.3: “Semiotics is in principle the discipline studying everything which can be used in order to lie. If something cannot be used to tell a lie, conversely it cannot be used to tell the truth: it cannot in fact be used ’to tell’ at all.” ↩︎\nSee Fixing Skewed Nested Joins in Spark with Asymmetric Salting ↩︎\n","permalink":"https:\/\/cdelmonte.dev\/essays\/where-data-system-abstractions-break-semiotic-reading\/","summary":"Many of the most surprising performance pathologies in modern data systems are semiotic failures: structural divergences between what an interface signifies and what the underlying system does.","title":"Where Data System Abstractions Break: A Semiotic Reading"},{"content":"The question behind the number Explaining Delta Lake pruning sounds like a counting problem. Given a table and a WHERE predicate, how many files can be eliminated before the engine opens the table’s data files? Delta Lake answers that question through two metadata mechanisms: partition pruning drops whole directories from partition values declared in the transaction log; data skipping drops individual files from per-file min\/max statistics carried in the same log.\nFor a user-facing tool, that number matters. It tells whether the physical layout and the predicate are doing useful work. But for a tool that claims to explain pruning, the harder problem is not the count. The harder problem is the chain of trust behind the count. Which expression was actually measured? Which parts of it were rewritten? Which fragments could be decided from partition values, which from file statistics, and which had to be left out? When a file survives, is that a proof that it may match, or only the consequence of an intentionally conservative approximation?\nThat is the problem delta-explain is built around. It is deliberately smaller than a query planner, but it still needs some of the same machinery: a predicate representation that all consumers share, a normalization pass whose rewrites preserve SQL semantics, a classifier that separates what different metadata domains can decide, and a measurement procedure that compares survivor sets across controlled scans of one snapshot.\nThe result is a small compiler pointed at an unusual target. It does not produce machine code, and it does not produce an execution plan. It produces a pruning explanation with bounded claims: exact where the input is concrete, conservative where the input is summarized, and incomplete where the predicate cannot be assigned safely.\nThis piece follows one predicate through that pipeline, then stops at the four design questions the walk raises: who owns the predicate’s representation; when a rewrite may be admitted; what conservatism actually is; and how to measure an optimizer from the outside.\nScope. This piece is about the design model and its reasons, not the tool in use. The companion piece, delta-explain: Making Delta Lake Pruning Visible, covers output format, worked examples, and CI gates. The documentation holds the contract and the reference material.\nOne predicate through the pipeline The predicate we follow through the pipeline is deliberately the awkward form of a plain filter:\nNOT (pickup_date != '2024-01-03' OR trip_distance <= 5) It has the same meaning as pickup_date = '2024-01-03' AND trip_distance > 5, but it is written the way a generated query or a hurried refactor might write it. That makes it useful. It forces the tool through the whole chain: parse the SQL, import it into its own predicate representation, normalize it, classify it, lower it, and measure it. A simpler predicate would show the result, but not the machinery behind it.\nRun against a real NYC-taxi table, with five daily pickup_date partitions and four files each, the predicate exists in five forms during a single invocation. The table is the fixture behind the validation repo, and every form quoted below is a real dump, not a reconstruction.1\n1. The owned AST, before normalization. The SQL parser first turns the WHERE string into its own source tree. delta-explain then imports that tree into its owned predicate AST. This is the representation used by the rest of the pipeline. Its vocabulary is deliberately narrower than SQL and closer to pruning itself: column-op-literal comparisons, junctions, null checks, IN, and BETWEEN. The tree has been translated out of the parser’s vocabulary into delta-explain’s own, but its logical shape is untouched. The NOT (... OR ...) the user wrote is still a NOT over an OR. Normalization, the first pass that changes shape, comes next:\nrendered: NOT (pickup_date <> '2024-01-03' OR trip_distance <= 5) Not(Or([ Cmp { col: ColRef(["pickup_date"]), op: Ne, lit: Str("2024-01-03") }, Cmp { col: ColRef(["trip_distance"]), op: Le, lit: Number("5") }, ])) The literal is worth noticing. Number("5") is still the raw token, not a typed value. That is because the owned AST is built for reading and classification, not for evaluating the predicate against a table schema. Literal typing happens later, when the predicate is lowered into the form the kernel can evaluate.\n2. The owned AST, normalized. The normalization pass now applies De Morgan’s law, pushes the negation to the leaves, and flips the comparison operators on the way:\nrendered: pickup_date = '2024-01-03' AND trip_distance > 5 And([ Cmp { col: ColRef(["pickup_date"]), op: Eq, lit: Str("2024-01-03") }, Cmp { col: ColRef(["trip_distance"]), op: Gt, lit: Number("5") }, ]) Ne became Eq, Le became Gt, and Not and Or are gone from the tree entirely. This rewrite is sound but not neutral. Because it preserves SQL’s three-valued logic, it can never change which files survive. It can only change how the pruning is explained, for example by making one fragment visible to one phase and another fragment visible to another. That is the license normalization needs, and its limit.\n3. The classification. Once the predicate has this shape, the analyzer can walk the normalized conjuncts and route each fragment by the metadata domain that can decide it. The four buckets are the vocabulary of that routing. partition_safe is for fragments decidable from partition values through the kernel. partition_exact is for fragments decidable from partition values but outside the kernel’s grammar. stats_safe is for fragments decidable from file statistics. unsplittable is for fragments that span domains or leave the pruning language altogether.\nFor this predicate, the classification is clean:\npartition_safe: Some("pickup_date = '2024-01-03'") partition_exact: None stats_safe: Some("trip_distance > 5") unsplittable: None confidence: Conservative A comparison on pickup_date, the partition column, can be used in the partition phase. A comparison on trip_distance, a non-partition column with file statistics, belongs to the statistics phase. Nothing has to be left aside. This classification is already a small explanation: it says which part of the predicate can be checked from partition values, which part can be checked from file statistics, and whether any fragment has fallen outside the pruning language.\n4. The kernel predicates, one per phase scan. The classified fragments are then lowered into delta-kernel’s Predicate IR. The partition-only scan receives the partition-safe fragment. The full scan receives the predicate that the kernel can evaluate after normalization and after unsupported fragments have been stripped. This is the full-scan lowering:\nJunction(JunctionPredicate { op: And, preds: [ Binary(BinaryPredicate { op: Equal, left: Column(ColumnName { path: ["pickup_date"] }), right: Literal(String("2024-01-03")) }), Binary(BinaryPredicate { op: GreaterThan, left: Column(ColumnName { path: ["trip_distance"] }), right: Literal(Double(5.0)) }), ]}) The change in vocabulary is deliberate. Eq became Equal, Cmp became BinaryPredicate, and the literal that was Number("5") upstream is now Double(5.0), coerced against trip_distance’s schema type at lowering. The same schema-driven coercion is what lets a quoted '2024-01-01' compared to a DATE column prune as a date.\nBoth forms derive from the same owned tree, so the explanation the analyzer renders and the scan the kernel runs cannot silently refer to different predicates.\n5. The survivor sets. The scans over one snapshot close the loop:\nbaseline: 20 files partition-only scan: 4 files full scan: 2 files The first drop, from the baseline to the partition-only scan, eliminates 16 files and is credited to partition pruning. The second drop, from the partition-only scan to the full scan, eliminates 2 more files and is credited to data skipping. That day holds four files laid out by distance, and the two whose trip_distance max is <= 5 cannot match the predicate.\nThe measurement works by subtraction. Each scan adds one layer of predicate information to the previous scan. The difference between two adjacent survivor sets is the contribution of the layer that was added.\nThe same pipeline also shows what happens when the predicate leaves the pruning language. In UPPER(pickup_date) = '2024-01-03' AND trip_distance > 8, the UPPER(pickup_date) = '2024-01-03' fragment classifies as unsplittable and never reaches lowering. The dump says it in one line:\nscan predicate after stripping unsupported fragments: trip_distance > 8 The unsupported fragment survives only in the report, as an UNSUPPORTED_EXPRESSION warning and a degraded confidence, while the kernel scans with what remains: 20 files to 7, on the same table. The result is not a failed explanation, but a bounded one. The tool reports the fragment it could measure, keeps the fragment it could not measure visible, and degrades the confidence instead of pretending that the whole predicate contributed to pruning.\nThe walk is complete: one string, five forms, two numbers. It is also a compressed list of design decisions. The same owned tree feeds several consumers. The first transformation rewrote the user’s predicate and needed a semantic license. One phase reasoned from concrete partition values while another reasoned from file statistics. And the two numbers came from metadata scans built for measurement rather than from an engine’s execution plan.\nThe four sections that follow explain those decisions one at a time.\nWho owns the representation? Every system that reads the same expression in more than one way faces the same choice: a shared representation with many consumers, or a private representation per consumer. Compilers settled this long ago in favor of a common IR, and not for elegance. Two representations of the same program do not usually fail loudly. They drift apart one bugfix at a time.\ndelta-explain is a small instance of that decision. sqlparser-rs parses the WHERE string and produces the parser’s source tree. delta-explain immediately converts that tree into its owned predicate AST, the vocabulary seen in the walk above. From that point on, every internal consumer reads the same owned tree.\nThat boundary matters because delta-kernel-rs, the Rust implementation of the Delta protocol, is the only door to the table. It replays the transaction log, checkpoint Parquet included, and runs the metadata scans. To ask the kernel a pruning question, delta-explain must lower its owned AST into the kernel’s Predicate IR. But the lowering is only one consumer. The analyzer reads the same owned AST and produces the classification shown in the report. The kernel bridge reads it and produces the predicates evaluated by the scans. The partition evaluator reads it again and decides what can be decided from partition literals when the kernel grammar cannot express the fragment. Three consumers, not three parsers.\nThat invariant is what makes the survivor-count arithmetic meaningful. Each path contributes a different part of the explanation: the analyzer names the fragments, the kernel scan produces the survivor sets, and the partition evaluator contributes exact decisions over partition literals. If those paths owned different predicate models, they could drift silently. The numbers would still look plausible, but the report could credit a fragment that was not exactly the one scanned or evaluated.\nOne shared tree prevents that class of error by construction. ADR 0001 records the decision: the parser’s own types are quarantined in a single conversion module, and every other consumer sees only the owned tree. No module gets to smuggle in a second, subtly different view of the predicate.\nWhen is a rewrite admissible? The second decision in the walk is the most consequential one: the tool rewrote the user’s predicate before measuring it. Optimizers do this constantly. Canonicalization is one of the first things a query planner does to a filter. It is also where planners sometimes go wrong quietly, because every rewrite is a claim that two expressions mean the same thing.\ndelta-explain follows a narrower rule: a rewrite may change how pruning is explained, but not which files survive. To make that safe, every admitted rewrite must preserve SQL’s three-valued semantics. De Morgan pushdown and small boolean simplifications are the routine cases. The instructive one is LIKE.\npickup_date LIKE '2024-01-0%' is not a comparison the kernel can lower. But when the pattern is a literal prefix, with no wildcards before the trailing % and no escapes, and the column is string-typed, it can be rewritten as a half-open range:\npickup_date >= '2024-01-0' AND pickup_date < '2024-01-1' Under the same lexicographic code-point ordering the metadata evaluator uses for partition values and file statistics, every string with that prefix is at or above the lower bound, and no string with that prefix reaches the successor prefix. So the normalization pass turns the Like node into two comparisons before classification ever sees it:\n== owned AST (before normalization) == rendered: pickup_date LIKE '2024-01-0%' AND trip_distance > 8 And([ Like { col: ColRef(["pickup_date"]), pattern: "2024-01-0%", negated: false }, ... ]) == owned AST (normalized) == rendered: pickup_date >= '2024-01-0' AND pickup_date < '2024-01-1' AND trip_distance > 8 And([ Cmp { col: ColRef(["pickup_date"]), op: Ge, lit: Str("2024-01-0") }, Cmp { col: ColRef(["pickup_date"]), op: Lt, lit: Str("2024-01-1") }, ... ]) The gate is soundness, not convenience. The rewrite fires only on a string-typed column, because the successor step is a fact about string order, not about numbers or dates. Where it does fire, the result is inside the pruning language. On the taxi table, the LIKE form scans 20 to 20 to 7, identical to its hand-written range form. A construct is admitted only when the rewrite is provably equivalent to the original predicate. Everything else degrades. ADR 0005 records why LIKE is a first-class node in the shared AST, extended once in the normalization pass, rather than something special-cased outside it.\nThe same rule explains why the tool refuses broader normal forms. It does not normalize arbitrary predicates into CNF or DNF, the way a textbook transformation would. Those forms can change the shape of what the user wrote, and they can grow exponentially. The price of refusing is smaller than the risk: a top-level OR stays whole, and the measurement section returns to that cost.\nLiteral typing follows the same discipline. A quoted '2026-07-02' against a DATE column is coerced and can prune as a date. But a DECIMAL literal that exceeds the column’s scale is rejected rather than silently rounded, and an offset-bearing literal against a wall-clock TIMESTAMP_NTZ is rejected as ambiguous rather than silently normalized. This is not fussiness. A mistyped literal can reach the kernel as a type mismatch; the conservative result is to keep every file. On a date-partitioned table, that can turn an apparently selective predicate into a report with zero pruning, exactly where pruning matters most.\nWhere equivalence cannot be proven, the honest moves are refusal or degradation, never a guess.\nConservatism is a property of the domain The walk left an asymmetry unexplained: partition pruning came out exact, while data skipping came out conservative, on the same predicate over the same table. The labels do not measure how carefully either mechanism is implemented. They follow from the kind of information each mechanism reasons from.\nData skipping evaluates the predicate over each file’s min\/max statistics. This is abstract interpretation in miniature. A range like trip_distance: 1.86..3.75 is not the file’s data. It is a summary of values the file may contain. Evaluating trip_distance > 8 over that range therefore answers a different question from the user’s query. It does not ask whether a matching row exists. It asks whether a matching row can be ruled out.\nWhen the summary proves that no row can match, the file can be dropped. That decision is a proof. When the summary still leaves a match possible, the file must survive. That decision is prudence, not evidence that a matching row is actually present. A file whose trip_distance range is 4..12 survives trip_distance = 5, even though it may contain no row where trip_distance is exactly 5. Over-approximation is the price of deciding from summaries. From min\/max statistics alone, the tradeoff is unavoidable: keep every file that may match, and drop only files that cannot match.\nPartition values live in a different domain. pickup_date = 2024-01-03 is not a summary of a file’s rows. For that partition column, it is the value every row in the file carries. Evaluation is concrete, so it can be exact. The partition_exact bucket exists to exploit that fact for fragments the kernel cannot express.\nFor example, pickup_date LIKE '%01-03%' is outside the kernel’s grammar and outside the prefix rewrite. A substring pattern is not a range. But against the partition literal '2024-01-03', the fragment is still decidable. The tool evaluates it itself, using the partition evaluator as a third interpreter over the shared AST.\nThat evaluator uses four outcomes: TRUE, FALSE, NULL, and UNKNOWN. TRUE keeps the file. FALSE and NULL drop it exactly, because the fragment is constant across the file and SQL keeps rows only when the predicate evaluates to true. UNKNOWN means something different: the evaluator does not know how to decide the fragment, so it keeps the file conservatively.\nOn the taxi table, pickup_date LIKE '%01-03%' AND trip_distance > 5 prunes 20 files to 4, then 4 to 2, the same as the equality form. The kernel could not have lowered the LIKE, but the partition evaluator could decide it from the literal partition value. ADR 0006 records the choice to grow this evaluator over the shared AST rather than reach for a second parser.\nRead this way, the confidence labels stop being adjectives and become consequences. A phase whose inputs are concrete values is exact. A phase whose inputs are summaries is conservative. Incomplete marks the third case: part of the predicate could not be assigned safely to either domain, so its contribution cannot be separated. The report is not describing effort. It is describing domains.\nMeasuring an optimizer from outside The last question is the one the tool exists for. An execution plan shows the fused result of pruning: one scan, one surviving file list, the elimination already folded in. To explain the result, a measurement has to un-fuse what the engine fuses, and the standard experimental move applies: hold everything constant, vary one input, difference the outcomes.\ndelta-explain’s version is three nested metadata scans over one snapshot: a baseline with no predicate, a partition-only scan with the partition fragments, and a full scan with everything the kernel can evaluate. Each scan adds constraints to the one before and removes none, so the survivor sets are nested. A file is eliminated at exactly one point along the chain, and the drop between adjacent scans is credited to the mechanism introduced at that point.\nThe single snapshot matters for the same reason a controlled experiment controls: every scan reads the same table state, so no concurrent commit can shift a count between measurements. And the measurement never touches what it measures at the data level. The kernel’s metadata path resolves the file list and the per-file statistics without opening a single table data file.2 That is what makes the numbers engine-independent, and it also makes the cost of the analysis proportional to the log rather than to the data. The same replay carries each file’s raw statistics, so the verbose per-file view and the --assert-stats check read the exact scan that produced the counts. There is no second source to disagree with the first.\nWhat cannot be measured is declared. A top-level OR that mixes partition and non-partition columns stays whole, which is the price of refusing CNF. It reaches the full scan and can still prune there: pickup_date = '2024-01-03' OR trip_distance > 30 still scans 20 to 6 on the taxi table. But the subtraction can no longer separate the mechanisms, so the phase and the run degrade to incomplete. The fragment costs the explanation, not necessarily the pruning.\nEven then, the output keeps the boundary visible. A degraded phase line does not print the full predicate as if it all pruned. It shows the honored fragment and annotates what was stripped, trip_distance > 8 (+1 unsupported fragment, keeps all files), so the count and the predicate credited for it always agree.\nThe guarantee Everything above serves one invariant: the survivor set is a superset of the files that contain matching rows. Rewrites may reshape fragments, classification may leave attribution incomplete, and unsupported constructs may drop out of the scan predicate, but every move errs in the same direction: keeping more files, never fewer.\nThat is the boundary that makes the explanation usable. A file that is dropped has been ruled out by metadata. A file that survives has not been proven relevant; it has only not been proven impossible. The distinction is the whole contract.\nThat is what makes a log-only explanation possible: uncertainty may keep files, but it may never eliminate them.\nReferences delta-explain, semantics contract: conservative survivor sets, degradation rules, exit codes, and the versioned JSON schema. docs\/semantics.md delta-explain-validation: an independent re-check of every claim in this piece against the same taxi table, down to a plain-Python replay of the Delta log that reproduces the tool’s kept set file by file. repository Christian Del Monte. delta-explain: Making Delta Lake Pruning Visible. The companion piece, covering output format, worked examples, and CI gates. cdelmonte.dev Produced by a diagnostic build of the tool (cargo run --features debug-ir -- <table> -w "<predicate>" --debug-ir ir.txt); whitespace is compacted for the page, tokens are verbatim. validate.py --layers in the validation repo regenerates the exact walk. ↩︎\nThe exact identifier names in the kernel API are not stable across delta-kernel-rs minor versions; delta-explain’s behavior reflects the version pinned to a 0.x minor in its Cargo.toml, per Cargo’s standard SemVer rules. ↩︎\n","permalink":"https:\/\/cdelmonte.dev\/essays\/a-small-compiler-for-explaining-delta-lake-pruning\/","summary":"How delta-explain turns one predicate and the Delta log into a trustworthy explanation of file pruning, through shared representation, safe rewrites, and conservative measurement.","title":"A Small Compiler for Explaining Delta Lake Pruning"},{"content":"What defines a Spark SQL\/DataFrame job’s performance is not when execution happens, but what the optimizer can see.\nIn this article, \u201cSpark\u201d refers primarily to Spark SQL and the DataFrame\/Dataset API. RDDs use the same underlying Spark runtime and scheduler, but their transformations are not represented as Catalyst logical plans and are therefore not optimized by Catalyst.\nThree Kinds of \u201cLazy\u201d The word \u201clazy\u201d is used to describe at least three different ideas. They are related, but they are not the same thing, and the difference matters.\nLazy evaluation is a semantic property of a language: values are not computed until they are needed. Haskell is the canonical example. Each computation is local, demand-driven, and memoized: a thunk fires when its result is required, not before, and once evaluated, is not recomputed.\nDeferred execution is a property of an API. Operations are recorded but not executed immediately. Java\u2019s Stream API works this way: intermediate operations like filter and map are registered, and execution begins only when a terminal operation like collect is invoked.\nDAG compilation (a term used here to distinguish this property from the previous two, not a standard term in compiler literature) is a property of the execution model. Operations are collected, analyzed, reordered, rewritten, and optimized before execution begins. The entire query plan is visible and subject to whole-plan optimization. SQL query engines follow this model: a query planner rewrites and optimizes a query before running it.\nThe discriminating criterion is scope. In lazy evaluation, each thunk is local and demand-driven. In DAG compilation, the system sees the entire plan at once and rewrites it globally. Lazy evaluation implies deferral by its very nature; DAG compilation requires deferred execution as a prerequisite. But deferred execution alone says nothing about what happens between declaration and execution.\nWhat Spark Actually Does Spark SQL’s DataFrame\/Dataset API combines deferred execution with whole-plan analysis and optimization. At the API level, transformations are recorded. At runtime, Catalyst (Spark’s optimizer framework, which this article treats as a query compilation pipeline) analyzes the logical plan, optimizes it, produces a physical plan, and hands it to the scheduler. The fact that execution is deferred is not the point; it is a consequence of global compilation. Spark delays execution to enable whole-plan optimization.\nThe pattern is not unique to Spark. No one calls LLVM \u201clazy\u201d because it transforms source code into IR, optimizes it, and emits machine code. The structural property Spark shares with LLVM is that a complete representation of the computation is available before execution begins, and an optimization phase rewrites that representation globally. The domains differ (Catalyst operates on relational algebra, LLVM on SSA-form IR; one optimizes data access patterns, the other optimizes machine code), but the high-level architecture is the same: declare, then compile, then execute. Without code generation, this would make Catalyst a query optimizer with a compiler-like architecture, a meaningful but incomplete analogy. Tungsten’s Whole-Stage Code Generation closes part of that gap: codegen-compatible regions of the physical plan are fused into generated Java code and compiled at runtime via Janino, making the compiler analogy not merely structural but, for those execution regions, literal. Spark is closer to LLVM than to Haskell.\nThe compiler model also explains failures that the \u201clazy\u201d framing cannot.\nWhere the Compiler Model Bites Consider a simple filter-then-aggregate pipeline:\ndf = spark.read.parquet("events\/") result = df.filter(col("country") == "IT").groupBy("city").count() result.show() Catalyst pushes the filter down into the Parquet scan: the physical plan reads only the required columns and hands the predicate to the Parquet reader, allowing the reader to skip data that can be ruled out from its metadata. This is predicate pushdown, and it happens because Catalyst can see the entire plan before execution begins.\nNow consider the same logic, routed through a Python UDF:\nis_italian = udf(lambda c: c == "IT", BooleanType()) result = df.filter(is_italian(col("country"))).groupBy("city").count() result.show() Functionally equivalent. But Catalyst cannot see inside the UDF. It is an opaque box. The predicate is not pushed down, so it can no longer participate in Parquet-level filtering. Unless partition pruning or other pushable predicates reduce the input independently, the whole candidate set (though only the needed columns) has to reach Spark’s UDF evaluation path: each row is serialized to Python, the function is evaluated, and the result is sent back. You can verify this with explain():\n# Native expression == Physical Plan == *(2) HashAggregate(keys=[city], functions=[count(1)]) +- *(2) HashAggregate(keys=[city], functions=[partial_count(1)]) +- *(1) Filter (country = IT) +- *(1) ColumnarToRow +- FileScan parquet [city,country] PushedFilters: [EqualTo(country,IT)] # UDF == Physical Plan == *(2) HashAggregate(keys=[city], functions=[count(1)]) +- *(2) HashAggregate(keys=[city], functions=[partial_count(1)]) +- *(1) Filter UDF(country) +- *(1) ColumnarToRow +- FileScan parquet [city,country] PushedFilters: [] Notice PushedFilters. In the first plan, the predicate reaches the storage layer. In the second, it does not. The \u201clazy\u201d model cannot explain this difference: both plans defer execution equally. The compiler model explains it directly: Catalyst optimizes what it can see. A UDF is opaque; a native expression is transparent.\nThis is not specific to Python. Scala UDFs are equally opaque to Catalyst. The issue is not the language; it is that any UDF is a ScalaUDF or PythonUDF node in the expression tree, not a native Catalyst expression, and therefore not rewritable.\nIf you must use a UDF, prefer pandas_udf. Catalyst still cannot push predicates through it, but Arrow-based batching significantly reduces the per-row serialization overhead that makes scalar Python UDFs expensive at scale. The optimizer limitation remains; the serialization cost changes shape.\nNested structures create a subtler blind spot. Suppose your Parquet schema has a struct column:\nroot |-- event_id: string |-- address: struct | |-- country: string | |-- city: string |-- timestamp: long You filter on the nested field:\ndf = spark.read.parquet("events\/") df.filter(col("address.country") == "IT").select("event_id", "address.city") This looks entirely native. No UDF, no opaque function, just dot notation on a Column. Catalyst resolves it, places it in the logical plan, and optimizes around it. But predicate pushdown to the storage layer is a different matter.\nThere are two distinct optimizations at play, and they do not behave the same way on nested fields:\nNested column pruning works. Spark can project only the struct subfields it needs (address.country and address.city) rather than reading the entire address struct. This has been supported and enabled by default via spark.sql.optimizer.nestedSchemaPruning.enabled since Spark 2.4.1, and works reliably with Parquet.\nNested predicate pushdown has historically been more data-source-specific and less predictable than its flat-column equivalent. Whether the predicate address.country = "IT" reaches the Parquet reader depends on the data source implementation, the Spark version, and the predicate type. In earlier versions of Spark, simple nested predicates often did not reach the storage layer, and the filter would stay above the scan with PushedFilters: []. Modern Spark versions can translate simple predicates on nested struct fields into data-source filters when the underlying source supports nested predicate pushdown. More complex nesting and predicates remain data-source dependent.\nThe broader point remains: the logical plan is optimized in both cases, but what reaches the storage layer is a boundary between the optimizer and the data source API. Catalyst hands off a set of predicates; the data source decides which ones it can handle.\nUDFs and nested schemas are not wrong in themselves. What you give up when you use them is the compiler\u2019s optimization scope, wholly or partially. Sometimes that tradeoff is worth it. But you should know you are making it.\nCaching as Materialization Boundary The same perspective reframes caching. If you think in terms of lazy versus eager execution, cache() becomes a way to \u201cforce Spark to compute here.\u201d If you think in terms of compilation, caching becomes a materialization boundary in the dataflow graph.\nEach action generates a separate QueryExecution: a distinct logical plan, optimized plan, and physical plan. Catalyst optimizes within that boundary. It cannot optimize across actions. When two jobs share a common subgraph, Catalyst recomputes it independently for each. Caching materializes an intermediate result so that subsequent jobs do not recompute it from scratch.\nbase = spark.read.parquet("events\/").filter(col("year") == 2024) # Without cache: the filter + scan is compiled and executed twice italian = base.filter(col("country") == "IT").count() german = base.filter(col("country") == "DE").count() # With cache: the shared subgraph is materialized once base.cache() italian = base.filter(col("country") == "IT").count() german = base.filter(col("country") == "DE").count() This is analogous to an intermediate target in a build system. You introduce a materialization point where the plan structure requires it (where multiple downstream jobs depend on the same computed result), not where you feel the urge to force execution.\nNote that cache() is itself lazy: it marks a DataFrame for caching but does not trigger computation. The first action after cache() materializes the result. This reinforces the point: even the API for materialization follows the compiler model, not an eager one. One subtlety worth knowing: the cache manager matches by canonicalized plan structure, an intermediate level between syntactic identity and full semantic equivalence. Canonicalization normalizes attribute names and rewrites commutative expressions into a canonical form. So df.filter(col("x") > 1) and df.where(col("x") > 1) share the cache, and so do df.filter(col("x") > 1) and df.filter(lit(1) < col("x")), because canonicalization rewrites both to the same form. But more complex structural differences (such as reordering joins, combining filters differently, or adding an unnecessary withColumn that introduces a new projection node) can break cache sharing despite logical equivalence.\nThe corollary is unpersist(). If caching is a structural decision about where to split the plan, removing a cache entry is equally structural. In a long pipeline, forgetting to unpersist can saturate executor memory, a practical failure directly tied to treating caching as an afterthought rather than an architectural choice.\nMisplaced caching is a common symptom of the \u201clazy\u201d mental model. If Spark is a compiler, caching is a structural decision about where the plan should be split, not a way to control when it works.\nFilter Placement and Join Semantics Another area where the compiler model pays off is understanding how filter placement interacts with join semantics. Consider these two logically equivalent pipelines:\n# A: filter, then join orders_it = orders.filter(col("country") == "IT") result_a = orders_it.join(products, "product_id") # B: join, then filter joined = orders.join(products, "product_id") result_b = joined.filter(col("country") == "IT") For an inner join with a deterministic filter on a column from orders, Catalyst will produce the same physical plan for both: it pushes the filter before the join regardless of how you wrote it. This is the benefit of global compilation: the optimizer sees the whole plan, not the sequence of API calls.\nBut this equivalence is not universal. With a left outer join, naively pushing a filter on the nullable (right) side below the join would change semantics: it would eliminate rows before they get a chance to produce null-padded results. So Catalyst never pushes such a filter blindly. What it does instead is subtler. If the filter is null-rejecting (it evaluates to false or null whenever its inputs from the right side are null), the optimizer rule EliminateOuterJoin proves that no null-padded row could survive the filter anyway, rewrites the outer join as an inner join, and then pushes the filter as usual. If the filter is not null-rejecting, is non-deterministic, or Catalyst cannot prove its null-rejecting behavior (as can happen with opaque UDF expressions), the outer join cannot be eliminated this way. In that case, the distinction between filtering before and after the outer join remains semantically observable, and Catalyst cannot rewrite one form into the other.\nHere is a concrete case. Suppose you want all orders, with product details where available, but only for products currently in stock:\n# Left join: keep all orders, attach product info where it exists result = orders.join(products, "product_id", "left_outer") \\ .filter(col("in_stock") == True) This silently drops every order whose product is not in the products table, or whose product is out of stock. The in_stock column comes from products, the nullable (right) side of the join; rows with no match have in_stock = null, and the filter eliminates them. What was a left outer join now behaves like an inner join with an extra predicate.\nRun explain() on this query and you can watch the compiler reach the same conclusion: in_stock = true is null-rejecting, so EliminateOuterJoin rewrites the left outer join as an inner join and then pushes the filter into the scan of products. The optimized plan is efficient, and it is a faithful compilation of the query as written. The problem is that the query as written is not the query you meant. If you intended to filter before the join (selecting only in-stock products to join against, while keeping every order), you need to say so explicitly:\nin_stock_products = products.filter(col("in_stock") == True) result = orders.join(in_stock_products, "product_id", "left_outer") Now orders with no matching in-stock product still appear in the output, with null product columns, which is what a left join is supposed to do. Note that both versions end up with the filter pushed into the scan of products; what differs is the join that survives compilation. In the first version, the filter above the join let the optimizer eliminate the outer join entirely. In this one, the filter applies before the join, so the left outer join and its null-padded rows survive.\nThe optimizer did nothing wrong in either case. It compiled the semantics you wrote; only you know the semantics you meant.\nWhat Compilation Cannot Reach The same framework clarifies where Spark\u2019s optimization ends. Catalyst compiles and optimizes the logical plan of a single query because that plan is fully known before execution begins. You declare the entire dataflow, then trigger it. Within that query, you do not read intermediate results to dynamically decide the next step. Adaptive Query Execution (AQE, enabled by default since Spark 3.2 via spark.sql.adaptive.enabled) may refine the physical plan at runtime (coalescing post-shuffle partitions, switching join strategies based on actual data sizes, rewriting skew joins), but it still operates within the boundary of a single query plan. It is runtime refinement, not interactive replanning.\nWhere the flow is interactive and sequential (an OLTP transaction, a step-by-step notebook that inspects results and branches), global compilation is not possible. This is not a limitation of Spark\u2019s implementation; it is a structural constraint. A whole-program compiler needs the whole program. If you feed it one statement at a time, each statement is still compiled, but the optimization scope shrinks to a single step, and the opportunities for global optimization are significantly reduced.\nThis explains why chaining multiple actions in a loop, each depending on the result of the previous one, tends to produce suboptimal plans. It is not because Spark is slow; it is because the compiler scope is too narrow to optimize across iterations.\nThe compiler model also illuminates fault recovery. Spark’s RDD lineage (the chain of transformations from source to result) is a recomputable intermediate representation that the runtime preserves. When a partition is lost, Spark does not restore from a checkpoint by default; it re-executes the lineage for that partition. Caching is the explicit memoization point; without it, recomputation is the default. You opt in to materialization; everything else is recomputable by design.\nSpark Connect, introduced in Spark 3.4 and central in Spark 4.x, makes the separation between declaration and execution architecturally explicit: the client submits an unresolved logical plan over gRPC, and the server analyzes, optimizes, and executes it. The boundary is no longer just conceptual: it is a network call.\nCalling Spark lazy is not wrong. But it is reductive, and the reduction has a cost. The question is never \u201cwhen does Spark execute?\u201d It is always \u201cwhat can Catalyst see?\u201d\n","permalink":"https:\/\/cdelmonte.dev\/deep-dives\/spark-is-not-lazy-spark-compiles-dataflow\/","summary":"Why calling Spark ’lazy’ is technically reductive, and how thinking of it as a dataflow compiler changes the way you design pipelines.","title":"Spark Is Not Just Lazy. Spark Compiles Dataflow."},{"content":"Data skew in Spark is usually discussed in terms of key frequency: one key appears millions of times, its partition dwarfs the rest. But there is a second form: a key that appears only once, carrying a nested array of thousands or millions of records. The task assigned to that key must process the entire payload alone. This article examines asymmetric salting, a technique for this class of skew.\nA Metaphor Think of organizing package delivery across a city. Each building\u2019s deliveries are routed through a dedicated sorting depot, all packages for the same address grouped together. One day, a single box addressed to one building turns out to contain thousands of smaller packages inside. The building appears only once on the manifest, but the sheer volume overwhelms its depot.\nSpark works the same way. During a groupBy or join, Spark routes all records sharing the same key to the same shuffle partition via pmod(hash(key), numPartitions). If that key\u2019s payload is a deeply nested or large array, the entire partition becomes skewed and the task assigned to it becomes a bottleneck.\nThis form of skew manifests when the nested array crosses a shuffle boundary intact: when a join or aggregation forces redistribution by key before the nested structure has been expanded.\nWhy Adaptive Query Execution (AQE) Isn’t Enough Since Spark 3.0, AQE can detect oversized shuffle partitions (exceeding both skewedPartitionThresholdInBytes, default 256 MB, and skewedPartitionFactor \u00d7 median partition size, default 5\u00d7) and split them into subpartitions. This optimization is controlled by spark.sql.adaptive.skewJoin.enabled (default true since Spark 3.2) and applies only to SortMergeJoin and ShuffledHashJoin plans; it does not activate for BroadcastHashJoin. If an explode happens before the shuffle, AQE can in principle detect and split the resulting oversized partition.\nBut AQE’s thresholds operate on serialized shuffle output size (compressed by default; controlled by spark.shuffle.compress). A row containing a nested array of 10,000 structs may serialize to a few megabytes in a compressed shuffle block (well below AQE’s 256 MB threshold) while expanding to hundreds of megabytes or more in memory when the array is materialized during a collect_list or UDF evaluation. AQE sees an acceptable partition and assumes balanced workload. Asymmetric salting gives the developer explicit control over how the load is distributed, rather than relying on a heuristic that operates on the wrong metric for this class of skew.\nAsymmetric Salting The intuition is straightforward: if the array were exploded before the shuffle, the individual elements would be separate rows and could be distributed normally. This is why the first step is to explode the payload before the join’s exchange.\nBut exploding alone is not sufficient. After flattening, all rows still share the same join key. The join’s exchange routes them to the same partition, now large not because a single row was heavy, but because one key dominates the row count. Salting is the explicit lever that forces parallelism for that key, regardless of Spark’s partitioning heuristics.\nThis is where asymmetric salting differs from traditional salting. Traditional salting treats both sides of a join symmetrically: the skew comes from repeated keys, so both sides are salted the same way. Asymmetric salting is structurally different. Only the heavy side is decomposed (via explode); the light side is merely replicated with matching salt values. The two sides undergo different transformations to achieve balanced parallelism. The primary target is skewed joins and enrichment pipelines where a nested payload must be matched against a reference dataset; the re-aggregation that often follows is a common but not always necessary step, and, as discussed later, can itself become a bottleneck.\nAlgorithm The algorithm has seven steps.\nInput:\nA left-side dataset containing lightweight entities uniquely identified by a key. A right-side dataset with a foreign key pointing to the left-side key, and a nested array of structs, which may be very large for some keys. Problem: Some right-side records contain massive nested structures for specific keys, causing a single task to become overloaded in Spark during the join, despite a uniform key distribution.\nObjective: Redistribute the processing of these large payloads across multiple tasks, without altering the logical meaning of the data.\nAlgorithm Steps Detect the skew source: Identify that the skew originates not from repeated keys but from a single key associated with a large nested array on the right-hand side. Two practical signals: in the Spark UI, look for a single task with duration far above the median in the join or aggregation stage; in the data, run df.groupBy(key).agg(F.size(payload_col).alias("sz")).orderBy(F.desc("sz")).show() to find the heavy keys.\nExpand the payload: Convert the nested structure into a flat representation, so that individual nested elements become distinct rows. This enables fine-grained distribution of the load.\nGenerate a virtual partitioning key: For each expanded row, compute a salt value, for instance by hashing a distinguishing attribute and taking pmod over a fixed number of buckets. This salt value is used to artificially partition the data across multiple tasks. Alternatively, a sequential index via row_number provides perfect round-robin distribution but introduces an additional shuffle (see Performance Considerations).\nReplicate the salt space on the left side: On the left-side dataset, generate one copy of each row for every possible salt value. This ensures that every sub-partition of the exploded right-side data finds a corresponding match in the join. This replication has a cost: the left side grows by a factor equal to the number of salt buckets, which should be taken into account when sizing the salt space (see Performance Considerations).\nConstruct a salted join key: Combine the original join key and the salt value to form a new composite key used for joining. This ensures that multiple logical partitions of the heavy key can be processed in parallel.\nExecute the join: Perform the join using the salted composite key. This enables Spark to distribute the join across multiple tasks even for originally skewed keys.\n(Optional) Reaggregate the payload: If needed, group the resulting data by the original key and reconstruct the nested structure. When using a left outer join, take care to filter out NULL entries introduced by the replication of the left side: keys with no match on the right side will produce one NULL row per salt bucket instead of a single NULL row. See the array_compact appendix note for the recommended approach.\nSemantic Correctness Spark’s execution model requires that all records sharing the same key are co-located for operations like groupBy and join; under the default hash-partitioning strategy, this is achieved by routing all records with the same key to the same shuffle partition. Asymmetric salting temporarily breaks this contract by splitting the records of a single logical key across multiple partitions via a synthetic composite key. Correctness is restored by the re-aggregation step (step 7), which collapses the salted sub-partitions back into the original key space. Alternatively, if the pipeline’s output model is the exploded representation keyed by (original_key, element_id), re-aggregation is unnecessary, but the synthetic salt column must be dropped, as it is an implementation artifact with no semantic meaning. If re-aggregation is applied but groups by the salted key instead of the original key, the output is semantically wrong: a single logical entity appears as multiple fragments. The composition of salting and re-aggregation preserves the logical key space. This invariant is what makes the approach valid.\nThe Re-aggregation Bottleneck The re-aggregation step deserves special attention. Although asymmetric salting distributes the join workload across multiple tasks, the final groupBy(original_key).collect_list(...) gathers all records for each key back into a single task. For the very keys that caused the original skew, this means the entire payload, now joined and potentially enriched, must be materialized in memory on one executor. If the array was large enough to cause problems in the first place, reconstructing it can reintroduce the same bottleneck. To debug, inspect the intermediate DataFrame after the join and before re-aggregation: verify row counts per key and check for unexpected NULLs.\nAsymmetric salting is therefore fully effective in two scenarios: either the join reduces the per-key payload enough to fit in executor memory, or the pipeline remains in exploded form and writes to a child table or partitioned dataset. As a decision rule: if the output must remain nested and the maximum per-key cardinality reaches 10\u2076 rows or more, salting only moves the bottleneck from join to re-aggregation; a different output shape, such as writing to a flat child table keyed by (original_key, element_id), is the correct solution.\nOne caveat: collect_list does not guarantee element order after shuffles. If downstream consumers expect a stable ordering, sort the collected array explicitly, for instance with F.sort_array() or by collecting a struct with a sort key: transform(array_sort(collect_list(named_struct('ord', sort_col, 'val', payload))), x -> x.val). Sorting the DataFrame before groupBy is not sufficient; the shuffle does not preserve pre-shuffle order.\nPerformance Considerations: Is Asymmetric Salting Expensive? Asymmetric salting is expensive. Two factors dominate. First, explode multiplies the right side: for n rows with an average of m nested elements, the exploded dataset has n \u00d7 m rows. Second, left-side replication: each row must be duplicated once per salt bucket, producing L \u00d7 s rows. In the toy example this is negligible (4 \u00d7 5 = 20), but in production with millions of left-side rows and a high salt count, this multiplication becomes a significant source of memory and shuffle overhead.\nTotal Shuffle Cost The total shuffle cost depends on the strategy. A non-salted sort-merge join incurs two exchanges (one per join input). The hash-based salting variant adds one exchange for re-aggregation, for three total. The row_number variant adds the window shuffle on top of that, for four total. This means the technique replaces two skewed exchanges with three or four balanced ones, a trade-off worthwhile only when the skewed task’s duration dominates the total job runtime. Monitor the Spark UI’s stage breakdown before and after salting to verify the overhead is justified.\nChoosing the Number of Salt Buckets Choosing s is a trade-off between parallelism and replication overhead. Effective parallelism is bounded by min(s, spark.sql.shuffle.partitions, total_executor_cores): setting s above either limit adds replication cost without additional parallelism. Conversely, s should be large enough that max_array_size \/ s produces chunks comparable to the average partition size. In practice, values between 5 and 50 cover most production scenarios. Two constraints must hold simultaneously: max_array_size \/ s \u2248 average_partition_size (to balance the skewed key) and L \u00d7 s should remain manageable relative to the original left-side size (to limit replication overhead). As an order of magnitude: if the largest array has 1M elements and the average partition holds ~100k rows, then s \u2248 10 balances the skewed key into partitions of comparable size.\nSelective Salting In production pipelines, it is common for only a handful of keys to be responsible for the skew, while the vast majority of keys carry small payloads. Applying asymmetric salting to the entire dataset, replicating the full left side and exploding the full right side, is wasteful in such cases. A more efficient strategy is to split the pipeline into two paths: identify the skewed keys (e.g. by computing array sizes and filtering above a threshold), apply salting only to those keys, execute a standard join for the rest, and then union the two results.\nThe typical pattern: compute the set of skewed keys as a small, deduplicated, broadcast DataFrame; split both sides using semi-join (skewed path) and anti-join (normal path); apply salting only to the skewed splits; standard join for the rest; combine with unionByName. The left-side replication must only apply to the skewed left split. If the skewed-key set grows to thousands of keys, the semi\/anti joins themselves become expensive and the split loses its advantage. Getting the splits wrong (filtering only one side) is a common source of missing rows or duplicates.\nA compact skeleton of the split-path pattern:\n# 1. Identify skewed keys (small, deduplicated, broadcast) skewed_keys = ( df_right.select("KEY", F.size("PAYLOAD").alias("sz")) .where("sz > threshold") .select("KEY").distinct() ) skewed_keys = F.broadcast(skewed_keys) # 2. Split both sides left_skewed = df_left.join(skewed_keys, on="KEY", how="semi") left_normal = df_left.join(skewed_keys, on="KEY", how="anti") right_skewed = df_right.join(skewed_keys, on="KEY", how="semi") right_normal = df_right.join(skewed_keys, on="KEY", how="anti") # 3. Salted join on skewed path only result_skewed = salted_join(left_skewed, right_skewed, num_salts) # 4. Standard join on normal path result_normal = left_normal.join(right_normal, on="KEY", how="left") # 5. Combine result = result_skewed.unionByName(result_normal) For incremental pipelines, compute the threshold dynamically (for example, median_payload_size * skew_factor) rather than hardcoding it, since the skew profile may shift between runs.\nCode Example This example demonstrates asymmetric salting to mitigate data skew in a left outer join in Apache Spark. The left side (df_buildings) contains a list of building IDs, while the right side (df_deliveries) includes a nested array of packages per building. One building ("A") has significantly more packages than the others, causing skew.\nTo address this, the nested array on the right side is first exploded, turning each package into its own row. A salt_index is then assigned to each row and combined with the building ID into a struct-based JOIN_KEY. Two salt strategies are shown: a hash-based approach (recommended for production, no extra shuffle or sort) and a row_number-based approach (perfect round-robin distribution, but requires a window shuffle and sort; see Note 6).\nOn the left side, each building is replicated for every possible salt index, generating the same JOIN_KEY values. This enables a join that spreads the heavy building’s records across multiple partitions, ensuring better parallelism.\nAfter the join, the results are re-aggregated to restore the original nested format. Tests are included to confirm both that all 1000 packages for building "A" are preserved, and that no spurious NULL entries appear for buildings with few or no packages.\nfrom pyspark.sql import SparkSession, functions as F from pyspark.sql.window import Window from pyspark.sql.types import StructType, StructField, StringType, ArrayType spark = SparkSession.builder.appName("AsymmetricSaltingExample").getOrCreate() # Sample data: Buildings (left side) buildings_data = [("A",), ("B",), ("C",), ("D",)] df_buildings = spark.createDataFrame(buildings_data, ["BUILDING_ID"]) # Sample data: Deliveries (right side), skewed on BUILDING_ID = 'A' # "C" has an empty array, "D" has null \u2014 both normalize to [] in the output. deliveries_data = [ ("A", [{"package_id": f"A-{i}"} for i in range(1000)]), ("B", [{"package_id": "B-1"}]), ("C", []), ("D", None), ] deliveries_schema = StructType( [ StructField("BUILDING_ID", StringType(), True), StructField("PACKAGES", ArrayType(StructType([StructField("package_id", StringType(), True)])), True), ] ) df_deliveries = spark.createDataFrame(deliveries_data, deliveries_schema) # Explode nested array on the skewed side. # explode_outer preserves rows with null or empty arrays by emitting a # single null-element row (see Note 5). df_exploded = df_deliveries.withColumn("PACKAGE", F.explode_outer("PACKAGES")) # --- Salt strategy -------------------------------------------------------- # RECOMMENDED: hash-based salt. No extra shuffle, no sort, deterministic. # Null payloads (from explode_outer) are routed to bucket 0 explicitly. # The struct(isNull, id) wrapper distinguishes null from non-null package_ids in the hash input. num_salts = 5 df_salted_right = df_exploded.withColumn( "salt_index", F.when(F.col("PACKAGE").isNull(), F.lit(0)) .otherwise(F.pmod( F.hash(F.struct( F.col("PACKAGE.package_id").isNull(), F.col("PACKAGE.package_id"), )), F.lit(num_salts) )) ) # ALTERNATIVE: row_number-based salt. Perfect round-robin but adds a # shuffle + sort via the window function. Uncomment to use. # # window_spec = Window.partitionBy("BUILDING_ID").orderBy(F.col("PACKAGE.package_id")) # df_salted_right = df_exploded.withColumn( # "salt_index", (F.row_number().over(window_spec) - 1) % num_salts # ) # -------------------------------------------------------------------------- df_salted_right = ( df_salted_right .withColumn("JOIN_KEY", F.struct(F.col("BUILDING_ID"), F.col("salt_index"))) # BUILDING_ID is carried by the left side of the join; drop the right-side copy to avoid ambiguity. .drop("BUILDING_ID", "salt_index", "PACKAGES") ) # Left side: replicate each row for every salt_index (0 to num_salts - 1). # broadcast() ensures the salt DataFrame is broadcast to all executors rather than shuffled. df_salts = spark.createDataFrame([(i,) for i in range(num_salts)], ["salt_index"]) df_left_expanded = ( df_buildings.crossJoin(F.broadcast(df_salts)) .withColumn("JOIN_KEY", F.struct(F.col("BUILDING_ID"), F.col("salt_index"))) .drop("salt_index") ) # Join on salted key df_joined = df_left_expanded.join(df_salted_right, on="JOIN_KEY", how="left") # Re-aggregation: restore nested format. # array_compact (Spark >= 3.4) removes NULLs from unmatched salt replicas. df_result = df_joined.groupBy("BUILDING_ID").agg( F.array_compact(F.collect_list("PACKAGE")).alias("PACKAGES") ) # Materialize once; count() forces full evaluation before tests. df_result = df_result.cache() try: df_result.count() df_result.show(truncate=False) # ------------------------ # Tests for correctness # ------------------------ def test_asymmetric_salting(): rows = {row["BUILDING_ID"]: row["PACKAGES"] for row in df_result.collect()} # Test 1: All 1000 packages for building 'A' are preserved assert len(rows["A"]) == 1000, f"Expected 1000 packages for 'A', got {len(rows['A'])}" # Test 2: Buildings with few packages have the correct count (no duplicates from salt replicas) assert len(rows["B"]) == 1, f"Expected 1 package for 'B', got {len(rows['B'])}" # Test 3: Empty-array building produces an empty PACKAGES array (not null, not inflated) assert len(rows["C"]) == 0, f"Expected 0 packages for 'C', got {len(rows['C'])}" # Test 4: Null-array building also normalizes to an empty PACKAGES array assert len(rows["D"]) == 0, f"Expected 0 packages for 'D', got {len(rows['D'])}" # Test 5: No NULL entries leak into any PACKAGES array for bid, packages in rows.items(): null_count = sum(1 for p in packages if p is None) assert null_count == 0, f"Found {null_count} NULL(s) in PACKAGES for building '{bid}'" test_asymmetric_salting() finally: df_result.unpersist() spark.stop() Conclusions Asymmetric salting offers a targeted solution to data skew in Spark when the issue lies not in key frequency but in oversized payloads per key. By exploding nested data and introducing synthetic keys, it enables Spark to parallelize workloads that would otherwise overwhelm a single task.\nWhile more expensive than built-in optimizations, due to both the explosion of the right side and the replication of the left side, it is effective when AQE falls short, especially for complex aggregations involving nested arrays. It is worth noting that this is one of several strategies for handling nested skew; alternatives such as map-side pre-aggregation, bucketing, or custom partitioners may be more appropriate depending on the specific pipeline topology and data characteristics. Used judiciously, and with careful calibration of the number of salt buckets, asymmetric salting turns skewed joins into scalable, balanced computations.\nAppendix: Implementation Notes collect_list and skew. collect_list materializes the entire group in memory on a single executor. Unlike scalar aggregations (count, sum), it requires the full group before producing a result. This makes it one of the most common triggers for skew-related OutOfMemoryError in nested-data pipelines.\nexplode vs explode_outer. explode silently drops rows where the array is null or empty. explode_outer preserves them by emitting a single row with a null value for the exploded column, important for left-join semantics where every key must appear in the output.\nAQE skew detection parameters. skewedPartitionThresholdInBytes (default: 256 MB) and skewedPartitionFactor (default: 5\u00d7) must both be met. Thresholds operate on serialized shuffle output size (compressed by default via spark.shuffle.compress). See Spark’s configuration reference.\nSalt strategies. The code example uses pmod(hash(...)) as the production default. Three alternatives avoid the window shuffle:\n(F.rand(seed) * s).cast("int"): lightest option, statistically uniform but not perfectly so. Use a seed for reproducibility in test environments. F.monotonically_increasing_id() % s: deterministic per-run but distribution is uneven because the partition ID is encoded in the upper bits. pmod(hash(struct(isNull(col), col)), s): deterministic, reproducible, distributes well for high-cardinality attributes. Use pmod (not %) because hash() returns a signed 32-bit integer. Hash-based distribution becomes uneven when the number of distinct elements within a key is low relative to the salt bucket count; in such cases, rand-based salting or a smaller bucket count produces better balance. The row_number() variant forces a shuffle + sort via Window.partitionBy, concentrating the skewed key’s rows into a single task before salting, the very problem salting is meant to solve. Use only when perfect round-robin is required and the sort cost is acceptable.\nA note on hash(null): Spark’s MurmurHash3 never returns null. For a null column value it returns the seed value (currently 42), an implementation detail, not a contract. The struct wrapper struct(isNull(id), id) ensures that (true, null) hashes differently from (false, real_id). Always use an explicit guard for null payloads rather than relying on hash behavior.\nBroadcast hint for replication. The explicit F.broadcast() on the salt DataFrame ensures the cross join runs locally per partition. Without it, Spark usually broadcasts automatically (if size < autoBroadcastJoinThreshold), but this threshold is sometimes set to -1 in large-scale jobs. The explicit hint removes the ambiguity.\nStruct vs string composite key. F.struct(key, salt) preserves types and avoids ambiguity. F.concat_ws("-", key, salt) is readable but if the key itself contains the separator, parsing the composite key back into its components becomes unreliable. The struct-based approach eliminates this class of bugs entirely.\narray_compact and null handling. collect_list already skips nulls, but array_compact (Spark >= 3.4) makes the intent explicit. For Spark 3.1\u20133.3, use F.filter(collect_list(...), lambda x: x.isNotNull()).\nNULL multiplication in left outer joins. Between the salted join and re-aggregation, cardinality is temporarily inflated: keys with no right-side match appear s times (one NULL per salt bucket), and keys with partial matches produce s \u2212 k NULL rows. If monitoring or audit stages run on the intermediate DataFrame, the inflated counts can trigger false alerts. Re-aggregation corrects this.\nAlternative strategies. Asymmetric salting is one of several approaches: map-side pre-aggregation, repartitionByRange, custom RDD partitioners, persistent bucketBy, AQE automatic skew join optimization (Spark 3.0+, with improvements in 3.2+), and broadcast side switching. Each involves different trade-offs in complexity, overhead, and optimizer compatibility.\nNull vs empty array normalization. The code normalizes both null and [] to []. If your domain requires the distinction (auditing, SLA enforcement), add a flag column before exploding: withColumn("was_null", col("PACKAGES").isNull()) and restore it after re-aggregation via F.when(col("was_null"), lit(None)).otherwise(col("PACKAGES")).\nBibliography Apache Spark. Configuration: Runtime SQL Configuration. In Spark 3.5.8 Documentation. Retrieved from https:\/\/spark.apache.org\/docs\/3.5.8\/configuration.html#runtime-sql-configuration\nApache Spark. Performance Tuning: Adaptive Query Execution. In Spark 3.5.8 Documentation. Retrieved from https:\/\/spark.apache.org\/docs\/3.5.8\/sql-performance-tuning.html\nCanadian Data Guy. A deep dive into skewed joins, groupby bottlenecks, and smart strategies to keep your Spark jobs flying. Canadian Data Guy’s No Fluff Newsletter. Retrieved June 2025 from https:\/\/www.canadiandataguy.com\/p\/a-deep-dive-into-skewed-joins-groupby\nCox Automotive Data Solutions. The Taming of the Skew, Part Two. Data Driven Blog. Retrieved from https:\/\/coxautomotivedatasolutions.github.io\/datadriven\/spark\/data%20skew\/joins\/data-skew-2\/\nDatabricks. Adaptive query execution. Databricks Documentation. Retrieved from https:\/\/docs.databricks.com\/aws\/en\/optimizations\/aqe\n","permalink":"https:\/\/cdelmonte.dev\/deep-dives\/fixing-skewed-nested-joins-spark-asymmetric-salting\/","summary":"In large-scale Spark pipelines, skew can occur when a single key carries a disproportionately large nested payload. Asymmetric salting offers a targeted solution: explode, salt, join in parallel, and optionally re-aggregate.","title":"Fixing Skewed Nested Joins in Spark with Asymmetric Salting"},{"content":"Suppose we keep a purchases table in Parquet, one file per day, each file sorted by customer. Suppose also that a view over the data reports, at the end of the month, how many purchases each customer made: in SQL, a COUNT(*) grouped by customer.\nSince every file is sorted by customer, we would expect the query engine to use that order to optimize the GROUP BY. Yet EXPLAIN, or EXPLAIN VERBOSE, sometimes shows the aggregation running without using the order that is already there. Why? Where along the way does the ordering of the rows in the files get lost?\nTo see how a query engine propagates ordering, we will follow it from the file scan to the aggregate. We focus on Parquet scans in Apache DataFusion, but the principles apply to other relational query engines as well. The distinction between the order of a file and the order of a partition is general; the steps and settings described are those of DataFusion 55.1.0 and its listing tables.\nTwo things to settle before we set off. Asking for the result in customer order, with ORDER BY, is a different matter. It concerns the order of the rows the query returns, not the order an operator receives. And \u201csorted\u201d here means that the rows inside each file follow the sort key used by whoever wrote them. Directory partitioning and clustering schemes also shape the data layout, but on their own they do not guarantee that row order.\n1. The example The example is simple: three days of purchases, one Parquet file per day.\n\/data\/purchases\/monday.parquet \/data\/purchases\/tuesday.parquet \/data\/purchases\/wednesday.parquet Every file is sorted by customer. These are the rows in each file, in the order in which they are stored:\n\/data\/purchases\/monday.parquet\ncustomer city product amount Anna Rome Book 12 Bruno Milan Coffee 4 Zeno Rome Pen 2 \/data\/purchases\/tuesday.parquet\ncustomer city product amount Anna Rome Coffee 4 Bruno Milan Book 12 Zeno Rome Book 12 \/data\/purchases\/wednesday.parquet\ncustomer city product amount Anna Rome Pen 2 Bruno Milan Pen 2 Zeno Rome Coffee 4 The table built on these files declares the order in which the rows were written:\nCREATE EXTERNAL TABLE purchases STORED AS PARQUET WITH ORDER (customer ASC) LOCATION '\/data\/purchases\/'; And the query counts purchases per customer (to keep the example small, we configure the engine to use two partitions; by default it uses as many as the machine’s cores):\nSET datafusion.execution.target_partitions = 2; SELECT customer, COUNT(*) FROM purchases GROUP BY customer; Every file is sorted by customer, and every file contains Anna, Bruno and Zeno: their customer ranges overlap, so no file ends before another begins. This is the layout we are probing. The table declares the order by customer, and the query groups by the same column. Yet this is what EXPLAIN gives us:\nEXPLAIN FORMAT INDENT SELECT customer, COUNT(*) FROM purchases GROUP BY customer; ProjectionExec: expr=[customer@0 as customer, count(Int64(1))@1 as count(*)] AggregateExec: mode=FinalPartitioned, gby=[customer@0 as customer], aggr=[count(Int64(1))] RepartitionExec: partitioning=Hash([customer@0], 2), input_partitions=2 AggregateExec: mode=Partial, gby=[customer@0 as customer], aggr=[count(Int64(1))] DataSourceExec: file_groups={2 groups: [[data\/purchases\/monday.parquet, data\/purchases\/tuesday.parquet], [data\/purchases\/wednesday.parquet]]}, projection=[customer], file_type=parquet Nothing in this plan says the order was preserved. If it had been, the plan would mark it on the scan with output_ordering and on the aggregate with ordering_mode; here neither appears. Even with the experimental setting that groups files by their statistics turned on (datafusion.execution.split_file_groups_by_statistics = true), the plan stays the same.\nWhat is going on? Why is the order gone?\n2. The premise: order starts inside a file Let’s start at the beginning.\nWe assumed that the files were sorted internally when they were written. The engine does not know this on its own: it has to be told, and in DataFusion there are two ways to tell it.\nThe first is the WITH ORDER clause in the definition of the external table, the one in our example, which informs the planner of the existing order.\nThe second is the file’s metadata. In Parquet, rows are divided into row groups, and the metadata of each row group has a field, sorting_columns, where the writer can record which columns its rows are sorted by. DataFusion reads this field only from the first row group of each file and takes it as the order of the whole file. If every file has this field, the engine derives the table’s ordering from their common prefix, even without a declaration. If even one file lacks it, no ordering is inferred. DataFusion reads this metadata alongside the file statistics, so this happens only when datafusion.execution.collect_statistics is on, as it is by default.\nSome words of caution are due here. If the table declares WITH ORDER and the files carry the metadata too, the declaration wins. Furthermore, the field is not always filled in: DataFusion writes it when it saves sorted rows, for example with COPY \u2026 ORDER BY, but other writers may leave it out. Finally, the engine takes the order on trust: it does not check it. If Monday’s file had in fact been written by time of purchase, declaring the order by customer, or recording it in the metadata, would hand the planner a false premise, and an optimization built on that premise could produce wrong results. With the metadata there is one more risk: sorting_columns only says that each row group is sorted internally. For the whole file to be sorted, the boundary between consecutive row groups has to respect the order too, and DataFusion does not check that. [1]\n3. The scan operator produces partitions Sorted files and an order known to the engine are just a promising starting point. The question now is what happens to them. The answer lies in the physical plan. [2]\nA physical plan is a tree of operators, and each operator produces its output as a number of partitions, parallel streams of batches. Partition i of an operator usually reads from partition i of its input, so each partition is a lane that runs up the tree, and the engine executes each lane as an independent unit of work (this is also called intra-operator parallelism). [3] Most operators keep the number of lanes they receive: a projection, a filter or a partial aggregate neither adds nor removes any. The number is set at the bottom by the scan, and it changes only where the plan says so: a repartition operator moves rows between lanes and may change their number, and a merge or coalesce operator brings them down to one.\nThe scan is the operator at the bottom of the tree, the one with no input. Its partitions are decided by how it organizes the files into groups: each partition is the stream that reads one group, one file after the other, while different partitions run in parallel. The order within a partition therefore depends on reading the files of its group in sequence. Two sorted files read one after the other stay sorted only if the first ends no later than the second begins. The scan concatenates the files of a group, it does not merge them.\nBy default the engine aims for as many partitions, and so as many groups, as the processor has cores. We can set this number explicitly by assigning a value N to target_partitions.\nHow are the files assigned to those groups? It takes up to six steps. [4]\nStatistics. With statistics collection enabled, the engine reads the column minima and maxima from each file footer, along with sorting_columns if present, as we saw in section 2. Default grouping. It sorts the files by path and cuts them into contiguous blocks of \u2308number of files \/ N\u2309 files each. Each block is a group. The values inside the files play no part in this step: only the file path and its position in the list matter. Candidate order. It is the one from WITH ORDER, if there is one; otherwise the one common to the sorting_columns of all the files; otherwise none. Without a candidate order the scan has nothing to preserve, and it goes straight to step 6. Statistics-based grouping. If it is on, the engine tries to build different groups. It sorts the files by ascending minimum and starts from N empty groups. It places each file in the least full group among those that are empty and those whose last file ends strictly before the new one begins. If no group fits, it opens one more. At the end, if there are no more than N groups, it uses them; otherwise it falls back to the groups of step 2 and notes it in the debug log. Validation. It checks the candidate order on the groups obtained. In every group with more than one file, the maximum of each file must be less than or equal to the minimum of the next. If all the groups pass, the scan advertises its ordering. If even one fails, or if statistics are missing, the scan does not advertise it. Splitting. If the table has more rows than one batch (8192 by default), the engine may split the files into byte ranges to fill more partitions. With an advertised ordering it does so only if the groups are fewer than N, and it splits only groups made of a single file: each piece becomes a group of its own, and the order remains. Without an order, if the files together exceed repartition_file_min_size (1 MiB by default), it redistributes all the bytes into N groups of equal size, even across file boundaries. In our example N is 2, and the two file groups in the plan are the two partitions of the scan. [5] Step 1 reads the same statistics from every file: minimum Anna, maximum Zeno. Step 2 puts Monday and Tuesday in the same group and leaves Wednesday on its own. Step 3 takes the candidate order from the declaration: by customer. Step 4, when it is on, would want one group per file, because no file can follow another, but three groups exceed N, and the proposal is discarded. Step 5 finds Zeno before Anna in the group of Monday and Tuesday, and the scan does not advertise its ordering. Step 6 does not apply because there are too few rows. So an order known for individual files may not survive this stage. [6]\nWith N equal to 3, or with the default on a machine with more cores, every file would get a group of its own at step 2, and every partition would stay sorted. The three sorted partitions still overlap, though: all three contain Anna. For a GROUP BY customer this is not a problem. The partial aggregate works on each partition separately and only needs that partition to be sorted; Anna’s counts from the different partitions are combined later, as section 4 shows.\nBut take thirty daily files and N = 8: step 2 fills blocks of a fixed size until the files run out, and the last block takes what is left.\nstep 2 with 30 files and N = 8 block size \u230830 \/ 8\u2309 = 4 full blocks 7, holding 28 files last block 2 files groups 8 The higher parallelism does not help: the engine still forms exactly N groups, so with thirty files most of them hold several files each, and the order is lost in the same way as before.\nTo summarize, what matters is which files end up in the same group: by path at step 2, by range at step 4. With overlapping files like ours no grouping can put two files together and keep them sorted, so the order survives only while the files are no more than N. Files with disjoint ranges can do better, as section 6.2 shows.\nFile order and partition order, as a GROUP BY sees them Files (each sorted by customer)monday.parquetAnnaMonBrunoMonZenoMontuesday.parquetAnnaTueBrunoTueZenoTueScan with one grouppartition 0 = monday, then tuesdayAnnaMonBrunoMonZenoMonAnnaTue...Zeno, then Anna: the partition is not sortedFile order: rows inside one file follow the key. It says nothing about the next file. Files (each sorted by customer)monday.parquetAnnaMonBrunoMonZenoMontuesday.parquetAnnaTueBrunoTueZenoTueScan with two groupspartition 0 = mondayAnnaMonBrunoMonZenoMonpartition 1 = tuesdayAnnaTueBrunoTueZenoTueeach partition is sorted; the scan can advertise the orderPartition order: the files of a group, concatenated, follow the key. One property per partition. Files (each sorted by customer)monday.parquetAnnaMonBrunoMonZenoMontuesday.parquetAnnaTueBrunoTueZenoTueTwo sorted partitionspartition 0AnnaMonBrunoMonZenoMonpartition 1AnnaTueBrunoTueZenoTueaggregatepartition 0aggregatepartition 1GROUP BY customer: each partial aggregate uses the order of its own partition.Both partitions contain Anna; the final aggregate combines her counts later. For GROUP BY customer this is enough. Each partial aggregate works on one partition and uses its order on its own; that both partitions contain Anna does not matter, because the final aggregate combines her counts later. ← Back 3 \/ 3 Next → One last point about groups and partitions. The six steps fix the groups at planning time, but execution could still move files between them: by default, a partition that has run out of work can take files assigned to another. When the table has an order, declared or derived from the metadata, DataFusion switches this mechanism off, and every file stays in its group. [5]\n4. How the order climbs the plan Suppose the scan operator really does deliver sorted partitions. What happens then? The order now has to climb the plan, one operator at a time.\nThe planner makes its decisions before execution, so it cannot inspect the rows the operators will produce: it relies on what each operator declares about its own output, including its ordering. Each operator, when it is built, works out the order of its output from the order of its input; since the plan is built from the scan upward, the order propagates in the same direction. Different operators treat the order differently. A filter drops rows but does not move them, so its output keeps the order of its input. An operator that shuffles rows loses it, unless it does something to preserve it.\nFor a partial aggregate (the aggregation that runs within a partition), the question is how much of that order concerns its keys. The plan shows it with ordering_mode: Sorted if the incoming order begins with all the keys of the GROUP BY, in any sequence; PartiallySorted if it begins with only some of them. Only the leading run of keys counts: as soon as the order reaches a column outside the GROUP BY, the keys after it no longer help, unless a filter has fixed that column to a single value, in which case the planner drops it from the order. In both modes the aggregate emits its groups in the order in which their keys arrive, so its output is sorted by the same keys and the next operator receives that order. If none of the keys forms a useful ordering prefix, the plan shows no ordering_mode: the aggregate does not use the input ordering, its output is unsorted, and it passes no ordering on. [7]\nIn our example, with two partitions, the order is already lost at the scan. To see how it travels along the plan we need a scan that preserves it and passes it on. Raising the example’s target partitions to three is enough: every file then gets a group of its own, as we saw in section 3.\nSET datafusion.execution.target_partitions = 3; ProjectionExec: expr=[customer@0 as customer, count(Int64(1))@1 as count(*)] AggregateExec: mode=FinalPartitioned, gby=[customer@0 as customer], aggr=[count(Int64(1))], ordering_mode=Sorted RepartitionExec: partitioning=Hash([customer@0], 3), input_partitions=3, preserve_order=true, sort_exprs=customer@0 ASC NULLS LAST AggregateExec: mode=Partial, gby=[customer@0 as customer], aggr=[count(Int64(1))], ordering_mode=Sorted DataSourceExec: file_groups={3 groups: [[data\/purchases\/monday.parquet], [data\/purchases\/tuesday.parquet], [data\/purchases\/wednesday.parquet]]}, projection=[customer], output_ordering=[customer@0 ASC NULLS LAST], file_type=parquet Read from the bottom up: the scan advertises its ordering (output_ordering), the partial aggregate receives it and works in Sorted mode, the repartition operator preserves it (preserve_order=true), and the final aggregate works in Sorted mode too.\nSo far we have seen the order reach the partial aggregate and get passed on. What remains is how the repartition operator manages to preserve it. The final aggregate requires all the partial states for a customer to land in the same partition, and to make that possible the planner inserts a repartition operator that redistributes the states by a hash of the key. Every output partition thus receives states from all the scan partitions, and even if every incoming stream is sorted, it could receive Zeno’s state from Monday before Anna’s state from Tuesday: taking the batches as they arrive would break the order. [7]\nHash repartition of two sorted partitions, with and without preserve_order Scan partitions (each sorted by customer)partition 0, Mondaypartition 1, TuesdayOutput partitions, hash(customer)output 0: Anna, Zenooutput 1: BrunoAnnaMonBrunoMonZenoMonAnnaTueBrunoTueZenoTue Scan partitions (each sorted by customer)partition 0, Mondaypartition 1, TuesdayOutput partitions, hash(customer)output 0: Anna, Zenooutput 1: BrunoRepartitionExec, preserve_order=falseAnnaMonBrunoMonZenoMonAnnaTueBrunoTueZenoTueZenoMonAnnaTueZeno before Anna: output 0 is not sortedBrunoMon Scan partitions (each sorted by customer)partition 0, Mondaypartition 1, TuesdayOutput partitions, hash(customer)output 0: Anna, Zenooutput 1: BrunoRepartitionExec, preserve_order=trueAnnaMonBrunoMonZenoMonAnnaTueBrunoTueZenoTueAnnaMonBrunoMonBrunoTuemerge for output 0: next of each streamAnnaMonAnnaTuesmallest goes out Scan partitions (each sorted by customer)partition 0, Mondaypartition 1, TuesdayOutput partitions, hash(customer)output 0: Anna, Zenooutput 1: BrunoRepartitionExec, preserve_order=trueAnnaMonBrunoMonZenoMonAnnaTueBrunoTueZenoTueAnnaMonAnnaTueBrunoMonBrunoTuemerge for output 0: next of each streamZenoMonAnnaTuesmallest goes out Scan partitions (each sorted by customer)partition 0, Mondaypartition 1, TuesdayOutput partitions, hash(customer)output 0: Anna, Zenooutput 1: BrunoRepartitionExec, preserve_order=trueAnnaMonBrunoMonZenoMonAnnaTueBrunoTueZenoTueAnnaMonAnnaTueZenoMonBrunoMonBrunoTuemerge for output 0: next of each streamZenoMonZenoTuesmallest goes out Scan partitions (each sorted by customer)partition 0, Mondaypartition 1, TuesdayOutput partitions, hash(customer)output 0: Anna, Zenooutput 1: BrunoRepartitionExec, preserve_order=trueAnnaMonBrunoMonZenoMonAnnaTueBrunoTueZenoTueAnnaMonAnnaTueZenoMonZenoTueBrunoMonBrunoTuemerge for output 0: next of each stream(done)(done) Final state with preserve_order: every output partition is sorted by customer, so the final aggregate on each output can close a customer’s group as soon as the next customer arrives. ← Back 6 \/ 6 Next → To solve this, the repartition operator has an order-preserving variant, which merges the incoming streams for each output partition: it compares the next customer in each stream and emits the smallest, so Anna comes before Zeno.\nThe planner inserts this variant whenever the repartition operator’s input is sorted and has more than one partition, then checks whether the operators above make use of it. It stays only if the operator above preserves the order, or if the order lets that operator emit results before the end of its input, as our final aggregate does. Otherwise the plain repartition operator takes its place, because the merge has a runtime cost.\nThe fallback shows up, for example, in a partitioned hash join between two tables sorted by customer with nothing but a projection above it: nothing downstream uses the order, so the repartition operators below the join stay plain. The join itself passes on the order of its second input, so a GROUP BY customer above the join brings the order-preserving variant back on that side. [8]\n5. What the aggregate gains The climb ends at the aggregate, the operator that uses the order: the partial one inside each partition and the final one after the repartition. Now we can look at what the order means during execution, and whether it helps.\nBack to the count per customer:\nSELECT customer, COUNT(*) FROM purchases GROUP BY customer; To group the rows by customer, DataFusion hashes the key of each row and keeps one entry per customer in a hash table, where the count accumulates. The order does not change how the rows are counted; it changes when an entry can leave the table.\nInside a partition sorted by customer, once the partial aggregate reaches Bruno, no more rows for Anna can arrive in that partition: her count is complete and her entry can leave. Without a useful order, Anna could turn up again further on, and her entry would have to stay until the end of the input. After each incoming batch, the partial aggregate emits the completed groups and removes their entries from the table. [9]\nWhat the partial aggregate emits is Anna’s count in that partition, not her total. The totals are computed by the final aggregate, which receives Anna’s states in order from the repartition operator, as we saw in section 4. When Bruno turns up in its stream, the final aggregate knows in turn that no more partial states for Anna can arrive, and emits Anna’s total. This is why the planner keeps the order-preserving variant of the repartition operator. [10]\nEarly emission in an ordered aggregation Partial aggregate, partition 0: Monday's file, input sorted by customerbatch 1 (the whole file)AnnaBook, 12BrunoCoffee, 4ZenoPen, 2group table (state in memory)Annacount 1Brunocount 1Zenocount 1emitted partial statesAfter the batch the current group is Zeno. Anna and Bruno cannot return: the input is sorted. Partial aggregate, partition 0: Monday's file, input sorted by customerbatch 1 (the whole file)AnnaBook, 12BrunoCoffee, 4ZenoPen, 2group table (state in memory)Zenocount 1emitted partial statesAnnacount 1Brunocount 1Anna's and Bruno's partial states leave the table before the end of the input. Partial aggregate, partition 0: Monday's file, input sorted by customerbatch 1 (the whole file)AnnaBook, 12BrunoCoffee, 4ZenoPen, 2group table (state in memory)emptyemitted partial statesAnnacount 1Brunocount 1Zenocount 1End of input: the last group is emitted. These are partial states, not totals. Final aggregate: partial states from the three partitions, by hash of customerfrom partition 0 (Monday)Annacount 1Brunocount 1Zenocount 1from partition 1 (Tuesday)Annacount 1Brunocount 1Zenocount 1from partition 2 (Wednesday)Annacount 1Brunocount 1Zenocount 1totalsAnnatotal 3Brunototal 3Zenototal 3The final aggregate combines the partial states of each customer from every partition.Early emission changes when a partial state leaves memory, not what the total is. The final aggregate receives Anna’s partial states from the three partitions, Monday, Tuesday and Wednesday, and combines them into her total of 3. Early emission shortens how long a state stays in memory; the result is the same. ← Back 4 \/ 4 Next → With the order on all the keys, as here, each aggregate keeps only the last group open after each batch, and the state it holds between batches does not grow with the number of customers. If the order covers only some of the keys, as in GROUP BY customer, product, the advantage shrinks: a group closes only when the customer changes, and many product groups can accumulate for a single customer. [9]\nIf the order covers none of the keys of the GROUP BY, there is no gain at all: when we count purchases by city, Rome appears both under Anna and under Zeno. Ordering by customer gives us no boundary at which to close the group for Rome. In fact, in this query the scan does not even advertise the order: it reads only city, and an ordering on a column that is not read cannot be expressed on the scan’s output. [6]\n6. Diagnosis and remedies So far we have looked at the mechanism. But in practice, how can we tell where the order is lost, and how do we keep it?\n6.1. Where the order stops The first question is answered by the plan. Read EXPLAIN from the bottom up, as in section 4. If the aggregate shows no ordering_mode, it is not using an ordering useful to its keys. Trace its input back toward the scan to find the last operator that still advertised one. From there, these questions retrace the chain from the files upward:\nAre the rows inside each file really sorted as declared, or as recorded in the metadata? (section 2) If the table does not declare the order, do all the files report sorting_columns in the first row group? (section 2) Are file statistics being collected (datafusion.execution.collect_statistics)? (section 3, statistics) If statistics-based grouping is on, do the proposed groups fit within the target? (section 3, statistics-based grouping) Can the files of each group be concatenated in the assigned sequence? (section 3, validation) Does the query read the sort column? An ordering on a column the scan does not read disappears at the scan. (section 5) Does the scan advertise its ordering, and does the repartition operator preserve it? (section 4) Does the aggregate show ordering_mode for its keys? (sections 4 and 5) One caveat. The plan does not say why an order was discarded. The rejection of statistics-based grouping, for example, appears only in the debug log; with datafusion-cli you can see it by setting RUST_LOG=datafusion_catalog_listing=debug.\n6.2. How to keep it When the diagnosis shows that the order is lost in the scan, because the files of a group cannot be concatenated or because statistics-based grouping is discarded, there are a few things you can do to keep it.\nThe first is to change the layout of the files before querying them: for example, files covering separate customer ranges can be concatenated in range order.\nThe layout alone, however, is not enough: the scan groups have to respect it too. [6] Suppose each day is written into two files, one for customers from A to L and one for those from M to Z, each sorted by customer:\n\/data\/purchases\/a-l-monday.parquet Anna, Bruno, Carla \/data\/purchases\/a-l-tuesday.parquet Anna, Bruno, Carla \/data\/purchases\/a-l-wednesday.parquet Anna, Bruno, Carla \/data\/purchases\/m-z-monday.parquet Marco, Sara, Zeno \/data\/purchases\/m-z-tuesday.parquet Marco, Sara, Zeno \/data\/purchases\/m-z-wednesday.parquet Marco, Sara, Zeno The table and the query are the same, and the engine works with three partitions:\nSET datafusion.execution.target_partitions = 3; The default grouping cuts the list by path into blocks of two files. The first group puts Monday’s and Tuesday’s A\u2013L files together, which overlap, so the order is lost:\nProjectionExec: expr=[customer@0 as customer, count(Int64(1))@1 as count(*)] AggregateExec: mode=FinalPartitioned, gby=[customer@0 as customer], aggr=[count(Int64(1))] RepartitionExec: partitioning=Hash([customer@0], 3), input_partitions=3 AggregateExec: mode=Partial, gby=[customer@0 as customer], aggr=[count(Int64(1))] DataSourceExec: file_groups={3 groups: [[data\/purchases\/a-l-monday.parquet, data\/purchases\/a-l-tuesday.parquet], [data\/purchases\/a-l-wednesday.parquet, data\/purchases\/m-z-monday.parquet], [data\/purchases\/m-z-tuesday.parquet, data\/purchases\/m-z-wednesday.parquet]]}, projection=[customer], file_type=parquet A solution is to turn on statistics-based grouping:\nSET datafusion.execution.split_file_groups_by_statistics = true; Off by default, this setting builds groups whose files can be concatenated according to their statistics. [11]\nProjectionExec: expr=[customer@0 as customer, count(Int64(1))@1 as count(*)] AggregateExec: mode=FinalPartitioned, gby=[customer@0 as customer], aggr=[count(Int64(1))], ordering_mode=Sorted RepartitionExec: partitioning=Hash([customer@0], 3), input_partitions=3, preserve_order=true, sort_exprs=customer@0 ASC NULLS LAST AggregateExec: mode=Partial, gby=[customer@0 as customer], aggr=[count(Int64(1))], ordering_mode=Sorted DataSourceExec: file_groups={3 groups: [[data\/purchases\/a-l-monday.parquet, data\/purchases\/m-z-monday.parquet], [data\/purchases\/a-l-tuesday.parquet, data\/purchases\/m-z-tuesday.parquet], [data\/purchases\/a-l-wednesday.parquet, data\/purchases\/m-z-wednesday.parquet]]}, projection=[customer], output_ordering=[customer@0 ASC NULLS LAST], file_type=parquet There are still three groups, but each pairs a day’s A\u2013L file with its M\u2013Z file. Those two files can be concatenated in order. With the same number of partitions, the order reaches the aggregate.\nTurning the setting on, however, does not guarantee that the proposed groups are used: as we saw at the statistics-based grouping step in section 3, the engine discards them if they exceed the target. [12]\nRaising the target can make the grouping acceptable, and sometimes it is enough on its own. With six partitions the default grouping gives every file a group of its own, even without statistics:\nSET datafusion.execution.target_partitions = 6; ProjectionExec: expr=[customer@0 as customer, count(Int64(1))@1 as count(*)] AggregateExec: mode=FinalPartitioned, gby=[customer@0 as customer], aggr=[count(Int64(1))], ordering_mode=Sorted RepartitionExec: partitioning=Hash([customer@0], 6), input_partitions=6, preserve_order=true, sort_exprs=customer@0 ASC NULLS LAST AggregateExec: mode=Partial, gby=[customer@0 as customer], aggr=[count(Int64(1))], ordering_mode=Sorted DataSourceExec: file_groups={6 groups: [[data\/purchases\/a-l-monday.parquet], [data\/purchases\/a-l-tuesday.parquet], [data\/purchases\/a-l-wednesday.parquet], [data\/purchases\/m-z-monday.parquet], [data\/purchases\/m-z-tuesday.parquet], ...]}, projection=[customer], output_ordering=[customer@0 ASC NULLS LAST], file_type=parquet The order reaches the aggregate this way too, but with twice as many partitions. The target controls more than the number of scan groups: in this two-stage plan, it also sets the number of partitions the repartition operator creates for the final aggregate.\nIt pays not to overdo it, though. A higher target can mean more concurrent operators and more pressure on memory, and preserving the order already has a cost, the merge inside the repartition operator.\nTerminal recording (clips\/stats-grouping.cast) 7. Conclusions The order that matters to our GROUP BY is the one guaranteed within each partition it receives. To exploit the order stored in Parquet, DataFusion must be told that the order exists, establish that guarantee at the scan and preserve it through the intervening operators. In the original example, the guarantee fails at the scan: reading Monday and Tuesday in sequence takes the customer order from Zeno back to Anna.\nFile layout, the grouping of the files and the partition target determine whether the scan can produce sorted partitions. Adjusting them can preserve the order, but may also change parallelism and resource use. Further up the plan, carrying that order through a repartition requires merging streams.\nEXPLAIN shows whether the aggregate receives and uses a useful ordering. Measurement tells us whether the aggregation work it saves outweighs the cost of preserving it. [13]\nThe examples can be reproduced with datafusion-cli 55.1.0; the cost of order is measured in a small reproducible lab.\nReferences The settings and scan behavior described here refer to DataFusion’s listing-table path. Implementation references are pinned to the 55.1.0 release (tag 55.1.0).\nWITH ORDER and its correctness requirements, precedence of the declaration over the metadata and order read from sorting_columns. Goetz Graefe and William J. McKenna, The Volcano Optimizer Generator: Extensibility and Efficient Search, ICDE 1993: logical and physical algebra, with sort order as a physical property; in DataFusion, the logical scan becomes a physical operator. Andy Pavlo, Query Execution II, lecture notes 14 of 15-445\/645 Database Systems, Carnegie Mellon University, Fall 2024, section 5: intra-operator parallelism. Collection of file statistics and order, default grouping, statistics-based grouping, conditions for splitting files and splitting with and without order. File groups and sequential reads, when files can change partition and when that does not happen. Ordering validation, boundary comparison and orderings projected onto the columns the scan reads. Using Ordering for Better Plans in Apache DataFusion, 11 March 2025, aggregate mode and the order of its output and distribution required by the final aggregate. Order-preserving repartition and its runtime cost, when the variant is enabled, insertion with the ordered variant and when the planner keeps it. Ordered partial aggregate emission. Aggregation with ordered grouping keys, emission of totals in the final aggregate. Statistics-based grouping configuration. Group-count guard and debug log. Tuning memory-limited queries. ","permalink":"https:\/\/cdelmonte.dev\/explainers\/sorted-files-ordered-scan\/","summary":"A purchases table, one Parquet file per day, every file sorted by customer: you would expect a GROUP BY customer to use that order, yet EXPLAIN often shows it ignored. This explainer follows the ordering from the file scan to the aggregate in Apache DataFusion, shows where it gets lost, how to read it in the plan, and what you can change.","title":"Why Sorted Parquet Files Do Not Guarantee an Ordered Scan in DataFusion"},{"content":"Introduction Parquet does not always store a column’s values directly. When a column repeats many values, dictionary encoding stores the distinct values once and represents the column as a stream of small integer indices into that dictionary. Those indices are bit-packed, and reading the column requires unpacking them back into an int[] before any value can be looked up.\nThis explainer walks through that representation and the unpacking step. It is the background companion to Unpacking Parquet: Explicit SIMD, Scalar Baselines, and What HotSpot Makes of Them, which benchmarks and analyzes the unpacking step in depth.\nDictionary encoding A Parquet data page does not always contain the original column values directly.\nConsider a column of city names:\nBerlin Rome Berlin Paris Rome Berlin With dictionary encoding, Parquet stores the distinct values once and then refers to them by index:\n0 \u2192 Berlin 1 \u2192 Rome 2 \u2192 Paris Instead of repeating the city names, the encoded value stream contains their dictionary indices:\n0, 1, 0, 2, 1, 0 The original values may be strings, large integers, or any other supported type. The indices are just compact identifiers that point into the dictionary.\nBit-packing the indices The indices need only as many bits as the dictionary size requires. Because the dictionary above has three entries, two bits are enough for each index. The same mechanism scales: if a dictionary holds up to 8192 entries, its indices run from 0 to 8191 and need 13 bits each.\n2^13 = 8192 Those indices are packed one after another, without aligning each one to a byte boundary:\nindex 0 index 1 index 2 13 bits 13 bits 13 bits \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 The first index may start at bit 0, the second at bit 13, the third at bit 26, and so on. This packing is what makes the representation compact, and it is also what makes reading it back non-trivial: a value rarely starts on a byte boundary, and it may straddle two bytes.\nThe encoded-to-decoded boundary Reading the page runs the process in reverse:\nbit-packed dictionary indices \u2193 bit unpacking \u2193 int[] of dictionary indices \u2193 dictionary lookup \u2193 original column values The first transition is the one this background is about:\nbit-packed dictionary indices \u2193 bit unpacking \u2193 int[] of dictionary indices Each packed index has to be located, extracted, widened to a Java integer, and written into an output array, before the dictionary lookup can reconstruct the original values. This operation is called bit unpacking.\nFrom packed bits to integers Reconstructing the int[] of indices reverses the packing: a compact bit stream has to become an array of Java integers again.\nTo decode one value, a program generally has to:\ndetermine where the value starts in the bit stream; load enough bytes to contain it; shift away the bits that precede it; apply a mask to retain only the relevant bits; store the result as an integer. A simplified scalar decoder looks like this:\nlong bitPosition = 0; for (int i = 0; i < count; i++) { long bytePosition = bitPosition >>> 3; int bitOffset = (int) (bitPosition & 7); long word = source.get(LONG_LE, bytePosition); output[i] = (int) ((word >>> bitOffset) & mask); bitPosition += bitWidth; } The decoder processes one value at a time. At a width of 13 bits, the bit positions advance by a fixed increment, so the pattern is perfectly regular; but it is not a constant-stride, element-by-element array operation, and that is the distinction a compiler’s auto-vectorizer cares about.\nWhere this fits Bit unpacking is a small, hot kernel: a Parquet reader runs it for every dictionary-encoded page before it can look anything up. How fast it runs, and how much an explicit SIMD implementation helps over ordinary scalar Java, depends on the exact shape of that loop and on what the JVM’s compiler makes of it. That is the subject of Unpacking Parquet: Explicit SIMD, Scalar Baselines, and What HotSpot Makes of Them.\n","permalink":"https:\/\/cdelmonte.dev\/explainers\/parquet-bit-unpacking-explained\/","summary":"Parquet rarely stores a column’s values directly. Dictionary encoding keeps the distinct values once and represents the column through small bit-packed integer indices. This explainer walks through dictionary encoding, how the indices are bit-packed, and the bit-unpacking step that reconstructs them into an int[] before every dictionary lookup.","title":"From Dictionary Indices to Integers: Parquet Bit Unpacking Explained"},{"content":"Parsing arithmetic expressions looks simple\u2026 until precedence enters the picture. Why does * bind tighter than +, and how can a parser enforce these rules without complex grammar machinery? Two classic algorithms provide radically different answers: Dijkstra’s stack-based Shunting Yard and Pratt’s recursive Top-Down Operator Precedence. Side by side, they reveal the same underlying intuition expressed through two very different machines.\nWinter is the perfect time of year to rediscover vintage algorithms and savour them between one line of code and the next. It may seem odd at a time when everyone seems busy chasing the next big thing in software development; and yet, every now and then a walk outside the usual paths can only do good. Let us call it my hour of freedom.\nIt is precisely in this space of freedom that I stumbled upon the Shunting Yard, an algorithm described by Dijkstra in 1961 in the report ALGOL 60 Translation. In the literature, it has never been the subject of extensive treatment and, curiously, the original text assigns it no name at all. The nickname “Shunting Yard” has settled over time, more through teaching tradition than by the author’s designation.\nI am not sure why, but reading Dijkstra brought to mind another example of an algorithm that, while elegant and surprisingly modern, went for decades without much visibility in traditional compilation textbooks. I am referring to the Pratt parser, formally introduced by Vaughan R. Pratt in 1973 in the paper Top Down Operator Precedence, presented at the first POPL symposium.\nHere too the trajectory is curious: the idea never quite disappears, yet it remains at the margins of mainstream teaching, overshadowed by the dominance of LL and LR parsers. Only many years later is it rediscovered and brought to the attention of a new generation of developers, notably through Douglas Crockford’s popularising work and, more recently, through Robert Nystrom with Crafting Interpreters.\nIt is hard not to notice a certain historical symmetry: two algorithms born in eras when compilation theory was still taking shape, both elegantly simple, both left for a long time outside the main spotlight.\nPerhaps, even more than the historical context, it was the weight both place on precedence that led me to set them side by side. Barely a decade apart, Dijkstra and Pratt seem to answer, each in their own way, the same fundamental question: how to handle precedence rules in an expression in a systematic way?\nThe solutions are profoundly different in their tools, yet surprisingly akin in their goal. To clarify this parallelism, it is worth briefly telling the story of both algorithms, without resorting to heavy formalisms.\nThe Shunting Yard The algorithm assumes a small conceptual “machine” composed of four elements:\nan input tape containing the expression in infix notation; an operator stack, used as a waiting area; an output tape, on which the expression is built in RPN (Reverse Polish Notation); a precedence (and associativity) table that orders operators from least binding to most binding. The idea is simple: operands pass immediately to output, while operators are held on the stack until we are certain that no other operator with higher precedence (or equal precedence, in the case of left-associativity) needs to be emitted first.\nThe algorithm Scanning the input from left to right, for each token t:\nIf t is a number (or an identifier): write it to the output tape. If t is an operator op: while there is an operator top on top of the stack with precedence greater than or equal to op (left-associativity), move top from the stack to the output tape; then push op onto the stack. If t is (: push it onto the stack (it acts as a “barrier”). If t is ): move operators from the stack to the output tape until ( is encountered; remove ( from the stack (but do not write it to output). At the end of input: flush the stack, moving all remaining operators to output. This is the operational rule that, in effect, implements precedence: a * stays “suspended” above a +, and is not emitted until it becomes inevitable.\nWorked example Let us take the expression:\n1 + 2 * 4 - (10 + 1 * 2) For brevity, I denote the state as:\nOUT = output tape (RPN) STK = operator stack (top on the right) The main states, token by token:\nToken Action OUT STK \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 1 output 1 + push 1 + 2 output 1 2 + * push (prec > +) 1 2 + * 4 output 1 2 4 + * - pop *, pop +, push - 1 2 4 * + - ( push 1 2 4 * + - ( 10 output 1 2 4 * + 10 - ( + push 1 2 4 * + 10 - ( + 1 output 1 2 4 * + 10 1 - ( + * push 1 2 4 * + 10 1 - ( + * 2 output 1 2 4 * + 10 1 2 - ( + * ) pop *, pop +, remove ( 1 2 4 * + 10 1 2 * + - end flush stack: pop - 1 2 4 * + 10 1 2 * + - The result on the output tape is:\n1 2 4 * + 10 1 2 * + - This is simply the RPN form of the original infix expression, and it can in turn be evaluated by an even simpler machine: a value stack on which operators are applied as they appear.\nShunting Yard conceptual machine. Operator precedence is enforced by an explicit stack: operators are temporarily held in suspension and flushed to the output when precedence rules require it. The result is a flat RPN sequence ready for evaluation.\nShunting Yard conceptual machine. Operator precedence is enforced by an explicit stack: operators are temporarily held in suspension and flushed to the output when precedence rules require it. The result is a flat RPN sequence ready for evaluation. (Generated with AI tool by Christian Del Monte)\nThe Pratt Parser The method, introduced by Vaughan Pratt in 1973 under the name Top-Down Operator Precedence, rests on a simple and powerful idea: assign each operator a binding strength (the so-called binding power) and let that decide whether an operator belongs to the current expression or to a higher syntactic level. Where the Shunting Yard’s precedence governs when a stack must be flushed, the Pratt parser’s precedence governs how far the recursion is allowed to reach. This is a significant shift in perspective: instead of holding operators in suspension and releasing them at the right moment, the parser directly builds a syntactic structure in which precedence is already embedded in the shape of the tree.\nThe Pratt parser can be described as a surprisingly compact machine:\nan input tape with a cursor advancing over tokens, a central function parse_expr(min_bp) that drives the entire process, and a binding power table that assigns each operator a numeric pair (left_bp, right_bp). Unlike the Shunting Yard, there is no explicit operator stack, no structure accumulating symbols waiting to be released. Precedence is not implemented through push and pop, but as a numeric threshold that bounds the expansion of recursion. Each call to parse_expr establishes a boundary, min_bp, and only operators whose left binding power is strictly greater than this threshold can be absorbed into the current context; the rest are left to the level above. The hierarchy of operations thus emerges from the depth of the recursive calls, not from the discipline of an explicit stack.\nOf course, at the level of a concrete implementation, a stack does exist: it is the call stack of the recursion. But here the conceptual difference from the Shunting Yard becomes apparent: in the Shunting Yard the stack is an integral part of the algorithmic model and is manipulated directly; in the Pratt parser the stack is implicit, an artefact of top-down execution. In the Pratt parser, precedence does not discipline a pile of operators but delimits the syntactic region that can be constructed before returning control to the calling level. And it is precisely in this shift (from the temporal management of operators to the structural delimitation of recursion) that the comparison with Dijkstra becomes truly interesting.\nThe algorithm Let lhs be the expression produced by the initial token. Given a function parse_expr(min_bp):\nRead the next token t. Interpret t as an initial-position token (a number yields an immediate value; an opening parenthesis triggers a recursive sub-expression; a prefix minus builds a unary node). While the next token op is a binary operator with left_bp(op) > min_bp: Consume op. Compute r_bp = right_bp(op). Compute rhs = parse_expr(r_bp). Update lhs = Bin(op, lhs, rhs). Return lhs. Left-associativity arises naturally from choosing asymmetric pairs such as (10, 11) for + and - and (20, 21) for * and \/: when parse_expr is called recursively with min_bp = 11 and the next operator is another + with left_bp = 10, the condition 10 > 11 is false, so the second + is not consumed by the current level but handled by the outer one, yielding left-associativity with no explicit check. Were the condition >= instead of >, the operator would be absorbed by the current level, producing right-associative behavior.\nWorked example Let us trace the same expression as before through the Pratt parser:\n1 + 2 * 4 - (10 + 1 * 2) The table below includes an OUT_DBG column: a conceptual debug band that records the order in which values and operators are finalised. This is not part of the actual algorithm; it is here so we can compare it with the Shunting Yard’s RPN output.\nCTX Token Action lhs OUT_DBG \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 0 1 initial \u2192 number 1 1 0 + 10 > 0 \u2192 consume \u2014 1 11 2 initial \u2192 number 2 1 2 11 * 20 > 11 \u2192 consume \u2014 1 2 21 4 initial \u2192 number 4 1 2 4 21 - 10 > 21 false \u2192 return 4 1 2 4 11 (return) combine 2 * 4 (2*4) 1 2 4 * 11 - 10 > 11 false \u2192 return (2*4) 1 2 4 * 0 (return) combine 1 + (2*4) (1+(2*4)) 1 2 4 * + 0 - 10 > 0 \u2192 consume \u2014 1 2 4 * + 11 ( initial \u2192 parse_expr(0) \u2014 1 2 4 * + 0 10 initial \u2192 number 10 1 2 4 * + 10 0 + 10 > 0 \u2192 consume \u2014 1 2 4 * + 10 11 1 initial \u2192 number 1 1 2 4 * + 10 1 11 * 20 > 11 \u2192 consume \u2014 1 2 4 * + 10 1 21 2 initial \u2192 number 2 1 2 4 * + 10 1 2 21 ) stop \u2192 return 2 1 2 4 * + 10 1 2 11 (return) combine 1 * 2 (1*2) 1 2 4 * + 10 1 2 * 11 ) stop \u2192 return (1*2) 1 2 4 * + 10 1 2 * 0 (return) combine 10 + (1*2) (10+(1*2)) 1 2 4 * + 10 1 2 * + 11 (return) close (...), return to outer (10+(1*2)) 1 2 4 * + 10 1 2 * + 0 end return, combine lhs - rhs ((1+(2*4))-(10+(1*2))) 1 2 4 * + 10 1 2 * + - Final output AST (the true Pratt output):\n((1 + (2 * 4)) - (10 + (1 * 2))) OUT_DBG (conceptual debug band):\n1 2 4 * + 10 1 2 * + - Notice anything? It is identical to the RPN produced by the Shunting Yard:\nAlgorithm Real output Debug output \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 Shunting Yard RPN 1 2 4 * + 10 1 2 * + - Pratt AST ((1 + (2 * 4)) - (10 + (1 * 2))) Pratt (debug band) \u2014 1 2 4 * + 10 1 2 * + - Pratt (Top-Down Operator Precedence) conceptual machine. Operator precedence is encoded as binding power and enforced through recursion: instead of flushing a stack, the parser limits how far the recursive descent may expand. The result is a syntax tree where precedence is visible in the structure itself.\nPratt (Top-Down Operator Precedence) conceptual machine. Operator precedence is encoded as binding power and enforced through recursion: instead of flushing a stack, the parser limits how far the recursive descent may expand. The result is a syntax tree where precedence is visible in the structure itself. (Generated with AI tool by Christian Del Monte)\nClosing the Circle The parallel can now be seen in full. The Shunting Yard and the Pratt parser solve the same problem, giving an unambiguous order to operations in an expression, but they do so with profoundly different tools and philosophies. The Shunting Yard is an iterative machine: it scans tokens one by one, routes them between a stack and an output tape, and entrusts precedence to a numeric comparison that decides when to flush the stack. Its final product is a flat sequence in reverse Polish notation, free of parentheses and ready to be evaluated left to right. The Pratt parser is a recursive machine: each call to parse_expr carves out a fragment of the expression, and precedence governs not a stack but the depth to which the recursion is allowed to reach. Its final product is not a flat sequence but a syntax tree, in which the hierarchy of operations is already visible in the structure of the nodes.\nIn a sense, the Shunting Yard defers decisions, it holds operators back until their place is certain, while the Pratt parser anticipates them, building the correct structure as it goes. The explicit stack of the one and the recursion of the other are two different ways of keeping track of the same state: which operators are still pending and at what level of precedence they belong.\nIf one accepts this reading, the two algorithms are not competing alternatives but complementary viewpoints on the same problem, separated by a decade and reunited by the same insight: that precedence, when treated as data rather than as a grammatical rule, makes expression parsing far simpler than classical theory might suggest. Precedence is not merely a property of grammar, but a strategy for controlling evaluation structure.\nThis equivalence holds for fixed-precedence binary expressions, the domain explored here. Beyond it, the two designs begin to diverge in ergonomics rather than expressive power: Pratt parsing extends more naturally to prefix, postfix, and mixfix operators, while the Shunting Yard can support them only with additional machinery. The underlying intuition is shared; the engineering trade-offs are not.\nReferences Primary sources Dijkstra, E. W. (1961). ALGOL 60 Translation. Mathematisch Centrum Report MR 35. PDF Pratt, Vaughan R. (1973). Top Down Operator Precedence. Proceedings of the 1st ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages (POPL). ACM Secondary sources Crockford, Douglas (n.d.). Top Down Operator Precedence. Link Kladov, Alex (matklad) (2020). Simple but Powerful Pratt Parsing. Link Kladov, Alex (matklad) (2020). From Pratt to Dijkstra. Link Nystrom, Robert (2011). Pratt Parsers: Expression Parsing Made Easy. Link Nystrom, Robert (2021). Crafting Interpreters. Genever Benning. ","permalink":"https:\/\/cdelmonte.dev\/deep-dives\/shunting-yard-and-pratt-parsing\/","summary":"Parsing arithmetic expressions looks simple\u2026 until precedence enters the picture. Two classic algorithms, Dijkstra’s Shunting Yard and Pratt’s Top-Down Operator Precedence, provide radically different answers that reveal the same underlying intuition.","title":"Two Algorithms, One Intuition: Shunting Yard and Pratt Parsing"},{"content":"A Kubernetes pod needs to talk to a Google Cloud API. No static key, no secret mounted from a vault, just an annotation on a service account and a chain of token exchanges that most of us never see. Workload Identity makes this possible. This article takes the mechanism apart, piece by piece, and then puts it back together with tcpdump and Wireshark.\nMost GKE tutorials treat Workload Identity as a configuration recipe: annotate a Kubernetes Service Account (KSA), bind it to a GCP Service Account (GSA), and the system works. What remains opaque is why it works. What actually happens when a pod authenticates to a Google Cloud API without holding any long-lived credential? Who issues the tokens, who validates them, and how does a Kubernetes identity become a GCP identity?\nTo answer these questions, it helps to start from the basics. A KSA is a namespace-scoped object managed by the Kubernetes API server. A GSA is a project-level identity managed by Google Cloud IAM. The two live in entirely separate worlds. Workload Identity bridges them by letting a KSA impersonate a GSA: the KSA presents its own short-lived token, and GCP’s infrastructure exchanges it, through a series of hops, for a GSA token that grants access to cloud resources. No static JSON key ever touches the pod.\nThe rest of this article follows that chain from end to end: first the setup, then the theory behind namespace identity, then the token relay itself, and finally a hands-on demonstration with tcpdump.\nSetting the Stage: Creating Service Accounts and Configuring Workload Identity If you are already familiar with GKE Workload Identity configuration and want to go straight to the mechanism, skip to The Token Relay Race.\nWe begin by creating the resources needed to observe the interaction. After authenticating with gcloud auth login, we set a few environment variables:\nGCP_PROJECT_ID="<gcp_project_id>" GKE_CLUSTER_NAME="my-cluster-name" GSA="test-gsa" Create the GSA:\ngcloud iam service-accounts create ${GSA} --description="some description text" --display-name=${GSA} Create a GKE cluster with Workload Identity enabled. Workload Identity must be enabled at cluster creation time, or later via a cluster update that sets the workload pool:\ngcloud container clusters create ${GKE_CLUSTER_NAME} --workload-pool=${GCP_PROJECT_ID}.svc.id.goog If using an existing cluster, verify that Workload Identity is active:\ngcloud container clusters describe ${GKE_CLUSTER_NAME} --location=<cluster location> The output should include:\nworkloadIdentityConfig: workloadPool: "PROJECT_ID.svc.id.goog" Next, set up the Kubernetes side. Define K8S_NAMESPACE=<name of the namespace>, create the namespace if needed, and make it the default for the current context:\nkubectl create namespace ${K8S_NAMESPACE} kubectl config set-context --current --namespace ${K8S_NAMESPACE} Create the KSA and annotate it with its GSA counterpart (KSA="test-ksa"):\nkubectl create sa ${KSA} kubectl annotate sa ${KSA} iam.gke.io\/gcp-service-account=${GSA}@${GCP_PROJECT_ID}.iam.gserviceaccount.com The resulting manifest should look like this:\napiVersion: v1 kind: ServiceAccount metadata: annotations: iam.gke.io\/gcp-service-account: test-gsa@<gcp project id>.iam.gserviceaccount.com name: test-ksa namespace: <name of the namespace> Finally, bind the two identities with an IAM policy:\ngcloud iam service-accounts add-iam-policy-binding ${GSA}@${GCP_PROJECT_ID}.iam.gserviceaccount.com \\ --role roles\/iam.workloadIdentityUser \\ --member "serviceAccount:${GCP_PROJECT_ID}.svc.id.goog[${K8S_NAMESPACE}\/${KSA}]" That annotation on the KSA (iam.gke.io\/gcp-service-account) is doing most of the heavy lifting. It tells GKE’s metadata server which GSA this KSA is allowed to impersonate. To verify, deploy a test pod with serviceAccountName: test-ksa running gcloud auth list, the output should show the GSA as the active account.\nBefore moving on, it is worth pausing on the --member parameter from the policy binding: serviceAccount:${GCP_PROJECT_ID}.svc.id.goog[${K8S_NAMESPACE}\/${KSA}]. This string is an example of Namespace Identity, a concept that deserves a brief digression.\nExcursus: Namespace Identity and Identity Sameness When Workload Identity is enabled, GCP assigns the project a unique identity namespace of the form PROJECT_ID.svc.id.goog. Within that namespace, a KSA is identified as:\nPROJECT_ID.svc.id.goog[KUBERNETES_NAMESPACE\/KUBERNETES_SERVICE_ACCOUNT]\nwhere:\nPROJECT_ID.svc.id.goog is the identity namespace created by GKE; KUBERNETES_NAMESPACE is the namespace the KSA belongs to; KUBERNETES_SERVICE_ACCOUNT is the KSA name. This leads to a subtle property: because both clusters share the same workload identity pool (PROJECT_ID.svc.id.goog), if two different GKE clusters within the same project both contain a namespace with the same name and a KSA with the same name, IAM sees both KSAs as the same principal. Both can impersonate the same GSA.\nAn example makes this concrete. Imagine an organization with two smart buildings in different cities, each managed by a separate GKE cluster. Both buildings have a keyless access system. In each cluster there is a namespace called access and a KSA called keyless-entry, with permissions to reach the building’s authentication credentials.\nBecause both clusters share the project-level identity namespace PROJECT_ID.svc.id.goog, both KSAs resolve to the same identity: PROJECT_ID.svc.id.goog[access\/keyless-entry]. A single IAM binding is enough to grant both of them access to the same GSA, and by extension, to the same cloud resources. Employees authorised on that project can move between buildings without any additional configuration.\nThis is powerful for simplified management, but it also means that naming collisions across clusters are not harmless. Choosing namespace and KSA names deliberately is essential.\nDiagram illustrating namespace identity and identity sameness, created by Christian Del Monte using PlantUML.\nThe Token Relay Race Configuration is in place. The KSA can impersonate the GSA. But what actually happens at runtime? The answer is a chain of token exchanges: a relay race in which different servers pass different tokens from hand to hand.\nFour actors are involved. The Kubernetes API Server issues signed JWTs for KSAs via projected service account tokens. The GKE Metadata Server (GKE MDS) sits on every node and orchestrates the token exchange. It does not persist credentials, only brokers short-lived token exchanges on behalf of pods. Google’s Security Token Service (STS) and Cloud IAM handle the federation and impersonation steps. And the GCP Resource Server is the final destination: the cloud service the application actually wants to reach.\nRequesting a GSA Token The exchange proceeds as follows:\nKubernetes mounts a projected service account token into the pod at \/var\/run\/secrets\/kubernetes.io\/serviceaccount\/token. This is a short-lived JWT signed by the cluster’s OIDC issuer, containing the KSA’s identity and an expiration time. The pod does not request it. The kubelet provisions it automatically.\nThe application (or a client library like google-cloud-sdk) presents this KSA JWT to the GKE Metadata Server, requesting a GSA token.\nThe Metadata Server forwards the KSA JWT to Google’s Security Token Service (STS).\nSTS validates the JWT against the cluster’s OIDC issuer and, if valid, issues a federated access token.\nThe Metadata Server uses this federated token to request impersonation of the target GSA from Cloud IAM. IAM checks the Workload Identity binding (roles\/iam.workloadIdentityUser) and, if the KSA is authorized, issues an OAuth2 access token representing the GSA.\nThe Metadata Server returns the GSA access token to the application.\nUsing the GSA Token The application sends a request to the GCP Resource Server (Cloud Storage, BigQuery, etc.), including the GSA token as a Bearer token in the Authorization header.\nThe Resource Server validates the token against Google’s OAuth2 infrastructure, not the Metadata Server. The Metadata Server’s role ends at token issuance; it is not involved in verification. If the token is valid and carries sufficient IAM permissions, the Resource Server grants access to the requested resources.\nSequence diagram of the process created by Christian Del Monte using PlantUML.\nCapturing a GSA Token with tcpdump Theory is useful, but seeing the tokens move across the wire makes the mechanism tangible. The Metadata Server listens at 169.254.169.254 and the pod-to-metadata-server traffic is plain HTTP (not TLS), so the token exchange is directly observable. Traffic from the Metadata Server to Google’s backend (STS, IAM) is TLS-encrypted and not visible in the capture. The plan is straightforward:\nCapture traffic to and from 169.254.169.254 using tcpdump. Run a pod that triggers KSA-to-GSA impersonation. Inspect the capture and locate the GSA token. Verify the token against Google’s OAuth 2.0 endpoint. Step 1: Capture the Traffic Deploy a DaemonSet that runs tcpdump on every node:\napiVersion: apps\/v1 kind: DaemonSet metadata: name: tcpdump-daemonset spec: selector: matchLabels: name: tcpdump template: metadata: labels: name: tcpdump spec: hostNetwork: true containers: - name: tcpdump image: google\/cloud-sdk:slim command: [ "\/bin\/sh", "-c", "apt-get update && apt-get install -y tcpdump && tcpdump -i any -vv host 169.254.169.254 -w \/host\/tmp\/metadata.pcap", ] volumeMounts: - name: host-tmp mountPath: \/host\/tmp securityContext: capabilities: add: ["NET_ADMIN", "NET_RAW"] volumes: - name: host-tmp hostPath: path: \/tmp Step 2: Trigger the Impersonation With the DaemonSet running, start a pod that uses the test-ksa service account and accesses a Google Cloud service. This triggers the full token relay described above.\nStep 3: Inspect the Capture Download the .pcap file from one of the DaemonSet pods:\nkubectl cp <tcpdump-pod>:\/host\/tmp\/metadata.pcap .\/metadata.pcap Open the file in Wireshark and filter for the Metadata Server:\nip.addr == 169.254.169.254 Right-click on an HTTP request to 169.254.169.254, select “Follow” \u2192 “HTTP Stream”, and look for the JSON response containing the access_token field. That is the GSA token.\nScreenshot of Wireshark illustrating step 3 taken by Christian Del Monte.\nStep 4: Verify the Token Google’s OAuth 2.0 token info endpoint can confirm what the token represents:\ncurl -H "Authorization: Bearer <GCP_OAUTH2_TOKEN>" \\ https:\/\/www.googleapis.com\/oauth2\/v1\/tokeninfo The response should look like this:\n{ "issued_to": "12345678910234456656", "audience": "12345678910234456656", "scope": "https:\/\/www.googleapis.com\/auth\/userinfo.email https:\/\/www.googleapis.com\/auth\/cloud-platform", "expires_in": 1043, "email": "test-gsa@<your GCP project id>.iam.gserviceaccount.com", "verified_email": true, "access_type": "online" } The email field confirms that the token belongs to the GSA, the same one the KSA was configured to impersonate.\nAlternative: Getting a GSA Token with a Direct REST Call There is a simpler way to observe the same exchange without deploying tcpdump. Since the Metadata Server accepts requests from any pod on the node, we can call it directly using the KSA token that Kubernetes mounts into every pod.\nRetrieve the token:\nkubectl exec -it <pod-name> -- cat \/var\/run\/secrets\/kubernetes.io\/serviceaccount\/token Then request the GSA token:\nTOKEN=$(kubectl exec -it <pod-name> -- cat \/var\/run\/secrets\/kubernetes.io\/serviceaccount\/token) curl -H "Authorization: Bearer $TOKEN" \\ -H "Metadata-Flavor: Google" \\ "http:\/\/169.254.169.254\/computeMetadata\/v1\/instance\/service-accounts\/default\/token" The response contains a short-lived OAuth2 access token representing the GSA, the same token we captured with Wireshark. The difference is only in the method of observation; the underlying mechanism is identical.\nWorkload Identity is often described as “Kubernetes authentication for Google Cloud.” In practice it is a federated identity system built on OIDC token issuance, STS token exchange, and IAM impersonation, with a metadata server that orchestrates the chain from the node. A pod presents a Kubernetes identity and receives a GCP identity in return, with no static credential ever stored on disk. Understanding that architecture turns a black-box configuration into a debuggable system.\nDisclaimer: A newer feature called Workload Identity Federation for GKE allows KSAs to be referenced directly using a principal identifier in IAM policies without using impersonation. This is not the subject of this article.\nReferences Workload Identity Overview, Google Cloud Documentation Kubernetes Service Accounts, Kubernetes Documentation GCP IAM Service Accounts, Google Cloud Documentation IAM Policy Binding, Google Cloud Documentation tcpdump Wireshark ","permalink":"https:\/\/cdelmonte.dev\/deep-dives\/kubernetes-gcp-service-accounts-workload-identity\/","summary":"GKE Behind the Scenes: Understanding the Interaction Between Kubernetes and GCP Service Accounts Through The Metadata Server.","title":"Inside GKE Workload Identity: How Kubernetes Identities Become GCP Service Accounts"},{"content":"I’m Christian Del Monte, a software architect based in Berlin. This blog is about what happens beneath the abstractions of modern data systems: execution models, memory representations, storage formats, runtime behavior, and the infrastructure boundaries that shape security and reliability.\nMy perspective comes from designing and operating data-intensive systems, particularly with Spark and Delta Lake, and from examining how query engines and execution runtimes work internally. I care about implementation details when they reveal something larger about a system’s architecture, not as ends in themselves. My background in semiotics also influences how I look at representations, boundaries, and the assumptions embedded in technical systems.\nI publish two kinds of articles:\nEssays develop an architectural argument across systems and technologies, such as Where Spark Changes Shape. Deep Dives investigate a specific mechanism through code, benchmarks, execution plans, or implementation details, such as The Hidden DSL in Catalyst. I also contribute to open-source projects, write for German software-engineering publications including iX and Java Magazin, and speak at conferences such as W-JAX and Berlin Expert Days.\nGitHub \u00b7 LinkedIn \u00b7 Selected Work \u00b7 RSS\n","permalink":"https:\/\/cdelmonte.dev\/about\/","summary":"About Christian Del Monte","title":"About"},{"content":"Verantwortlicher Christian Del Monte Esmarchstrasse 2 10407 Berlin Deutschland\nE-Mail: christian.del-monte@posteo.de\nAllgemeines zur Datenverarbeitung Diese Website ist eine statische Website ohne Benutzerkonten, Kontaktformulare oder Kommentarfunktionen. Es werden keine personenbezogenen Daten direkt durch die Website erhoben oder gespeichert.\nHosting Diese Website wird \u00fcber Cloudflare bereitgestellt. Beim Aufruf der Website werden technisch bedingt Verbindungsdaten (insbesondere die IP-Adresse des anfragenden Ger\u00e4ts) von Cloudflare verarbeitet. Diese Verarbeitung ist erforderlich, um die Website auszuliefern, und erfolgt auf Grundlage von Art. 6 Abs. 1 lit. f DSGVO (berechtigtes Interesse an der zuverl\u00e4ssigen Bereitstellung der Website).\nCloudflare, Inc. hat seinen Sitz in den USA. Die Daten\u00fcbermittlung in die USA erfolgt auf Grundlage der von Cloudflare implementierten Standardvertragsklauseln gem\u00e4\u00df Art. 46 Abs. 2 lit. c DSGVO.\nWeitere Informationen zum Datenschutz bei Cloudflare: https:\/\/www.cloudflare.com\/privacypolicy\/\nWebanalyse Diese Website verwendet Cloudflare Web Analytics. Dabei handelt es sich um einen Analysedienst, der keine Cookies setzt und keine personenbezogenen Daten einzelner Besucher speichert. Es werden ausschlie\u00dflich aggregierte Daten \u00fcber Seitenaufrufe erhoben (z. B. aufgerufene Seiten, Herkunftsland, verwendeter Browser). Eine Identifikation einzelner Personen ist nicht m\u00f6glich.\nDie Nutzung erfolgt auf Grundlage von Art. 6 Abs. 1 lit. f DSGVO (berechtigtes Interesse an der anonymen Auswertung des Nutzungsverhaltens zur Verbesserung der Website).\nCookies Diese Website setzt keine Cookies.\nIhre Rechte Sie haben gegen\u00fcber dem Verantwortlichen folgende Rechte in Bezug auf Ihre personenbezogenen Daten:\nRecht auf Auskunft (Art. 15 DSGVO) Recht auf Berichtigung (Art. 16 DSGVO) Recht auf L\u00f6schung (Art. 17 DSGVO) Recht auf Einschr\u00e4nkung der Verarbeitung (Art. 18 DSGVO) Recht auf Widerspruch gegen die Verarbeitung (Art. 21 DSGVO) Dar\u00fcber hinaus haben Sie das Recht, sich bei einer Datenschutz-Aufsichtsbeh\u00f6rde zu beschweren. Die zust\u00e4ndige Aufsichtsbeh\u00f6rde ist:\nBerliner Beauftragte f\u00fcr Datenschutz und Informationsfreiheit Friedrichstr. 219 10969 Berlin https:\/\/www.datenschutz-berlin.de\n","permalink":"https:\/\/cdelmonte.dev\/datenschutz\/","summary":"Datenschutzerkl\u00e4rung","title":"Datenschutzerkl\u00e4rung"},{"content":"Angaben gem\u00e4\u00df \u00a7 5 TMG Christian Del Monte Esmarchstrasse 2 10407 Berlin Deutschland\nKontakt E-Mail: christian.del-monte@posteo.de\nHaftungsausschluss Haftung f\u00fcr Inhalte Die Inhalte dieser Seiten wurden mit gr\u00f6\u00dfter Sorgfalt erstellt. F\u00fcr die Richtigkeit, Vollst\u00e4ndigkeit und Aktualit\u00e4t der Inhalte kann ich jedoch keine Gew\u00e4hr \u00fcbernehmen. Als Diensteanbieter bin ich gem\u00e4\u00df \u00a7 7 Abs.1 TMG f\u00fcr eigene Inhalte auf diesen Seiten nach den allgemeinen Gesetzen verantwortlich. Nach \u00a7\u00a7 8 bis 10 TMG bin ich als Diensteanbieter jedoch nicht verpflichtet, \u00fcbermittelte oder gespeicherte fremde Informationen zu \u00fcberwachen oder nach Umst\u00e4nden zu forschen, die auf eine rechtswidrige T\u00e4tigkeit hinweisen.\nHaftung f\u00fcr Links Diese Website enth\u00e4lt Links zu externen Websites Dritter, auf deren Inhalte ich keinen Einfluss habe. F\u00fcr die Inhalte der verlinkten Seiten ist stets der jeweilige Anbieter oder Betreiber der Seiten verantwortlich.\n","permalink":"https:\/\/cdelmonte.dev\/impressum\/","summary":"Impressum","title":"Impressum"},{"content":"A selection of work published outside this blog: articles for German software-engineering magazines, conference talks, and open-source contributions.\nWriting Mehr Flexibilit\u00e4t in der Softwareentwicklung mit wenig bekannten Design Patterns. Six-part series in Java Magazin, 2025. A tour of lesser-known design patterns and what they buy you in everyday software design.\nTeil 1: Exoten aus der Nische (Java Magazin 1.2025). How niche design patterns offer innovative solutions for highly configurable, data-driven systems. Teil 2: Erzeugen und trennen (Java Magazin 2.2025). Type Object Pattern and Dynamic Object Model for separating type definitions from concrete instances. Read. Teil 3: Beweglich bleiben (Java Magazin 4.2025). Exotic structural patterns, Extension Object and Entity-Component-System, for modularity and scalability in complex projects. Read. Teil 4: Multi-Methods und Specification Pattern (Java Magazin 5.2025). Behavioral patterns for complex interactions and ever-changing business rules. Teil 5: Ressourcen verwalten mit HOPP (Java Magazin 6.2025). The Half-Object Plus Protocol (HOPP) for managing resources and cutting remote-communication overhead in distributed architectures. Read. Teil 6: Rollen- und Attributmuster (Java Magazin 8.2025). Role Object and Property Pattern for decoupling behavior and state at runtime. Read. Datenverwaltung: Datenqualit\u00e4t von Data Lakes sichern. iX 2\/2024, p. 102 (Heise). The role of metadata in safeguarding data quality in data lakes, and how OpenTelemetry helps monitor metadata quality. Read.\nDie Herausforderung der Big Data Verarbeitung in objektorientierten Systemen. Spieleentwicklung als Inspirationsquelle. Dev-Insider, 2024. Processing big data in object-oriented systems, with ideas borrowed from game development: the Visitor and Dynamic Object Model patterns and the Entity-Component-System (ECS) architecture. Read.\nBig Data Meets Data Lake: Big Data, Their Challenges and Data Lake Architectures as a Solution. JAX Magazine, Volume 21, 2023. The challenges of big data and how data-lake architectures address them. Read.\nDas Rauschen in der Softwareentwicklung verhindern. ComputerWeekly.de, 2022. The cost of noise in decision-making across the software industry and how to prevent it, inspired by Noise: A Flaw in Human Judgment (Kahneman, Sibony, Sunstein). Read.\nTalks Data Lakehouse und Open Table Format: eine neue Architektur f\u00fcr Big Data. Berlin Expert Days, 29 September 2023. A new architecture for big data built on the lakehouse model and open table formats.\nData Lakes: Architekturen und Data Discovery Strategien. W-JAX, Munich, November 2022. Architectural patterns for data lakes and strategies for data discovery.\nData Lakes: Architekturen. techcamp, Hamburg, 2022. Architectural patterns for building data lakes.\nEvolution\u00e4re Architektur: Eine Einf\u00fchrung. techcamp, Hamburg, 2020. An introduction to evolutionary software architecture.\nOpen Source Bitbucket Push and Pull Request Plugin. Author and maintainer of the Jenkins plugin that triggers job builds on Bitbucket push and pull-request events. Plugin \u00b7 Repo.\ndelta-explain. A CLI diagnostic tool that shows, step by step, how Delta Lake prunes files for a given predicate. crates.io \u00b7 Repo.\nnanoidp. A lightweight, configurable identity provider for development and testing, supporting OAuth2\/OIDC and SAML 2.0 with a full web UI. PyPI \u00b7 Repo.\nArticle companion projects. Reproducible benchmarks behind posts on this blog:\ndecode-shape. A benchmark lab measuring where columnar integer bit-unpacking is SIMD-friendly and where it resists vectorization (JDK 25, Vector API). Repo. ","permalink":"https:\/\/cdelmonte.dev\/selected-work\/","summary":"Selected articles, talks, and open-source work by Christian Del Monte","title":"Selected Work"}]