You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3568ed4
Browse filesBrowse the repository at this point in the historyBrowse files
- Keep only the description of the current page, so that the page and its decompressor stream
are not kept alive.
- Read the dictionary page in the same try as initDictionary, and narrow its comment.
- Reject a DELTA_BINARY_PACKED mini block bit width above 64, and drop a redundant check.
- Test the page in the batch error message, the mini block size checks and the bit width check;
share the test helpers and simplify the ZSTD test.
Co-authored-by: Isaac <[email protected]>
Copy file name to clipboardExpand all lines: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
Copy file name to clipboardExpand all lines: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetDeltaEncodingSuite.scala
+19Lines changed: 19 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -204,6 +204,25 @@ abstract class ParquetDeltaEncodingSuite[T] extends ParquetCompatibilityTest
204
204
reader.initFromPage(100, craftHeader(128, 4, -1))
205
205
}
206
206
assert(e3.getMessage.contains("total value count"))
207
+
208
+
// The mini blocks must have a multiple of 8 values: 128 values do not split into 3 mini
209
+
// blocks, and split into 32 mini blocks of 4 values.
0 commit comments