Skip to content

Commit f095057

Browse files
andygrovepeter-toth
authored andcommitted
[SPARK-60262][SQL] Fail Parquet reads where two requested fields select the same column
### What changes were proposed in this pull request? With `spark.sql.parquet.fieldId.read.enabled=true`, `ParquetReadSupport.clipParquetGroupFields` resolves each requested field on its own: a field with a field id by id, and any other field by name. Two requested fields can therefore be clipped to the same Parquet field, and neither reader can read a column twice (see below). This PR makes such a read fail with a clear error instead of returning wrong values: - `clipParquetGroupFields` throws the new `PARQUET_FIELD_REQUESTED_MORE_THAN_ONCE` error when two or more requested fields of a struct are clipped to the same Parquet field. That covers a field id match plus a name match, and two fields with the same id, at the top level and inside nested structs. The check is in the clipping code that both readers use, so it covers the vectorized reader and the parquet-mr reader. - `VectorizedColumnReader.readBatch` throws a `ParquetDecodingException` when a column without repeated values runs out of pages with rows still to read, instead of returning. With the check above nothing in Spark reaches this, but returning leaves the rest of the vector unwritten without any signal, so this keeps a column that is ever read twice, or a truncated column chunk, from quietly returning wrong values. Repeated columns still stop at the end of the pages, because the end of a list is not known until then. ### Why are the changes needed? Take a file column `A` with field id 1. A requested `x` with id 1 selects it by id, and a requested `a` without an id selects it by name, so the requested Parquet schema lists `A` twice. Neither reader returns `A`'s values for both fields, and neither reports an error: - The vectorized reader builds one `VectorizedColumnReader` per requested leaf, but a Parquet page store has a single queue of pages per column. The first reader drains it. For the second, `readPage()` returns -1 and `readBatch` stops as if it had read every page, so its vector is never written, and its rows read as non-null `0` with on-heap vectors, or as whatever the memory holds with off-heap vectors. - The parquet-mr reader (`spark.sql.parquet.enableVectorizedReader=false`) has `ParquetRowConverter` map the clipped Parquet fields back to the requested fields by id first. Both copies of `A` carry id 1, so both map to `x`, and `a` reads null. ```scala val id1 = new MetadataBuilder().putLong("parquet.field.id", 1L).build() spark.createDataFrame( spark.sparkContext.parallelize(Seq(Row(1), Row(2), Row(3)), 1), StructType(Seq(StructField("A", IntegerType, nullable = true, id1)))) .write.parquet("/tmp/field-id-twice") spark.conf.set("spark.sql.parquet.fieldId.read.enabled", "true") val x = StructField("x", IntegerType, nullable = true, id1) val a = StructField("a", IntegerType, nullable = true) spark.read.schema(StructType(Seq(x, a))).parquet("/tmp/field-id-twice").show() ``` Before this change, `x` reads 1, 2, 3 and `a` reads 0, 0, 0 with the vectorized reader (the other way round when `a` comes first), and `a` reads null with the parquet-mr reader. A struct holding `x` and `a` behaves the same. Reading both fields correctly would take changes to how both readers map clipped fields back to the requested ones, and a way for the vectorized reader to serve one column to several leaves. This PR is the smaller step of replacing the silent wrong results with an error. Field-id reads (SPARK-38094) have returned these results since 3.3. ### Does this PR introduce _any_ user-facing change? Yes. Reading a schema in which two requested fields select the same Parquet column, which returned wrong values without an error (see above), now fails with `PARQUET_FIELD_REQUESTED_MORE_THAN_ONCE`. The wrong results come from field-id reads, so released versions since 3.3 have them as well. Reads that do not select a column more than once are not affected. ### How was this patch tested? New tests: - `ParquetFieldIdSchemaSuite`: `ParquetReadSupport.clipParquetSchema` fails, naming the requested fields and the Parquet field, for an id match plus a name match in either order (case-sensitive and not), two fields with the same id, a third field matching by name, and a nested struct. It does not fail when the fields select different Parquet fields, or none. - `ParquetFieldIdIOSuite`: reads with the vectorized and the parquet-mr reader fail for the same shapes (top level, nested, same id), and a requested field that matches no column still reads as null. - `ParquetVectorizedSuite`: a requested schema that lists a column twice, over a page store that shares one page reader per column as Parquet's does, fails in `VectorizedColumnReader.readBatch` instead of leaving the second vector unwritten. The new tests fail without this change (no exception is thrown, and the data reads back wrong) and pass with it. The `ParquetVectorizedSuite` test also fails when only the `readBatch` check is removed. Also ran, all passing: the packages `org.apache.spark.sql.execution.datasources.parquet` and `org.apache.spark.sql.execution.datasources.v2.parquet` with `FileBasedDataSourceSuite` (55 suites, 2075 tests passed, 7 ignored), and `SparkThrowableSuite`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (claude-sonnet-5-5) Closes #59510 from andygrove/SPARK-60262. Authored-by: Andy Grove <[email protected]> Signed-off-by: Peter Toth <[email protected]>
1 parent f6eaea6 commit f095057

8 files changed

Lines changed: 244 additions & 2 deletions

File tree

‎common/utils/src/main/resources/error/error-conditions.json‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6730,6 +6730,12 @@
67306730
},
67316731
"sqlState" : "42846"
67326732
},
6733+
"PARQUET_FIELD_REQUESTED_MORE_THAN_ONCE" : {
6734+
"message" : [
6735+
"Requested fields <requestedFields> resolve to the same Parquet field <parquetField>, which can be read only once. When <fieldIdReadConf> is enabled, a requested field with a field ID is resolved by ID and any other requested field by name. Request the Parquet field only once, and refer to the resulting column more than once in the query instead."
6736+
],
6737+
"sqlState" : "KD000"
6738+
},
67336739
"PARQUET_TYPE_ILLEGAL" : {
67346740
"message" : [
67356741
"Illegal Parquet type: <parquetType>."

‎docs/sql-migration-guide.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ license: |
4848
- Since Spark 4.4, `spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled` defaults to `true`. A V2 scan that reports a keyed partitioning but no explicit ordering now also reports itself sorted by its partition key expressions, because every row of such a partition evaluates them to the same value. Partition transforms such as `days(ts)` or `bucket(8, id)` are left out of that ordering. This lets Spark drop a `Sort` it would otherwise place above the scan. An aggregate that groups by a leading run of the other key expressions, in key order, may also be planned as a sort aggregate rather than a hash aggregate. This is because `spark.sql.execution.replaceHashWithSortAgg` keys off the reported ordering. To restore the previous behavior, set `spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled` to `false`.
4949
- Since Spark 4.4, `spark.sql.sources.v2.bucketing.preserveKeyOrderingOnCoalesce.enabled` defaults to `true`. When `GroupPartitionsExec` merges several input partitions that share one partition key value into a single output partition, it now keeps sort orders over the partition key expressions in the ordering it reports, since those expressions are constant within the merged partition. Sort orders over partition transforms such as `days(ts)` or `bucket(8, id)` are left out. Orders over other columns are still dropped, as the concatenation invalidates them, and a join that reduced the partition keys onto a common key space (see `spark.sql.sources.v2.bucketing.allowCompatibleTransforms.enabled`) reports no order, since the merged partitions then share only the reduced key. This can remove a `Sort` downstream of the merge. To restore the previous behavior, set `spark.sql.sources.v2.bucketing.preserveKeyOrderingOnCoalesce.enabled` to `false`.
5050
- Since Spark 4.4, the value columns of `UNPIVOT` and `Dataset.unpivot` merge the nested nullability (array element, map value, and struct field nullability) of all unpivoted values. Previously, a value column took the nested nullability of the first listed value, so a `NULL` array element, map value, or struct field of a later value could be returned as a non-`NULL` value such as `0`, and `IS NULL` on it returned `false`. A view created by an earlier Spark version over such an `UNPIVOT` may have recorded the non-nullable nested type in its schema and can fail to read; recreate it with `CREATE OR REPLACE VIEW`. To restore the previous behavior, set `spark.sql.legacy.unpivot.useFirstValueNestedNullability` to `true`.
51+
- Since Spark 4.4, when `spark.sql.parquet.fieldId.read.enabled` is true, a Parquet read fails with `PARQUET_FIELD_REQUESTED_MORE_THAN_ONCE` if two or more requested fields of a struct (or of the top-level schema) resolve to the same Parquet field. This happens when one requested field is matched by field ID and another by name, or when two requested fields have the same field ID. In Spark 4.3 and earlier, such a read returned wrong values without an error: with the vectorized reader one of the fields read as non-null default values (for example `0`) or uninitialized data, and with the parquet-mr reader (`spark.sql.parquet.enableVectorizedReader` set to `false`) one of the fields read as `null`. To read the Parquet field more than once, request it once and refer to the resulting column more than once in the query.
5152
- Since Spark 4.4, when `spark.sql.groupByOrdinal` is true (the default), the pipe `AGGREGATE` operator rejects an aliased integer literal in its `GROUP BY` clause, such as `TABLE t |> AGGREGATE sum(x) AS s GROUP BY 2 AS grp`, with `UNSUPPORTED_FEATURE.PIPE_OPERATOR_AGGREGATE_UNSUPPORTED_CASE`, because a bare integer literal in the same position is an ordinal. Previously the aliased literal was grouped as a constant. Bare ordinals and aliases on other grouping expressions are not affected. To restore the previous behavior, set `spark.sql.legacy.allowAliasedPipeAggregateOrdinals` to `true`.
5253

5354
## Upgrading from Spark SQL 4.2 to 4.3

‎sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1150,6 +1150,16 @@ private[sql] object QueryExecutionErrors extends QueryErrorsBase with ExecutionE
11501150
"matchedFields" -> matchedFields))
11511151
}
11521152

1153+
def parquetFieldRequestedMoreThanOnceError(
1154+
requestedFields: Seq[String], parquetField: String): SparkRuntimeException = {
1155+
new SparkRuntimeException(
1156+
errorClass = "PARQUET_FIELD_REQUESTED_MORE_THAN_ONCE",
1157+
messageParameters = Map(
1158+
"requestedFields" -> requestedFields.map(f => toSQLId(Seq(f))).mkString(", "),
1159+
"parquetField" -> toSQLId(Seq(parquetField)),
1160+
"fieldIdReadConf" -> toSQLConf(SQLConf.PARQUET_FIELD_ID_READ_ENABLED.key)))
1161+
}
1162+
11531163
def failedToMergeIncompatibleSchemasError(
11541164
left: StructType, right: StructType, e: Throwable = null): Throwable = {
11551165
new SparkException(

‎sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -257,6 +257,15 @@ void readBatch(
257257
if (readState.valuesToReadInPage == 0) {
258258
int pageValueCount = readPage();
259259
if (pageValueCount < 0) {
260+
if (readState.maxRepetitionLevel == 0) {
261+
// A column without repeated values has a value, possibly null, for every row. So
262+
// running out of pages with rows left to read means the column chunk holds fewer
263+
// values than expected or another reader of the column took its pages, and returning
264+
// would leave the rest of the vector unwritten.
265+
throw new ParquetDecodingException(String.format(
266+
"Column %s ran out of pages with %d of the %d rows of the batch left to read",
267+
descriptor, readState.rowsToReadInBatch, total));
268+
}
260269
// we've read all the pages; this could happen when we're reading a repeated list and we
261270
// don't know where the list will end until we've seen all the pages.
262271
break;

‎sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -507,6 +507,7 @@ object ParquetReadSupport extends Logging {
507507
matchCaseInsensitiveField(f)
508508
}
509509
}
510+
checkNoFieldRequestedMoreThanOnce(structType, clippedType)
510511
// Ignore MessageType, because it is the root of the schema, not a struct.
511512
if (returnNullStructIfAllFieldsMissing || !isStructWithMissingAllFields ||
512513
parquetRecord.isInstanceOf[MessageType]) {
@@ -517,6 +518,31 @@ object ParquetReadSupport extends Logging {
517518
}
518519
}
519520

521+
/**
522+
* Fails if two or more requested fields were clipped to the same Parquet field. That happens
523+
* when one is looked up by field id and another by name, or when both are looked up by the same
524+
* id. The requested Parquet schema would then list the field twice, but a column can be read
525+
* only once: the vectorized reader finds no pages left for the second copy and leaves its vector
526+
* unwritten, and the parquet-mr reader maps both copies back to the same requested field.
527+
*
528+
* The fields of a Parquet group have distinct names, so clipped fields with the same name are
529+
* the same field.
530+
*/
531+
private def checkNoFieldRequestedMoreThanOnce(
532+
structType: StructType,
533+
clippedFields: Seq[Type]): Unit = {
534+
val clippedNames = new util.HashSet[String]()
535+
clippedFields.foreach { clipped =>
536+
if (!clippedNames.add(clipped.getName)) {
537+
val requestedFields = structType.iterator.zip(clippedFields.iterator)
538+
.collect { case (requested, c) if c.getName == clipped.getName => requested.name }
539+
.toSeq
540+
throw QueryExecutionErrors.parquetFieldRequestedMoreThanOnceError(
541+
requestedFields, clipped.getName)
542+
}
543+
}
544+
}
545+
520546
/**
521547
* Finds the leaf node(s) under a given file schema node that is likely to be cheapest to fetch.
522548
* Note that multiple leaves can be selected if a map type is deemed to be the cheapest to fetch,

‎sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFieldIdIOSuite.scala‎

Lines changed: 87 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,8 @@ package org.apache.spark.sql.execution.datasources.parquet
1919

2020
import scala.jdk.CollectionConverters._
2121

22-
import org.apache.spark.SparkException
23-
import org.apache.spark.sql.Row
22+
import org.apache.spark.{SparkException, SparkRuntimeException}
23+
import org.apache.spark.sql.{DataFrame, Row}
2424
import org.apache.spark.sql.internal.SQLConf
2525
import org.apache.spark.sql.test.SharedSparkSession
2626
import org.apache.spark.sql.types._
@@ -266,4 +266,89 @@ class ParquetFieldIdIOSuite extends ParquetTest with SharedSparkSession {
266266
}
267267
}
268268
}
269+
270+
private def checkFieldRequestedMoreThanOnce(
271+
df: => DataFrame,
272+
requestedFields: Seq[String],
273+
parquetField: String): Unit = {
274+
// The reader raises the error on the executor, so it is the cause of the job failure.
275+
val cause = intercept[SparkException](df.collect()).getCause
276+
checkError(
277+
exception = cause.asInstanceOf[SparkRuntimeException],
278+
condition = "PARQUET_FIELD_REQUESTED_MORE_THAN_ONCE",
279+
parameters = Map(
280+
"requestedFields" -> requestedFields.map(f => s"`$f`").mkString(", "),
281+
"parquetField" -> s"`$parquetField`",
282+
"fieldIdReadConf" -> "\"spark.sql.parquet.fieldId.read.enabled\""))
283+
}
284+
285+
test("SPARK-60262: a field id match and a name match must not select the same column") {
286+
withTempDir { dir =>
287+
// The file column `A` has field id 1. A requested field with id 1 selects it by id, and a
288+
// requested field without an id selects it by name.
289+
val writeSchema = new StructType().add("A", IntegerType, true, withId(1))
290+
spark.createDataFrame(Seq(Row(1), Row(2), Row(3)).asJava, writeSchema)
291+
.write.mode("overwrite").parquet(dir.getCanonicalPath)
292+
293+
val byId = StructField("x", IntegerType, true, withId(1))
294+
Seq("false" -> "a", "true" -> "A").foreach { case (caseSensitive, nameOfA) =>
295+
withSQLConf(SQLConf.CASE_SENSITIVE.key -> caseSensitive) {
296+
val byName = StructField(nameOfA, IntegerType, true)
297+
withAllParquetReaders {
298+
Seq(Seq(byId, byName), Seq(byName, byId)).foreach { fields =>
299+
checkFieldRequestedMoreThanOnce(
300+
spark.read.schema(StructType(fields)).parquet(dir.getCanonicalPath),
301+
fields.map(_.name),
302+
"A")
303+
}
304+
}
305+
}
306+
}
307+
308+
// A requested field that matches no column does not select the column either.
309+
withAllParquetReaders {
310+
checkAnswer(
311+
spark.read.schema(new StructType().add(byId).add("b", IntegerType, true))
312+
.parquet(dir.getCanonicalPath),
313+
Seq(Row(1, null), Row(2, null), Row(3, null)))
314+
}
315+
}
316+
}
317+
318+
test("SPARK-60262: a field id match and a name match must not select the same nested column") {
319+
withTempDir { dir =>
320+
val writeSchema = new StructType().add(
321+
"s", new StructType().add("A", IntegerType, true, withId(2)), true, withId(1))
322+
spark.createDataFrame(Seq(Row(Row(1)), Row(Row(2)), Row(Row(3))).asJava, writeSchema)
323+
.write.mode("overwrite").parquet(dir.getCanonicalPath)
324+
325+
val readSchema = new StructType().add(
326+
"s",
327+
new StructType()
328+
.add("x", IntegerType, true, withId(2))
329+
.add("a", IntegerType, true),
330+
true,
331+
withId(1))
332+
withAllParquetReaders {
333+
checkFieldRequestedMoreThanOnce(
334+
spark.read.schema(readSchema).parquet(dir.getCanonicalPath), Seq("x", "a"), "A")
335+
}
336+
}
337+
}
338+
339+
test("SPARK-60262: two fields with the same field id must not select the same column") {
340+
withTempDir { dir =>
341+
val writeSchema = new StructType().add("A", IntegerType, true, withId(1))
342+
spark.createDataFrame(Seq(Row(1), Row(2), Row(3)).asJava, writeSchema)
343+
.write.mode("overwrite").parquet(dir.getCanonicalPath)
344+
345+
val readSchema = new StructType()
346+
.add("x", IntegerType, true, withId(1))
347+
.add("y", IntegerType, true, withId(1))
348+
withAllParquetReaders {
349+
checkFieldRequestedMoreThanOnce(
350+
spark.read.schema(readSchema).parquet(dir.getCanonicalPath), Seq("x", "y"), "A")
351+
}
352+
}
353+
}
269354
}

‎sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFieldIdSchemaSuite.scala‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ import scala.jdk.CollectionConverters._
2121

2222
import org.apache.parquet.schema.{MessageType, MessageTypeParser}
2323

24+
import org.apache.spark.SparkRuntimeException
2425
import org.apache.spark.sql.internal.SQLConf
2526
import org.apache.spark.sql.types._
2627

@@ -526,4 +527,74 @@ class ParquetFieldIdSchemaSuite extends ParquetSchemaTest {
526527
| }
527528
|}
528529
""".stripMargin)
530+
531+
test("SPARK-60262: clipping with field id fails on a Parquet field requested twice") {
532+
val fileSchema = MessageTypeParser.parseMessageType(
533+
"""message root {
534+
| optional int32 A = 1;
535+
| optional group s = 2 {
536+
| optional int32 B = 3;
537+
| }
538+
| optional int32 C = 4;
539+
|}
540+
""".stripMargin)
541+
542+
def clip(catalystSchema: StructType, caseSensitive: Boolean = false): MessageType =
543+
ParquetReadSupport.clipParquetSchema(
544+
fileSchema,
545+
catalystSchema,
546+
caseSensitive = caseSensitive,
547+
useFieldId = true,
548+
returnNullStructIfAllFieldsMissing = false)
549+
550+
def checkFails(
551+
catalystSchema: StructType,
552+
requestedFields: String,
553+
parquetField: String,
554+
caseSensitive: Boolean = false): Unit = {
555+
checkError(
556+
exception = intercept[SparkRuntimeException](clip(catalystSchema, caseSensitive)),
557+
condition = "PARQUET_FIELD_REQUESTED_MORE_THAN_ONCE",
558+
parameters = Map(
559+
"requestedFields" -> requestedFields,
560+
"parquetField" -> parquetField,
561+
"fieldIdReadConf" -> "\"spark.sql.parquet.fieldId.read.enabled\""))
562+
}
563+
564+
// One field selects the Parquet field by id and the other by name, in either order.
565+
checkFails(
566+
new StructType().add("x", IntegerType, true, withId(1)).add("a", IntegerType, true),
567+
"`x`, `a`", "`A`")
568+
checkFails(
569+
new StructType().add("a", IntegerType, true).add("x", IntegerType, true, withId(1)),
570+
"`a`, `x`", "`A`")
571+
checkFails(
572+
new StructType().add("x", IntegerType, true, withId(1)).add("A", IntegerType, true),
573+
"`x`, `A`", "`A`", caseSensitive = true)
574+
575+
// Both select it by id, with an unrelated field between them.
576+
checkFails(
577+
new StructType()
578+
.add("x", IntegerType, true, withId(1))
579+
.add("c", IntegerType, true)
580+
.add("y", IntegerType, true, withId(1)),
581+
"`x`, `y`", "`A`")
582+
// A third field selects it by name as well, and all three are reported.
583+
checkFails(
584+
new StructType()
585+
.add("x", IntegerType, true, withId(1))
586+
.add("y", IntegerType, true, withId(1))
587+
.add("a", IntegerType, true),
588+
"`x`, `y`, `a`", "`A`")
589+
590+
// The fields of a nested struct are clipped on their own.
591+
checkFails(
592+
new StructType().add(
593+
"s", new StructType().add("x", IntegerType, true, withId(3)).add("b", IntegerType), true),
594+
"`x`, `b`", "`B`")
595+
596+
// Fields that select different Parquet fields, or none, are fine.
597+
clip(new StructType().add("x", IntegerType, true, withId(1)).add("c", IntegerType, true))
598+
clip(new StructType().add("x", IntegerType, true, withId(1)).add("z", IntegerType, true))
599+
}
529600
}

‎sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorizedSuite.scala‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -551,6 +551,40 @@ class ParquetVectorizedSuite extends ParquetTest with SharedSparkSession {
551551
}
552552
}
553553

554+
test("SPARK-60262: a column requested twice fails instead of leaving a vector unwritten") {
555+
// The requested schema lists column `a` twice, as it would if two requested fields selected
556+
// it, one by field id and one by name. Parquet's own page stores hand out a single page reader
557+
// per column, which every reader of that column shares, so the reader of the first copy drains
558+
// the pages and the reader of the second finds none. `MemPageStore` would give each its own.
559+
val fileSchema = MessageTypeParser.parseMessageType("message root { optional int32 a; }")
560+
val requestedSchema = MessageTypeParser.parseMessageType(
561+
"message root { optional int32 a; optional int32 a; }")
562+
val ty = fileSchema.asGroupType().getType("a").asPrimitiveType()
563+
val cd = new ColumnDescriptor(Array("a"), ty, 0, 1)
564+
val memPageStore = new MemPageStore(3)
565+
// Three non-null values: the definition levels (length 2, an RLE run of 3 x 1), then the
566+
// PLAIN int32 values 1, 2 and 3.
567+
memPageStore.getPageWriter(cd).writePage(
568+
BytesInput.from(Array[Byte](2, 0, 0, 0, 6, 1, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0)), 3, 3,
569+
Statistics.createStats(ty), Encoding.RLE, Encoding.RLE, Encoding.PLAIN)
570+
val sharedReader = memPageStore.getPageReader(cd)
571+
val sharedStore = new PageReadStore {
572+
override def getPageReader(descriptor: ColumnDescriptor): PageReader = sharedReader
573+
override def getRowCount: Long = 3
574+
}
575+
val recordReader = new VectorizedParquetRecordReader(
576+
DateTimeUtils.getZoneId("EST"), "CORRECTED", "UTC", "CORRECTED", "UTC", true, 3)
577+
try {
578+
recordReader.initialize(fileSchema, requestedSchema,
579+
TestParquetRowGroupReader(Seq(TestPageReadStore(sharedStore, Seq(0L)))), 3)
580+
val e = intercept[ParquetDecodingException](recordReader.nextKeyValue())
581+
assert(e.getMessage.contains("ran out of pages"), e.getMessage)
582+
assert(e.getMessage.contains("[a]"), e.getMessage)
583+
} finally {
584+
recordReader.close()
585+
}
586+
}
587+
554588
truncateTypeTest("primitive type", IntegerType, LongType, IntegerType)
555589

556590
truncateTypeTest("basic struct",

0 commit comments

Comments
 (0)