Error on invalid type/logicalType #368
Conversation
- Ran into a nasty issue where I had incorrectly used a `time.Duration` with a `long` Avro type and `timestamp-millis`. Hamba was encoding the entire int64 for this case instead of erroring on an invalid schema. When Spark/SQL tried to [convert](https://github.com/apache/spark/blob/v3.5.0/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala#L142C38-L142C52) this into a microseconds (it wants everything as micros), it eventually tries to do this: ``` def millisToMicros(millis: Long): Long = { Math.multiplyExact(millis, MICROS_PER_MILLIS) } ``` And [Math.multiplyExact](https://docs.oracle.com/en/java/javase/22/docs/api/java.base/java/lang/Math.html#multiplyExact(long,int)) detects an overflow because it is trying to multiply the long by 1000, which doesn't fit. This fix will give the user an error if they use the `logicalType` incorrectly. The error will look something like `avro: time.Duration is unsupported for Avro long and logicalType timestamp-micros`.
stampy88
force-pushed
the
invalid_logical_type_combination
branch
from
4c7f21e to
bdc185a
Compare
|
Hey, I know this has been merged already but I had a question. Sorry to hijack this PR conversation! I don't see why using a
At least that is what I would expect the encoder to do because it has everything necessary to do the conversion automatically. Is there any reason why this behaviour is not desired? I can contribute with it if you feel is a good idea. EDIT: Ah, sorry, I just realized that |
3 participants
time.Durationwith alongAvro type andtimestamp-millis. Hamba was encoding the entire int64 for this case instead of erroring on an invalid schema. When Spark/SQL tried to convert this into a microseconds (it wants everything as micros), it eventually tries to do this:And Math.multiplyExact detects an overflow because it is trying to multiply the long by 1000, which doesn't fit. This fix will give the user an error if they use the
logicalTypeincorrectly. The error will look something likeavro: time.Duration is unsupported for Avro long and logicalType timestamp-micros.