You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit ca8c6ee
Browse filesBrowse the repository at this point in the historyBrowse files
fix(operator): read Parquet INT96 timestamps in microseconds
pyarrow reads an INT96 in nanoseconds by default, which only reach 1677
to 2262, so a valid date outside that range wrapped silently: 2500 read
back as 1915 and 1500 as 2084. The footer and the rows are both read in
microseconds now. The digits under the microsecond are dropped, where the
executor keeps them.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Copy file name to clipboardExpand all lines: common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/parquet/ParquetScanSourceOpDesc.scala
+9-1Lines changed: 9 additions & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -73,11 +73,19 @@ class ParquetScanSourceOpDesc extends ScanSourceOpDesc with StandaloneCodeGenera
73
73
// the footer states and knows nothing of an index, so it hands `customer_id`
74
74
// on like any other. Stripped, pandas reads every column the file holds,
75
75
// under the name the footer gives it and in the order it states them.
76
+
//
77
+
// An INT96 timestamp is read in microseconds, in the footer and in the rows
78
+
// alike. pyarrow's default is nanoseconds, which only reach 1677 to 2262, and
79
+
// a date outside them wraps silently: 2500 read back as 1915. The executor
80
+
// keeps the digits under the microsecond, which this drops.
Copy file name to clipboardExpand all lines: common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/parquet/ParquetScanSourceOpDescSpec.scala
0 commit comments