Repository navigation
Commit 0549ece
feat(workflow-operator): compare Filter's exported predicates the way the engine does (apache#8980)
### What changes were proposed in this PR?
Filter's exported Python compared the column to the value with pandas
operators. The engine's `FilterPredicate.evaluate` compares by the
column's type instead, so the two disagreed on numbers stored as text,
booleans, NaN and -0.0.
Each comparison in the export now goes through `_texera_filter_keep`, a
transcription of `FilterPredicate.evaluate`:
| Column type | Comparison |
|---|---|
| STRING | as Java doubles when both sides parse as one, otherwise by
UTF-16 text |
| BOOLEAN | the lower-case text "true"/"false" |
| INTEGER, DOUBLE | by `Double.compareTo`: NaN equals NaN and sorts
above every number, -0.0 sorts below 0.0; a hex value past the largest
double is infinity, as in Java |
| LONG | as integers, rejecting what `Long.valueOf` rejects |
| TIMESTAMP | by whole milliseconds; a value with an offset is read as
the wall clock it shows in the machine's zone, with the offset that zone
had at that instant |
The column type comes from the input schema. Without one, it is inferred
from the pandas dtype. `is null` and `is not null` are unchanged.
One approximation remains. The engine reads a TIMESTAMP value with
DateParserUtils, which accepts more formats than pandas. A value pandas
cannot read raises an error, so the export never gives a silently wrong
answer.
A value with an offset is placed in the machine's zone by
`_texera_local_wall_clock` from apache#8597. It uses the zone's IANA history,
as the JVM does, and not Windows' localtime. Before 1900 it takes the
zone's standard offset, as the engine does.
### Any related issues, documentation, discussions?
Closes apache#8977
Depends on apache#8597, which adds `_texera_local_wall_clock`, and on apache#9035,
which reads a column named in a different case.
### How was this PR tested?
A new test in `SpecializedFilterOpDescSpec` runs 33 predicates across
the six column types through `SpecializedFilterOpExec` and through the
export, and compares the kept rows by input position. A second test does
the same for timestamp values with an offset, with both the JVM and
Python set to Asia/Kolkata. Both fail on main and pass with this change.
A third test sets both to America/Mexico_City and compares
`2021-07-01T00:00:00Z` (summer time) and `1931-01-01T00:00:00Z` (an
older offset). With apache#8597 merged in, the Filter specs pass (42 tests). A
fourth test compares a LONG column `Key` through a predicate on `KEY`,
past 2^53, and passes with apache#8597 and apache#9035 merged in.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>1 parent 241b504 commit 0549ece
2 files changed
Lines changed: 461 additions & 29 deletions
File tree
- common/workflow-operator/src
- main/scala/org/apache/texera/amber/operator/filter
- test/scala/org/apache/texera/amber/operator/filter
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
24 | | - | |
| 24 | + | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
| 27 | + | |
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| 32 | + | |
| 33 | + | |
32 | 34 | | |
33 | 35 | | |
34 | 36 | | |
| |||
71 | 73 | | |
72 | 74 | | |
73 | 75 | | |
| 76 | + | |
74 | 77 | | |
75 | 78 | | |
76 | 79 | | |
77 | | - | |
| 80 | + | |
78 | 81 | | |
79 | 82 | | |
80 | 83 | | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
89 | 95 | | |
90 | 96 | | |
91 | 97 | | |
92 | 98 | | |
93 | 99 | | |
94 | | - | |
95 | | - | |
96 | | - | |
97 | | - | |
98 | | - | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | | - | |
103 | | - | |
104 | | - | |
105 | | - | |
106 | | - | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
107 | 116 | | |
108 | | - | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
109 | 226 | | |
0 commit comments