Skip to content

Commit 0549ece

Browse files
kz930claude
andauthored
feat(workflow-operator): compare Filter's exported predicates the way the engine does (apache#8980)
### What changes were proposed in this PR? Filter's exported Python compared the column to the value with pandas operators. The engine's `FilterPredicate.evaluate` compares by the column's type instead, so the two disagreed on numbers stored as text, booleans, NaN and -0.0. Each comparison in the export now goes through `_texera_filter_keep`, a transcription of `FilterPredicate.evaluate`: | Column type | Comparison | |---|---| | STRING | as Java doubles when both sides parse as one, otherwise by UTF-16 text | | BOOLEAN | the lower-case text "true"/"false" | | INTEGER, DOUBLE | by `Double.compareTo`: NaN equals NaN and sorts above every number, -0.0 sorts below 0.0; a hex value past the largest double is infinity, as in Java | | LONG | as integers, rejecting what `Long.valueOf` rejects | | TIMESTAMP | by whole milliseconds; a value with an offset is read as the wall clock it shows in the machine's zone, with the offset that zone had at that instant | The column type comes from the input schema. Without one, it is inferred from the pandas dtype. `is null` and `is not null` are unchanged. One approximation remains. The engine reads a TIMESTAMP value with DateParserUtils, which accepts more formats than pandas. A value pandas cannot read raises an error, so the export never gives a silently wrong answer. A value with an offset is placed in the machine's zone by `_texera_local_wall_clock` from apache#8597. It uses the zone's IANA history, as the JVM does, and not Windows' localtime. Before 1900 it takes the zone's standard offset, as the engine does. ### Any related issues, documentation, discussions? Closes apache#8977 Depends on apache#8597, which adds `_texera_local_wall_clock`, and on apache#9035, which reads a column named in a different case. ### How was this PR tested? A new test in `SpecializedFilterOpDescSpec` runs 33 predicates across the six column types through `SpecializedFilterOpExec` and through the export, and compares the kept rows by input position. A second test does the same for timestamp values with an offset, with both the JVM and Python set to Asia/Kolkata. Both fail on main and pass with this change. A third test sets both to America/Mexico_City and compares `2021-07-01T00:00:00Z` (summer time) and `1931-01-01T00:00:00Z` (an older offset). With apache#8597 merged in, the Filter specs pass (42 tests). A fourth test compares a LONG column `Key` through a predicate on `KEY`, past 2^53, and passes with apache#8597 and apache#9035 merged in. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 241b504 commit 0549ece

2 files changed

Lines changed: 461 additions & 29 deletions

File tree

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/filter/SpecializedFilterOpDesc.scala‎

Lines changed: 142 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -21,14 +21,16 @@ package org.apache.texera.amber.operator.filter
2121

2222
import com.fasterxml.jackson.annotation.{JsonProperty, JsonPropertyDescription}
2323
import org.apache.texera.amber.core.executor.OpExecWithClassName
24-
import org.apache.texera.amber.core.tuple.Schema
24+
import org.apache.texera.amber.core.tuple.{AttributeType, Schema}
2525
import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
2626
import org.apache.texera.amber.core.workflow.{InputPort, OutputPort, PhysicalOp, PortIdentity}
27-
import org.apache.texera.amber.operator.StandaloneCodeGenerator
27+
import org.apache.texera.amber.operator.{StandaloneCodeGenerator, StandaloneHelpers}
2828
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
2929
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.pyStringLiteral
3030
import org.apache.texera.amber.util.JSONUtils.objectMapper
3131

32+
import scala.util.Try
33+
3234
class SpecializedFilterOpDesc extends FilterOpDesc with StandaloneCodeGenerator {
3335

3436
@JsonProperty(value = "predicates", required = true)
@@ -71,39 +73,154 @@ class SpecializedFilterOpDesc extends FilterOpDesc with StandaloneCodeGenerator
7173
// which answers false on an empty list. Passing the frame through would be
7274
// the opposite answer.
7375
if (predicates.isEmpty) return "out1df = in1df.iloc[0:0].copy()"
76+
val schema = inputSchemas.values.headOption
7477
val conditions = predicates.map { p =>
7578
// FilterPredicate reads the field through Schema, which ignores case. See
7679
// [[columnAsStored]].
77-
val colLit = pyStringLiteral(columnAsStored(p.attribute, inputSchemas.values.headOption))
80+
val colLit = pyStringLiteral(columnAsStored(p.attribute, schema))
7881
p.condition match {
7982
case ComparisonType.IS_NULL => s"""(in1df[$colLit].isna())"""
8083
case ComparisonType.IS_NOT_NULL => s"""(in1df[$colLit].notna())"""
81-
case other =>
82-
val op = other.getName // returns "=", ">=", "<", etc. (see ComparisonType.java)
83-
val pyOp = if (op == "=") "==" else op
84-
// notna mirrors FilterPredicate, which answers false for every condition
85-
// but IS_NULL / IS_NOT_NULL once the field is null. Only `!=` needs it —
86-
// pandas answers True there, where every other operator answers False —
87-
// but guarding all of them keeps the one rule visible in one place.
88-
s"""(in1df[$colLit].notna() & (in1df[$colLit] $pyOp ${coerceValue(p.value)}))"""
84+
case other =>
85+
// FilterPredicate finds the field through Schema, which ignores case,
86+
// and compares by the type declared there. Without a schema the
87+
// column's dtype stands in for its type.
88+
val keep = schema
89+
.flatMap(s => Try(s.getAttribute(p.attribute)).toOption)
90+
.map(a => SpecializedFilterOpDesc.keepFor(a.getType))
91+
.getOrElse("_texera_filter_keep")
92+
val opLit = pyStringLiteral(other.getName)
93+
val valueLit = pyStringLiteral(Option(p.value).getOrElse(""))
94+
s"$keep(in1df[$colLit], $opLit, $valueLit)"
8995
}
9096
}
9197
s"out1df = in1df[${conditions.mkString(" | ")}].reset_index(drop=True)"
9298
}
9399

94-
// Try numeric coercion so generated code compares column values against the right type.
95-
// Strings that don't parse fall through to a quoted string literal.
96-
private def coerceValue(raw: String): String = {
97-
try {
98-
raw.toInt.toString
99-
} catch {
100-
case _: NumberFormatException =>
101-
try {
102-
raw.toDouble.toString
103-
} catch {
104-
case _: NumberFormatException =>
105-
pyStringLiteral(raw)
106-
}
100+
override def standaloneHelpers(): Seq[String] =
101+
Seq(StandaloneHelpers.AttributeCasts, SpecializedFilterOpDesc.FilterPredicates)
102+
}
103+
104+
object SpecializedFilterOpDesc {
105+
106+
/** The function that compares a column of `attributeType`, so a script calls only the one
107+
* its column needs.
108+
*/
109+
def keepFor(attributeType: AttributeType): String =
110+
attributeType match {
111+
case AttributeType.INTEGER | AttributeType.DOUBLE => "_texera_filter_keep_double"
112+
case AttributeType.LONG => "_texera_filter_keep_long"
113+
case AttributeType.BOOLEAN => "_texera_filter_keep_boolean"
114+
case AttributeType.TIMESTAMP => "_texera_filter_keep_timestamp"
115+
case _ => "_texera_filter_keep_string"
107116
}
108-
}
117+
118+
/** `FilterPredicate.evaluate`, one function per type the engine compares apart. A null passes
119+
* nothing.
120+
*
121+
* - `_texera_filter_keep_double`: INTEGER and DOUBLE by `Double.compareTo`, NaN above all.
122+
* - `_texera_filter_keep_long`: LONG, the value read by `Long.valueOf`.
123+
* - `_texera_filter_keep_boolean`: BOOLEAN as lower-case text.
124+
* - `_texera_filter_keep_timestamp`: TIMESTAMP by whole milliseconds in the machine's zone.
125+
* - `_texera_filter_keep_string`: STRING as numbers when both parse, else by UTF-16 units.
126+
* - `_texera_filter_keep`: the type read off the column's dtype, when no schema says it.
127+
*
128+
* The value is read with `_texera_java_trim`, `_texera_java_double` and `_texera_parse_int`,
129+
* and a double is ordered by `_texera_double_key`, all from
130+
* [[StandaloneHelpers.AttributeCasts]].
131+
*/
132+
val FilterPredicates: String =
133+
"""def _texera_filter_text_key(s):
134+
| return s.encode("utf-16-be", "surrogatepass")
135+
|
136+
|def _texera_filter_compared(left, right):
137+
| return (left > right) - (left < right)
138+
|
139+
|def _texera_filter_rows(column, condition, order):
140+
| passes = {
141+
| "=": lambda c: c == 0,
142+
| ">": lambda c: c > 0,
143+
| ">=": lambda c: c >= 0,
144+
| "<": lambda c: c < 0,
145+
| "<=": lambda c: c <= 0,
146+
| "!=": lambda c: c != 0,
147+
| }[condition]
148+
| return pd.Series(
149+
| [
150+
| False if missing else passes(order(v))
151+
| for v, missing in zip(column, column.isna())
152+
| ],
153+
| index=column.index,
154+
| dtype=bool,
155+
| )
156+
|
157+
|def _texera_filter_keep_double(column, condition, value):
158+
| target = _texera_double_key(_texera_java_double(value))
159+
|
160+
| def order(v):
161+
| return _texera_filter_compared(_texera_double_key(float(v)), target)
162+
|
163+
| return _texera_filter_rows(column, condition, order)
164+
|
165+
|def _texera_filter_keep_long(column, condition, value):
166+
| target = _texera_parse_int(value, 64)
167+
|
168+
| def order(v):
169+
| return _texera_filter_compared(int(v), target)
170+
|
171+
| return _texera_filter_rows(column, condition, order)
172+
|
173+
|def _texera_filter_keep_boolean(column, condition, value):
174+
| target = _texera_filter_text_key(_texera_java_trim(value).lower())
175+
|
176+
| def order(v):
177+
| return _texera_filter_compared(_texera_filter_text_key("true" if v else "false"), target)
178+
|
179+
| return _texera_filter_rows(column, condition, order)
180+
|
181+
|def _texera_filter_keep_timestamp(column, condition, value):
182+
| zone = _texera_local_zone()
183+
|
184+
| def wall_clock(ts):
185+
| ts = pd.Timestamp(ts).floor("ms")
186+
| if ts.tzinfo is None:
187+
| return ts
188+
| return pd.Timestamp(_texera_local_wall_clock(ts.to_pydatetime(), zone))
189+
|
190+
| target = wall_clock(_texera_java_trim(value))
191+
|
192+
| def order(v):
193+
| return _texera_filter_compared(wall_clock(v), target)
194+
|
195+
| return _texera_filter_rows(column, condition, order)
196+
|
197+
|def _texera_filter_keep_string(column, condition, value):
198+
| try:
199+
| value_number = _texera_double_key(_texera_java_double(value))
200+
| except ValueError:
201+
| value_number = None
202+
|
203+
| def order(v):
204+
| text = str(v)
205+
| if value_number is not None:
206+
| try:
207+
| number = _texera_double_key(_texera_java_double(text))
208+
| return _texera_filter_compared(number, value_number)
209+
| except ValueError:
210+
| pass
211+
| return _texera_filter_compared(
212+
| _texera_filter_text_key(text), _texera_filter_text_key(value)
213+
| )
214+
|
215+
| return _texera_filter_rows(column, condition, order)
216+
|
217+
|def _texera_filter_keep(column, condition, value):
218+
| dtype = column.dtype
219+
| if pd.api.types.is_bool_dtype(dtype):
220+
| return _texera_filter_keep_boolean(column, condition, value)
221+
| if pd.api.types.is_integer_dtype(dtype) or pd.api.types.is_float_dtype(dtype):
222+
| return _texera_filter_keep_double(column, condition, value)
223+
| if pd.api.types.is_datetime64_any_dtype(dtype):
224+
| return _texera_filter_keep_timestamp(column, condition, value)
225+
| return _texera_filter_keep_string(column, condition, value)""".stripMargin
109226
}

0 commit comments

Comments
 (0)