Skip to content

Commit 9e841f2

Browse files
committed
feat: bind number and true/false loop variables in generated Python operators
A number or true/false property of an operator whose Scala descriptor generates Python can now refer to a loop variable inside a control block. The code is generated before the loop runs, so PythonOperatorDescriptor.loopVariableBinding generates it from copies of the descriptor with probe values in the reference's place: 3 and 5, then 70 and 90 for an integer; 0.25 and 0.75, then 2.5 and 7.5 for a number; True and False. Where the two codes differ by exactly the two values, and nowhere else, the code calls self.loop_variable_value(name, kind) there instead: 'integer' for an int property, 'long' for one that takes a whole number past 32 bits. The value must also land as it is at 0, 1, -1, 1000000 and the other probes. When no probe pair generates the code, the properties' defaults and the values next to them are tried, so a descriptor's own check never reports a probe value. A value the descriptor checks or adjusts, one that lands in a text, a comment or a name, one the code computes with, never holds or reads when the operator starts (in open or __init__, before any state message), and one that decides the output schema leave the reference a compile error that names the property and why. WorkflowCompiler probes the code once per compile: the physical operator runs that binding's code, and unbindableStateReferences decides every reason from the same binding, the code's before the schema's, and the schema one only once every input has a schema, as the schema propagation does. Operator.loop_variable_value reads the variable's text as loop_variable_text spells it, from the values bound at the operator's first tuple, and converts it as a JVM operator's property binds it (StateReferenceBinding.coerce): trimmed as String.trim, an integer as new BigDecimal(text).longValueExact() reads it, and within 32 bits for an int property, as Jackson writes it there; a number as Double.parseDouble does (2.5d and 0x1p3, but not inf, nan, 1_000 or fullwidth digits), and a boolean as true or false in any case. A whole double past 10^16 reads as the whole number its fewest digits spell, as JDK 19 and later bind it; JDK 17 spells it with other digits (JDK-4511638). An integer past 64 bits is refused, as loop_variable_text refuses it. The executor manager also hands each Python operator the values its code reads through loop_variable_value, so binding fails, naming each, for a required value no state message carried or one its property cannot take, as the JVM's binding does, whether or not the code reads the value then.
1 parent 959939d commit 9e841f2

4 files changed

Lines changed: 785 additions & 61 deletions

File tree

‎amber/src/main/python/core/architecture/managers/executor_manager.py‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,10 @@
2727
from typing import Tuple, Optional
2828

2929
from core.models import Operator, SourceOperator
30-
from core.models.operator import referenced_loop_variables
30+
from core.models.operator import (
31+
referenced_loop_variable_values,
32+
referenced_loop_variables,
33+
)
3134

3235

3336
def pytexera_names() -> set:
@@ -435,6 +438,9 @@ class declaration.
435438
# Binding requires the loop variables the operator's properties
436439
# refer to: the ones its generated code reads.
437440
self.executor.require_loop_variables(referenced_loop_variables(code))
441+
self.executor.require_loop_variable_values(
442+
referenced_loop_variable_values(code)
443+
)
438444
assert isinstance(self.executor, SourceOperator) == self.executor.is_source, (
439445
self._source_mismatch_message(is_source)
440446
)

‎amber/src/main/python/core/models/operator.py‎

Lines changed: 212 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919
import overrides
2020
import pandas
2121
import re
22+
import unicodedata
2223
from functools import lru_cache
2324
from abc import ABC, abstractmethod
2425
from collections import defaultdict
@@ -132,6 +133,13 @@ def _loop_variable_text(name: str, variables: Mapping[str, Any], bound: bool) ->
132133
)
133134

134135

136+
# The same for a numeric or boolean property's reference, its kind beside it
137+
# (PythonOperatorDescriptor.loopVariableValueLookup).
138+
_LOOP_VARIABLE_VALUE_LOOKUP = re.compile(
139+
r"self\.loop_variable_value\('([A-Za-z_][A-Za-z0-9_]*)', '([a-z]+)'\)"
140+
)
141+
142+
135143
def referenced_loop_variables(code: str) -> List[str]:
136144
"""
137145
The loop variables ``code`` reads through ``Operator.loop_variable_text``,
@@ -141,6 +149,147 @@ def referenced_loop_variables(code: str) -> List[str]:
141149
return sorted(set(_LOOP_VARIABLE_TEXT_LOOKUP.findall(code)))
142150

143151

152+
def referenced_loop_variable_values(code: str) -> List[tuple]:
153+
"""
154+
The loop variables ``code`` reads through ``Operator.loop_variable_value``,
155+
each as its name and kind, once, in that order: in the code generated for
156+
an operator inside a control block, the ones its numeric and boolean
157+
properties refer to.
158+
"""
159+
return sorted(set(_LOOP_VARIABLE_VALUE_LOOKUP.findall(code)))
160+
161+
162+
# Each kind loop_variable_value reads, and what a value of it is called.
163+
_VALUE_KINDS = {
164+
"integer": "an integer",
165+
"long": "an integer",
166+
"number": "a number",
167+
"boolean": "a boolean",
168+
}
169+
170+
171+
def _check_value_kind(name: str, kind: str) -> None:
172+
"""Refuse a kind the generated code never asks ``loop_variable_value`` for."""
173+
if kind not in _VALUE_KINDS:
174+
raise ValueError(
175+
f"loop variable ${name} cannot be read as {kind!r}: "
176+
"the kind is one of 'integer', 'long', 'number' and 'boolean'"
177+
)
178+
179+
180+
def _loop_variable_value(
181+
name: str, kind: str, variables: Mapping[str, Any], bound: bool
182+
) -> Union[int, float, bool]:
183+
"""
184+
Loop variable ``name`` of ``variables`` as ``Operator.loop_variable_value``
185+
reads it as a ``kind``; ``bound`` as for ``_loop_variable_text``.
186+
"""
187+
_check_value_kind(name, kind)
188+
text = _loop_variable_text(name, variables, bound).strip(_JAVA_TRIMMED)
189+
if kind in ("integer", "long"):
190+
value = _parse_java_long(text)
191+
elif kind == "number":
192+
value = _parse_java_double(text)
193+
else:
194+
# Scala's toBooleanOption: equalsIgnoreCase, which also takes the
195+
# long s (ſ) for an s, as its upper case is S.
196+
value = {"TRUE": True, "FALSE": False}.get(text.upper())
197+
if value is None:
198+
raise RuntimeError(
199+
f"the operator refers to loop variable ${name}, "
200+
f"but its value {variables[name]!r} is not {_VALUE_KINDS[kind]}"
201+
)
202+
if kind == "integer" and not -(2**31) <= value < 2**31:
203+
raise RuntimeError(
204+
f"the operator refers to loop variable ${name}, "
205+
f"but its value {variables[name]!r} does not fit its property, "
206+
"a 32-bit integer"
207+
)
208+
return value
209+
210+
211+
# What Java's String.trim drops from either end of a text: every character from
212+
# U+0000 to U+0020, the space and the control characters below it, and nothing
213+
# above it.
214+
_JAVA_TRIMMED = "".join(map(chr, range(0x21)))
215+
216+
# The numerals Java's Double.parseDouble reads, besides NaN and Infinity: a
217+
# decimal one, or a hexadecimal one with a binary exponent, each with an
218+
# optional sign and an f, F, d or D suffix that changes nothing. Their digits
219+
# are ASCII ones.
220+
_JAVA_DECIMAL_DOUBLE = re.compile(
221+
r"[+-]?(?:[0-9]+\.?[0-9]*|\.[0-9]+)(?:[eE][+-]?[0-9]+)?[fFdD]?"
222+
)
223+
_JAVA_HEX_DOUBLE = re.compile(
224+
r"[+-]?0[xX](?:[0-9a-fA-F]+\.?|[0-9a-fA-F]*\.[0-9a-fA-F]+)[pP][+-]?[0-9]+[fFdD]?"
225+
)
226+
227+
# The numerals Java's new BigDecimal(String) reads, their digits as ASCII ones:
228+
# a sign, digits with at most one point, and an exponent; groups: the sign, the
229+
# digits before the point, the digits after it, the exponent.
230+
_JAVA_BIG_DECIMAL = re.compile(r"([+-]?)([0-9]*)(?:\.([0-9]*))?(?:[eE]([+-]?[0-9]+))?")
231+
232+
233+
def _parse_java_double(text: str) -> Optional[float]:
234+
"""
235+
``text`` read as Java's ``Double.parseDouble`` reads a trimmed text, or
236+
``None`` where that throws. Python's ``float`` reads more (``inf``,
237+
``nan``, ``1_000``, other scripts' digits) and less (``2.5d``,
238+
``0x1p3``).
239+
"""
240+
unsigned = text[1:] if text[:1] in ("+", "-") else text
241+
if unsigned == "NaN":
242+
return math.nan
243+
if unsigned == "Infinity":
244+
return -math.inf if text[0] == "-" else math.inf
245+
if _JAVA_DECIMAL_DOUBLE.fullmatch(text):
246+
return float(text.rstrip("fFdD"))
247+
if _JAVA_HEX_DOUBLE.fullmatch(text):
248+
try:
249+
return float.fromhex(text.rstrip("fFdD"))
250+
except OverflowError: # Java rounds it to an infinity
251+
return -math.inf if text[0] == "-" else math.inf
252+
return None
253+
254+
255+
def _parse_java_long(text: str) -> Optional[int]:
256+
"""
257+
``text`` read as Java's ``new BigDecimal(text).longValueExact()`` reads a
258+
trimmed text, or ``None`` where that throws: a numeral that is a whole
259+
number within a signed 64-bit long, such as ``42``, ``4.2e1`` or ``3.0``.
260+
Its digits may be any script's, but only of the Basic Multilingual Plane,
261+
as Java reads a text one UTF-16 unit at a time, and its exponent and
262+
scale must each fit a 32-bit int. Python's ``Decimal`` reads more
263+
(``inf``, ``nan``, ``1_000``, an exponent of any size).
264+
"""
265+
if any(ord(char) > 0xFFFF for char in text):
266+
return None
267+
ascii_text = "".join(
268+
str(unicodedata.decimal(char)) if char.isdecimal() else char for char in text
269+
)
270+
match = _JAVA_BIG_DECIMAL.fullmatch(ascii_text)
271+
if match is None or not (match[2] or match[3]):
272+
return None
273+
sign, whole, fraction, exponent = match.groups(default="")
274+
exponent_digits = exponent.lstrip("+-").lstrip("0")
275+
if len(exponent_digits) > 10:
276+
return None
277+
power = int(exponent_digits or "0") * (-1 if exponent[:1] == "-" else 1)
278+
scale = len(fraction) - power
279+
if not -(2**31) <= power < 2**31 or not -(2**31) <= scale < 2**31:
280+
return None
281+
digits = (whole + fraction).lstrip("0")
282+
if not digits:
283+
return 0
284+
# The value is significant * 10**shift, a whole number when shift >= 0.
285+
significant = digits.rstrip("0")
286+
shift = len(digits) - len(significant) - scale
287+
if shift < 0 or len(significant) + shift > 19:
288+
return None
289+
value = int(sign + significant) * 10**shift
290+
return value if -(2**63) <= value < 2**63 else None
291+
292+
144293
class Operator(ABC):
145294
"""
146295
Abstract base class for all operators.
@@ -211,6 +360,38 @@ def loop_variable_text(self, name: str) -> str:
211360
return _loop_variable_text(name, self.loop_variables, False)
212361
return _loop_variable_text(name, bound, True)
213362

363+
def loop_variable_value(self, name: str, kind: str) -> Union[int, float, bool]:
364+
"""
365+
The value of loop variable ``name`` as a ``kind``. Inside a control
366+
block, the code generated for an operator calls this where a numeric or
367+
boolean property holds ``$name``, in place of the value written into
368+
the code, so the value is read when the code runs. It is read from the
369+
variable's text (``loop_variable_text``, so the values bound at the
370+
operator's first tuple), as a JVM operator's numeric or boolean
371+
property binds it (``StateReferenceBinding``): the text trimmed as
372+
Java's ``String.trim`` trims it, then an integer as
373+
``new BigDecimal(text).longValueExact()`` reads it (a whole number
374+
within a signed 64-bit long, and within 32 bits for an int property,
375+
as Jackson refuses a larger one there), a number as
376+
``Double.parseDouble`` does, and a boolean as ``true`` or ``false`` in
377+
any case, never ``1`` or ``0``.
378+
379+
:param name: str, the loop variable's name, without the leading ``$``.
380+
:param kind: str, the property that holds ``$name``: ``integer`` (a
381+
Java ``int`` or ``Integer``), ``long`` (a ``long`` or ``Long``),
382+
``number`` or ``boolean``.
383+
:return: int, float or bool, the variable's value as ``kind``.
384+
"""
385+
_check_value_kind(name, kind)
386+
bound = self.__internal_bound_loop_variables
387+
if bound is None and self.__internal_loop_variables is None:
388+
raise RuntimeError(
389+
f"loop variable ${name} is read before the iteration's state arrived"
390+
)
391+
if bound is None:
392+
return _loop_variable_value(name, kind, self.loop_variables, False)
393+
return _loop_variable_value(name, kind, bound, True)
394+
214395
__internal_is_source: bool = False
215396

216397
# The loop variables received so far, by name, and the nesting depth of
@@ -226,6 +407,9 @@ def loop_variable_text(self, name: str) -> str:
226407
# The loop variables the operator's properties refer to, which binding
227408
# requires; see ``require_loop_variables``.
228409
__internal_required_loop_variables: Sequence[str] = ()
410+
# The same for its numeric and boolean properties, each with its kind; see
411+
# ``require_loop_variable_values``.
412+
__internal_required_loop_variable_values: Sequence[tuple] = ()
229413

230414
@property
231415
@overrides.final
@@ -280,6 +464,22 @@ def require_loop_variables(self, names: Iterable[str]) -> None:
280464
"""
281465
self.__internal_required_loop_variables = tuple(names)
282466

467+
@overrides.final
468+
def require_loop_variable_values(self, references: Iterable[tuple]) -> None:
469+
"""
470+
As ``require_loop_variables``, for the operator's numeric and boolean
471+
properties, which its generated code reads through
472+
``loop_variable_value``: each reference is the loop variable's name and
473+
the kind it is read as (``referenced_loop_variable_values``). Binding
474+
then also fails while one of them has no value its property can take.
475+
476+
:param references: Iterable[tuple], each loop variable's name, without
477+
``$``, and kind.
478+
"""
479+
self.__internal_required_loop_variable_values = tuple(
480+
(name, kind) for name, kind in references
481+
)
482+
283483
@overrides.final
284484
def bind_loop_variables(self) -> None:
285485
"""
@@ -293,12 +493,13 @@ def bind_loop_variables(self) -> None:
293493
message for UDF code to read. The next iteration runs on a fresh
294494
worker, which binds again.
295495
296-
Binding fails, naming each one in one error, while a loop variable the
297-
operator requires (``require_loop_variables``) has no value its text
298-
property can take: no state message carried it, or the value is not a
299-
scalar or not within 64 bits. As a JVM operator's binding does, it then
300-
binds nothing, whether or not the code would read the value, and a
301-
later binding tries again.
496+
Binding fails, naming each one in one error, the texts first, while a
497+
loop variable the operator requires (``require_loop_variables``,
498+
``require_loop_variable_values``) has no value its property can take:
499+
no state message carried it, or the value is not a scalar, not within
500+
64 bits, or not of the property's kind. As a JVM operator's binding
501+
does, it then binds nothing, whether or not the code would read the
502+
value, and a later binding tries again.
302503
"""
303504
if self.__internal_bound_loop_variables is not None:
304505
return
@@ -309,6 +510,11 @@ def bind_loop_variables(self) -> None:
309510
_loop_variable_text(name, variables, True)
310511
except RuntimeError as error:
311512
problems.append(str(error))
513+
for name, kind in self.__internal_required_loop_variable_values:
514+
try:
515+
_loop_variable_value(name, kind, variables, True)
516+
except RuntimeError as error:
517+
problems.append(str(error))
312518
if problems:
313519
raise RuntimeError("; ".join(problems))
314520
self.__internal_bound_loop_variables = variables

0 commit comments

Comments
 (0)