You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit ce118d3
Browse filesBrowse the repository at this point in the historyBrowse files
feat(amber): bind $K loop-variable references in operators
A property of any operator inside a control block may name a loop variable
as `$K`. The frontend keeps sending the literal string, whatever the
property's type; the backend now handles it in three steps.
At plan-parse time a Jackson module (StateReferenceModule, registered on
JSONUtils.objectMapper and on the three Dropwizard mappers) wraps the
deserializer of every StateReferencing bean -- LogicalOp now is one. An
object that holds a `$K` (or a client-sent sidecar) is rewritten: each
`$K` leaf becomes a placeholder of the property's JSON type (0 / 0.0 /
false; a String keeps the literal) and the property's JSON pointer ->
variable name is recorded in the `stateReferences` sidecar, which the
property panel never shows. Every other object is re-fed untouched from a
token buffer.
At compile time the compiler works out which operators lie inside a loop
block (LoopBlockMembership, a port of the frontend's block walk) and
makes the sidecar authoritative before the descriptor serializes it into
its executor's descString. Inside a block the sidecar holds every
reference, including the whole-string `$K` literals of a descriptor built
in Scala. Outside every block a string reference is the literal it looks
like (a cashtag such as "$AAPL" filters as on main) and is dropped, and a
typed placeholder is a compile error naming the property and the
variable. A runtime reconfiguration, which builds its executor without
the compiler, applies the same rules against the running plan.
At the worker, ExecFactory hands out a LateBoundExecutor exactly when the
descString's sidecar names a loop variable. It stays unbound until the
first state message, then patches the placeholders with the loop
variables (coerced to the placeholder's JSON type), instantiates and opens
the real executor once, and delegates every later call to it. A source is
never wrapped, since no state message reaches it. The worker registers
each state message on the executor (`registerState` / `loopState`) before
`processState` runs, and a reconfiguration or a checkpoint restore hands
the replacement executor the state its predecessor consumed.
An operator inside a block whose executor runs generated code (the UDFs
and the Python-generating descriptors) cannot bind a reference yet: the
compiler reports it instead of running with the placeholders.
Closesapache#8635
Copy file name to clipboardExpand all lines: amber/src/main/scala/org/apache/texera/amber/engine/architecture/worker/managers/SerializationManager.scala
Copy file name to clipboardExpand all lines: amber/src/main/scala/org/apache/texera/amber/engine/architecture/worker/promisehandlers/PrepareCheckpointHandler.scala
Copy file name to clipboardExpand all lines: amber/src/main/scala/org/apache/texera/amber/engine/architecture/worker/promisehandlers/UpdateExecutorHandler.scala
Copy file name to clipboardExpand all lines: amber/src/test/scala/org/apache/texera/amber/engine/architecture/worker/DataProcessorRPCHandlerInitializerSpec.scala
0 commit comments