Repository navigation
Commit 0567211
committed
feat(amber): bind $K loop-variable references in operators
A property of an operator inside a control block may name a loop variable
as `$K`. The frontend keeps sending the literal string, whatever the
property's type; the backend now handles it in three steps.
At plan-parse time a Jackson module (StateReferenceModule, registered on
JSONUtils.objectMapper and on the three Dropwizard mappers) wraps the
deserializer of every StateReferencing bean -- LogicalOp now is one. An
object that holds a `$K` (or a client-sent sidecar) is rewritten: each
`$K` leaf becomes a placeholder of the property's JSON type (0 / 0.0 /
false; a String keeps the literal) and the property's JSON pointer ->
variable name is recorded in the `stateReferences` sidecar, which the
property panel never shows. An object with no typed placeholder to put
in, which is every object that parsed before, is still read from its own
tokens.
At compile time the compiler works out which operators lie inside a loop
block (LoopBlockMembership, a port of the frontend's block walk) and
makes the sidecar authoritative before the descriptor serializes it into
its executor's descString. Inside a block the sidecar holds every
reference, including the whole-string `$K` literals of a descriptor built
in Scala. Outside every block a string reference is the literal it looks
like (a cashtag such as "$AAPL" filters as on main) and is dropped, and a
typed placeholder is a compile error naming the property and the
variable. A reference to a property the descriptor ignores is dropped
either way. Inside a block, a referencing operator with an input fed
from outside the block is a compile error too, since that input's tuples
could arrive before the loop state; an input that waits for one carrying
the state is fine. A runtime reconfiguration, which builds its executor
without the compiler, applies the same rules against the running plan,
and the Python export refuses a reference its script could not bind.
At the worker, ExecFactory hands out a LateBoundExecutor exactly when the
descString's sidecar names a loop variable. It stays unbound until the
first state message, then patches the placeholders with the loop
variables (coerced to the placeholder's JSON type), instantiates and opens
the real executor once, and delegates every later call to it. A source is
never wrapped, since no state message reaches it. The worker registers
each state message on the executor (`registerState` / `loopState`) before
`processState` runs, and a reconfiguration or a checkpoint restore hands
the replacement executor the state its predecessor consumed.
An operator inside a block whose executor runs generated code (the UDFs
and the Python-generating descriptors) cannot bind a reference yet: the
compiler reports it instead of running with the placeholders.
Closes apache#86351 parent 454089e commit 0567211
43 files changed
Lines changed: 3626 additions & 39 deletions
File tree
- amber/src
- main/scala/org/apache/texera
- amber/engine
- architecture/worker
- managers
- promisehandlers
- common
- web
- service
- test
- integration/org/apache/texera/amber/engine/e2e
- scala/org/apache/texera
- amber/engine/architecture/worker
- managers
- promisehandlers
- web
- service
- common
- workflow-compiler/src
- main/scala/org/apache/texera/common/compiler
- test/scala/org/apache/texera/common/compiler
- workflow-core/src
- main/scala/org/apache/texera/amber
- core
- executor
- state
- util
- test/scala/org/apache/texera/amber/core
- executor
- state
- workflow-operator/src
- main/scala/org/apache/texera/amber/operator
- metadata
- test/scala/org/apache/texera/amber/operator
- filter
- limit
- metadata
- workflow-compiling-service/src
- main/scala/org/apache/texera/service
- resource
- test/scala/org/apache/texera/service
- resource
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
128 | 128 | | |
129 | 129 | | |
130 | 130 | | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
131 | 136 | | |
132 | 137 | | |
133 | 138 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
27 | | - | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
28 | 34 | | |
29 | 35 | | |
30 | 36 | | |
| |||
53 | 59 | | |
54 | 60 | | |
55 | 61 | | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
56 | 67 | | |
57 | 68 | | |
58 | 69 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
62 | 67 | | |
63 | 68 | | |
64 | 69 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| |||
44 | 45 | | |
45 | 46 | | |
46 | 47 | | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
47 | 51 | | |
48 | 52 | | |
49 | 53 | | |
50 | 54 | | |
51 | 55 | | |
| 56 | + | |
52 | 57 | | |
53 | 58 | | |
54 | 59 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
31 | 34 | | |
32 | 35 | | |
33 | 36 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
| |||
136 | 137 | | |
137 | 138 | | |
138 | 139 | | |
| 140 | + | |
| 141 | + | |
139 | 142 | | |
140 | 143 | | |
141 | 144 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| 31 | + | |
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
| |||
98 | 99 | | |
99 | 100 | | |
100 | 101 | | |
| 102 | + | |
| 103 | + | |
101 | 104 | | |
102 | 105 | | |
103 | 106 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| 31 | + | |
30 | 32 | | |
31 | 33 | | |
32 | 34 | | |
| |||
60 | 62 | | |
61 | 63 | | |
62 | 64 | | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
63 | 71 | | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
69 | | - | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
70 | 94 | | |
71 | 95 | | |
72 | 96 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
44 | 49 | | |
45 | 50 | | |
46 | 51 | | |
| |||
188 | 193 | | |
189 | 194 | | |
190 | 195 | | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
191 | 203 | | |
192 | 204 | | |
193 | 205 | | |
| |||
317 | 329 | | |
318 | 330 | | |
319 | 331 | | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
320 | 360 | | |
321 | 361 | | |
322 | 362 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
27 | 28 | | |
| 29 | + | |
28 | 30 | | |
29 | | - | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
30 | 41 | | |
31 | 42 | | |
32 | 43 | | |
| |||
131 | 142 | | |
132 | 143 | | |
133 | 144 | | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
134 | 189 | | |
135 | 190 | | |
136 | 191 | | |
| |||
0 commit comments