Repository navigation
Commit 54ec7dc
committed
feat(amber): bind $K loop-variable references in operators
A property of an operator inside a loop block (LoopStart ... LoopEnd) may
name a loop variable as `$K`. The frontend keeps sending the literal
string whatever the property's type; the backend binds it in three steps.
Parse. LogicalOp is now a StateReferencing descriptor, with a
`stateReferences` sidecar (JSON pointer -> variable name) that the
property panel never shows. StateReferenceModule, registered on
JSONUtils.objectMapper and on the workflow-compiling-service mapper, adds
a Jackson problem handler: when a whole-string `$name` cannot be converted
to an integral, floating-point or boolean property, it returns the
placeholder 0 / 0.0 / false and records the value's pointer, relative to
the operator, in the operator's sidecar. A thin wrapper around each
concrete StateReferencing deserializer opens that recording frame. An
object that a polymorphic type deserializer hands over mid-way is first
copied into a buffer of its own, because Jackson replays the fields before
a late type id from a buffer whose parse context copies the first field's
value, not the object. String properties keep the literal, and any other
target (an enum, ...) keeps Jackson's ordinary error.
Compile. WorkflowCompiler works out which operators lie inside a loop
block (LoopBlockMembership, the frontend's block walk) and makes the
sidecar final before the physical plan serializes it into the executor's
descString. Inside a block it adds every whole-string `$name` of the
operator's JSON. Outside every block a string such as "$AAPL" stays the
literal it is on main, and a typed placeholder is a compile error naming
the property and the variable. Inside a block, an operator whose executor
runs generated code, or with an input fed from outside the block, is a
compile error too.
Worker. ExecFactory hands out a LateBoundExecutor when the descString's
sidecar is not empty. It keeps and forwards each state message, and binds
when the real executor is first needed, at the first tuple or finishing
port: the channel has delivered its state messages by then, so their
values are merged with the last one winning, and an inner loop's variable
shadows an outer one of the same name. It patches the placeholders,
coerced to their JSON type (a string property takes only a string, a
number or a boolean), builds the real executor through the normal factory
path, opens it and replays the kept state messages into it in order, so an
executor that acts on them (If) behaves as without references. Every later
call goes to that executor, which is never rebound. A variable still
missing then is an error naming it.
Closes apache#86351 parent 6defa2c commit 54ec7dc
16 files changed
Lines changed: 1757 additions & 6 deletions
File tree
- amber/src/test/integration/org/apache/texera/amber/engine/e2e
- common
- workflow-compiler/src
- main/scala/org/apache/texera/common/compiler
- test/scala/org/apache/texera/common/compiler
- workflow-core/src
- main/scala/org/apache/texera/amber
- core
- executor
- state
- util
- test/scala/org/apache/texera/amber/core
- executor
- state
- workflow-operator/src
- main/scala/org/apache/texera/amber/operator
- metadata
- test/scala/org/apache/texera/amber/operator/metadata
- workflow-compiling-service/src/main/scala/org/apache/texera/service
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
44 | 49 | | |
45 | 50 | | |
46 | 51 | | |
| |||
188 | 193 | | |
189 | 194 | | |
190 | 195 | | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
191 | 203 | | |
192 | 204 | | |
193 | 205 | | |
| |||
317 | 329 | | |
318 | 330 | | |
319 | 331 | | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
320 | 359 | | |
321 | 360 | | |
322 | 361 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
22 | 23 | | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
26 | | - | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
27 | 30 | | |
28 | | - | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
29 | 34 | | |
30 | 35 | | |
31 | 36 | | |
| |||
37 | 42 | | |
38 | 43 | | |
39 | 44 | | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
40 | 48 | | |
41 | 49 | | |
42 | 50 | | |
| |||
72 | 80 | | |
73 | 81 | | |
74 | 82 | | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
75 | 186 | | |
76 | 187 | | |
77 | 188 | | |
| |||
145 | 256 | | |
146 | 257 | | |
147 | 258 | | |
| 259 | + | |
148 | 260 | | |
149 | 261 | | |
150 | 262 | | |
151 | 263 | | |
152 | 264 | | |
153 | 265 | | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
154 | 273 | | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
155 | 277 | | |
156 | 278 | | |
157 | 279 | | |
| |||
0 commit comments