Skip to content

Commit cf75462

Browse files
authored
refactor(amber): count a materialized input port as completed once its END is processed (apache#8938)
### What changes were proposed in this PR? `InputManager.isPortCompleted` now treats a port read from materialization the same as a port fed by a link: it counts as completed once END_CHANNEL has been processed on it. `EndChannelHandler` marks that by setting `WorkerPort.completed`. Before, such a port reported its reader thread's `finished` flag. The thread sets that flag only after enqueuing END. So a DP thread waiting on the queue, typically the worker that got no tuples, could run `EndChannelHandler` before the flag was set. It then saw the port as not completed and skipped `finalizeOutput`. Nothing called it again, so the worker never finished and its region hung until the test deadline. That is the intermittent `TimeoutException: 1.minutes+30.seconds` in `LoopIntegrationSpec`. ``` Before: reader enqueues END -> DP processes END, flag not set yet -> output never finalized -> region hangs After: reader enqueues END -> DP processes END, port completed -> output finalized -> region completes ``` Two related effects of the change: - The flag could also be set while END was still queued. A worker with a second materialized input no longer finalizes before that port's END has been processed. - `startInputPortReaderThreads` uses the same check to skip ports that already completed. The second `StartWorker` of a two-phase region arrives only after the coordinator has the port's `PortCompleted`, which the worker sends after processing END. So it skips the same ports as before. - Nothing reads the reader thread's `finished` flag any more, so it is removed, and it can't stand in for port completion again. The reader-thread tests that read it now check the END_CHANNEL the reader enqueues instead. ### Any related issues, documentation, discussions? Closes apache#8937. The two `LoopIntegrationSpec` cases added in apache#8639 hit the race most often; the race itself dates from apache#3460. ### How was this PR tested? Tests first: the first commit adds the tests and fails on `main`, and the second commit makes them pass. - `InputManagerSpec`: a materialized port is completed once END has been processed, even before its reader thread has returned. It is not completed while END is still queued, even after the thread has returned. - `EndChannelHandlerSpec`: when END arrives on a materialized input port before its reader thread has returned, the worker emits the output `FinalizePort` and `FinalizeExecutor`. On `main` it emits only the input port's `FinalizePort`, which is the hang. ``` sbt "WorkflowExecutionService/testOnly *InputManagerSpec *EndChannelHandlerSpec *StartHandlerSpec *DataProcessorSpec *EndHandlerSpec *StartChannelHandlerSpec *AssignPortHandlerSpec *DPThreadSpec *WorkerSpec" sbt "WorkflowExecutionService/scalafmtCheckAll" "WorkflowExecutionService/scalafixAll --check" ``` `InputPortMaterializationReaderThreadSpec` reads real Iceberg tables. Locally only its two cases that don't need storage ran; CI runs the rest. `LoopIntegrationSpec`, run locally on Windows with the sbt JVM and its Python workers pinned to 3 CPUs (affinity mask `0x7`), one case per JVM: | Case | `main` | This PR | |---|---|---| | bind a `$i` reference after two branches of the loop body meet again | 3 of 5 runs hung on the first attempt; 1 also hung on the retry and failed as on CI | 10 of 10 passed on the first attempt, 42–44 s each | | bind the inner loop's variable where it shadows the outer loop's | (fails on CI, see apache#8937) | 10 of 10 passed on the first attempt, 64–70 s each | Unpinned, the whole spec passes with this PR: 8 of 8 cases, 29–59 s each, no retries. ``` AMBER_TEST_FILTER= STORAGE_ICEBERG_CATALOG_TYPE=postgres sbt "WorkflowExecutionService/testOnly *LoopIntegrationSpec -- -z \"after two branches of the loop body meet again\"" AMBER_TEST_FILTER=integration-only STORAGE_ICEBERG_CATALOG_TYPE=postgres sbt "WorkflowExecutionService/testOnly *LoopIntegrationSpec" ``` ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
1 parent d99fb4d commit cf75462

5 files changed

Lines changed: 110 additions & 56 deletions

File tree

‎amber/src/main/scala/org/apache/texera/amber/engine/architecture/messaginglayer/InputManager.scala‎

Lines changed: 4 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -116,20 +116,11 @@ class InputManager(
116116
def getPort(portId: PortIdentity): WorkerPort = ports(portId)
117117

118118
/**
119-
* For ports that read from materialization, the port completion is marked by the finish of the reader thread.
120-
* For other ports that connect to upstream links, the completion is marked by the completion the port.
119+
* Whether END_CHANNEL has been processed on every channel of the port (`EndChannelHandler` then
120+
* marks it completed), for a port read from materialization as well: what its reader thread has
121+
* done says nothing about whether the DP thread has processed the END it enqueued.
121122
*/
122-
def isPortCompleted(portId: PortIdentity): Boolean = {
123-
if (
124-
!this.inputPortMaterializationReaderThreads
125-
.contains(portId) || this.inputPortMaterializationReaderThreads(portId).isEmpty
126-
) {
127-
this.getPort(portId).completed
128-
} else {
129-
val existingThread = this.inputPortMaterializationReaderThreads(portId).head
130-
existingThread.finished
131-
}
132-
}
123+
def isPortCompleted(portId: PortIdentity): Boolean = this.getPort(portId).completed
133124

134125
def hasUnfinishedInput: Boolean = inputBatch != null && currentInputIdx + 1 < inputBatch.length
135126

‎amber/src/main/scala/org/apache/texera/amber/engine/architecture/worker/managers/InputPortMaterializationReaderThread.scala‎

Lines changed: 1 addition & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ import org.apache.texera.amber.util.VirtualIdentityUtils.getFromActorIdForInputP
5555

5656
import java.net.URI
5757
import java.util.concurrent.LinkedBlockingQueue
58-
import java.util.concurrent.atomic.{AtomicBoolean, AtomicLong}
58+
import java.util.concurrent.atomic.AtomicLong
5959
import scala.collection.mutable.ArrayBuffer
6060

6161
class InputPortMaterializationReaderThread(
@@ -75,12 +75,6 @@ class InputPortMaterializationReaderThread(
7575
}
7676
private val partitioner = toPartitioner(partitioning, workerActorId)
7777
private val batchSize = ApplicationConfig.defaultDataTransferBatchSize
78-
private val isFinished = new AtomicBoolean(false)
79-
80-
/**
81-
* Whether the reader thread has completed.
82-
*/
83-
def finished: Boolean = isFinished.get()
8478

8579
/**
8680
* Read from the materialization storage, and mimics the behavior of an upstream worker's output manager.
@@ -141,7 +135,6 @@ class InputPortMaterializationReaderThread(
141135
// Flush any remaining tuples in the buffer.
142136
if (buffer.nonEmpty) flush()
143137
emitECM(METHOD_END_CHANNEL, PORT_ALIGNMENT)
144-
isFinished.set(true)
145138
} catch {
146139
case e: Exception =>
147140
throw new RuntimeException(s"Error reading input port materializations: ${e.getMessage}", e)

‎amber/src/test/scala/org/apache/texera/amber/engine/architecture/messaginglayer/InputManagerSpec.scala‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -215,6 +215,49 @@ class InputManagerSpec extends AnyFlatSpec {
215215
assert(mgr.isPortCompleted(portId))
216216
}
217217

218+
// ---------------------------------------------------------------------------
219+
// isPortCompleted — materialized path
220+
// ---------------------------------------------------------------------------
221+
//
222+
// A port read from materialization is fed by a reader thread that enqueues END_CHANNEL. The DP
223+
// thread may process that END before the thread returns, and the thread may return while END is
224+
// still queued, so the port is completed when END has been processed (`WorkerPort.completed`,
225+
// set by EndChannelHandler), whatever state the reader thread is in.
226+
227+
private val materializedUri = new URI("file:///nowhere")
228+
229+
private def broadcastToSelf: Partitioning =
230+
BroadcastPartitioning(batchSize = 1, channels = Seq(channelId("upstream", "worker-1")))
231+
232+
"InputManager.isPortCompleted (materialized)" should
233+
"be true once END has been processed, before the reader thread has returned" in {
234+
val mgr = freshManager
235+
val portId = PortIdentity(0)
236+
mgr.addPort(portId, schema, List(materializedUri), List(broadcastToSelf))
237+
val reader = mgr.getInputPortReaderThreads(portId).head
238+
assert(!mgr.isPortCompleted(portId))
239+
240+
// What EndChannelHandler does on END, while the reader thread has not even started.
241+
mgr.getPort(portId).completed = true
242+
243+
assert(reader.getState == Thread.State.NEW)
244+
assert(mgr.isPortCompleted(portId))
245+
}
246+
247+
it should "be false while END is still queued, even after the reader thread has returned" in {
248+
val mgr = freshManager
249+
val portId = PortIdentity(0)
250+
mgr.addPort(portId, schema, List.empty, List.empty)
251+
val returnedReader = new NoOpReaderThread
252+
returnedReader.start()
253+
returnedReader.join()
254+
installReaderThreads(mgr, portId, List(returnedReader))
255+
256+
// The reader enqueued END and returned, but the DP thread has not processed END yet.
257+
assert(returnedReader.getState == Thread.State.TERMINATED)
258+
assert(!mgr.isPortCompleted(portId))
259+
}
260+
218261
// ---------------------------------------------------------------------------
219262
// startInputPortReaderThreads — no-op when nothing is registered
220263
// ---------------------------------------------------------------------------
@@ -276,6 +319,8 @@ class InputManagerSpec extends AnyFlatSpec {
276319
val alreadyStarted = new NoOpReaderThread
277320
alreadyStarted.start()
278321
alreadyStarted.join() // now TERMINATED, so a second start() is guaranteed to fail
322+
// Registered first, as addPort always does before building a port's reader threads.
323+
mgr.addPort(PortIdentity(0), schema, urisToRead = List.empty, partitionings = List.empty)
279324
installReaderThreads(mgr, PortIdentity(0), List(alreadyStarted))
280325

281326
val thrown = intercept[RuntimeException] {

‎amber/src/test/scala/org/apache/texera/amber/engine/architecture/worker/managers/InputPortMaterializationReaderThreadSpec.scala‎

Lines changed: 18 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -157,20 +157,15 @@ class InputPortMaterializationReaderThreadSpec extends AnyFlatSpec {
157157
hashAttributeNames = Seq("id")
158158
)
159159

160-
/**
161-
* Runs the reader inline and returns everything it enqueued, in order, together with the reader
162-
* itself so its completion flag can be read.
163-
*/
160+
/** Runs the reader inline and returns everything it enqueued, in order. */
164161
private def replay(
165162
uri: URI,
166163
partitioning: Partitioning,
167164
readerFor: ActorVirtualIdentity = workerId
168-
): (InputPortMaterializationReaderThread, Seq[WorkflowFIFOMessage]) = {
165+
): Seq[WorkflowFIFOMessage] = {
169166
val queue = new LinkedBlockingQueue[DPInputQueueElement]()
170-
val reader =
171-
new InputPortMaterializationReaderThread(uri, queue, readerFor, partitioning)
172-
reader.run()
173-
(reader, drain(queue))
167+
new InputPortMaterializationReaderThread(uri, queue, readerFor, partitioning).run()
168+
drain(queue)
174169
}
175170

176171
private def drain(queue: LinkedBlockingQueue[DPInputQueueElement]): Seq[WorkflowFIFOMessage] = {
@@ -197,9 +192,9 @@ class InputPortMaterializationReaderThreadSpec extends AnyFlatSpec {
197192

198193
behavior of "InputPortMaterializationReaderThread"
199194

200-
it should "bracket the replay with START_CHANNEL and END_CHANNEL and mark itself finished" in {
195+
it should "bracket the replay with START_CHANNEL and END_CHANNEL" in {
201196
val uri = materialize("bracket", tuples = Seq(tuple(1)))
202-
val (reader, messages) = replay(uri, broadcastTo(workerId))
197+
val messages = replay(uri, broadcastTo(workerId))
203198

204199
// The markers are the first and last things on the channel: a downstream input port that saw
205200
// data before START_CHANNEL, or END_CHANNEL before the last batch, would mis-order the port.
@@ -218,26 +213,24 @@ class InputPortMaterializationReaderThreadSpec extends AnyFlatSpec {
218213
}
219214
assert(start.ecmType == NO_ALIGNMENT)
220215
assert(end.ecmType == PORT_ALIGNMENT)
221-
assert(reader.finished)
222216
}
223217

224-
it should "not report itself finished before the replay runs" in {
218+
it should "enqueue nothing before the replay runs" in {
225219
val queue = new LinkedBlockingQueue[DPInputQueueElement]()
226-
val reader = new InputPortMaterializationReaderThread(
220+
new InputPortMaterializationReaderThread(
227221
baseURI("unstarted"),
228222
queue,
229223
workerId,
230224
broadcastTo(workerId)
231225
)
232226

233-
assert(!reader.finished)
234227
assert(queue.isEmpty)
235228
}
236229

237230
it should "send every message on one channel from a URI-derived sender, numbered from zero" in {
238231
val uri =
239232
materialize("channel", tuples = Seq(tuple(1)), states = Seq((State(Map("k" -> 1)), 0L, "")))
240-
val (_, messages) = replay(uri, broadcastTo(workerId))
233+
val messages = replay(uri, broadcastTo(workerId))
241234

242235
// The dummy sender is derived from the URI *and* the receiving worker, so two workers reading
243236
// the same materialization do not share a channel (and neither collides with a real upstream).
@@ -260,7 +253,7 @@ class InputPortMaterializationReaderThreadSpec extends AnyFlatSpec {
260253
)
261254
val uri = materialize("states", tuples = Seq(tuple(1), tuple(2)), states = states)
262255

263-
val (_, messages) = replay(uri, broadcastTo(workerId))
256+
val messages = replay(uri, broadcastTo(workerId))
264257

265258
// Downstream operators need their state set up before the tuples arrive, so every state
266259
// precedes every tuple regardless of how the two tables were written.
@@ -277,7 +270,7 @@ class InputPortMaterializationReaderThreadSpec extends AnyFlatSpec {
277270
it should "emit no state frame when the port materialized no state" in {
278271
val uri = materialize("nostate", tuples = Seq(tuple(1)))
279272

280-
val (_, messages) = replay(uri, broadcastTo(workerId))
273+
val messages = replay(uri, broadcastTo(workerId))
281274

282275
assert(stateFrames(messages).isEmpty)
283276
assert(deliveredIds(messages) == Seq(1))
@@ -287,7 +280,7 @@ class InputPortMaterializationReaderThreadSpec extends AnyFlatSpec {
287280
val ids = 1 to 5
288281
val uri = materialize("broadcast", tuples = ids.map(tuple))
289282

290-
val (_, messages) = replay(uri, broadcastTo(workerId, siblingWorkerId))
283+
val messages = replay(uri, broadcastTo(workerId, siblingWorkerId))
291284

292285
// Under the batch size, everything leaves in the single trailing flush.
293286
assert(dataFrames(messages).size == 1)
@@ -299,7 +292,7 @@ class InputPortMaterializationReaderThreadSpec extends AnyFlatSpec {
299292

300293
// Broadcast to the sibling only: `allReceivers` never contains this worker, so the membership
301294
// test fails for every tuple.
302-
val (reader, messages) = replay(uri, broadcastTo(siblingWorkerId))
295+
val messages = replay(uri, broadcastTo(siblingWorkerId))
303296

304297
assert(deliveredIds(messages).isEmpty)
305298
assert(dataFrames(messages).isEmpty)
@@ -310,16 +303,15 @@ class InputPortMaterializationReaderThreadSpec extends AnyFlatSpec {
310303
EmbeddedControlMessageIdentity("EndChannel")
311304
)
312305
)
313-
assert(reader.finished)
314306
}
315307

316308
it should "split the materialization between the workers of a hash-shuffled link" in {
317309
val ids = 1 to 20
318310
val uri = materialize("shuffle", tuples = ids.map(tuple))
319311
val partitioning = hashShuffleTo(workerId, siblingWorkerId)
320312

321-
val (_, mine) = replay(uri, partitioning)
322-
val (_, siblings) = replay(uri, partitioning, readerFor = siblingWorkerId)
313+
val mine = replay(uri, partitioning)
314+
val siblings = replay(uri, partitioning, readerFor = siblingWorkerId)
323315

324316
// Each worker's reader takes exactly its own buckets: together they cover the materialization
325317
// once. A filter that ignored the partitioner would duplicate every tuple; one that dropped
@@ -337,7 +329,7 @@ class InputPortMaterializationReaderThreadSpec extends AnyFlatSpec {
337329
val ids = 1 to (batchSize + 1)
338330
val uri = materialize("batching", tuples = ids.map(tuple))
339331

340-
val (_, messages) = replay(uri, broadcastTo(workerId))
332+
val messages = replay(uri, broadcastTo(workerId))
341333

342334
assert(dataFrames(messages).map(_.frame.length) == Seq(batchSize, 1))
343335
assert(deliveredIds(messages) == ids)
@@ -358,9 +350,8 @@ class InputPortMaterializationReaderThreadSpec extends AnyFlatSpec {
358350
assert(failure.getMessage.startsWith("Error reading input port materializations: "))
359351
assert(failure.getCause != null)
360352
assert(failure.getMessage.endsWith(failure.getCause.getMessage))
361-
// START_CHANNEL is emitted before the try, so the marker is already on the queue...
353+
// START_CHANNEL is emitted before the try, so the marker is already on the queue, but
354+
// END_CHANNEL never is: the port is never completed.
362355
assert(ecmIds(drain(queue)) == Seq(EmbeddedControlMessageIdentity("StartChannel")))
363-
// ...but the channel never completes, which is what stops the port being called complete.
364-
assert(!reader.finished)
365356
}
366357
}

‎amber/src/test/scala/org/apache/texera/amber/engine/architecture/worker/promisehandlers/EndChannelHandlerSpec.scala‎

Lines changed: 42 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,10 @@ import org.apache.texera.amber.engine.architecture.rpc.controlcommands.{
4444
EmptyRequest
4545
}
4646
import org.apache.texera.amber.engine.architecture.rpc.controlreturns.EmptyReturn
47-
import org.apache.texera.amber.engine.architecture.sendsemantics.partitionings.OneToOnePartitioning
47+
import org.apache.texera.amber.engine.architecture.sendsemantics.partitionings.{
48+
BroadcastPartitioning,
49+
OneToOnePartitioning
50+
}
4851
import org.apache.texera.amber.engine.architecture.worker.WorkflowWorker.{
4952
DPInputQueueElement,
5053
MainThreadDelegateMessage
@@ -60,6 +63,7 @@ import org.apache.texera.amber.engine.common.ambermessage.{StateFrame, WorkflowF
6063
import org.apache.texera.amber.engine.common.virtualidentity.util.COORDINATOR
6164
import org.scalatest.flatspec.AnyFlatSpec
6265

66+
import java.net.URI
6367
import java.util.concurrent.LinkedBlockingQueue
6468
import scala.collection.mutable.ArrayBuffer
6569
import scala.util.control.ControlThrowable
@@ -88,11 +92,10 @@ import scala.util.control.ControlThrowable
8892
* The `forall` in step 5 is why the fixture always registers two input ports: a worker whose
8993
* other input port is still open must NOT finalize its output, or the downstream region would
9094
* see the executor finish while data is still arriving on the second port. The fixture pins that
91-
* `forall`, but only over link-fed ports: `InputManager.isPortCompleted` branches, and for a port
92-
* fed from materialization it reports the reader THREAD's `finished` flag and ignores
93-
* `completed` entirely. That arm needs storage-backed input ports, which this fixture cannot
94-
* build without real Iceberg documents, so substituting `getPort(p).completed` for
95-
* `isPortCompleted(p)` is not distinguishable here — a known, stated gap rather than coverage.
95+
* `forall`. A port fed from materialization is completed the same way, by the END this handler
96+
* processes, whatever its reader thread has done since enqueuing that END, so the fixture can
97+
* register such a port with a reader thread that never starts and still expect the output to be
98+
* finalized.
9699
*
97100
* `produceStateOnFinish` has no override in main today — `OperatorExecutor` returns `None` for
98101
* every port — so the "operator produced a state" arm is reachable only through a test executor.
@@ -140,10 +143,14 @@ class EndChannelHandlerSpec extends AnyFlatSpec {
140143
* @param otherInputPortCompleted whether the *other* input port has already finished. The
141144
* common single-input case behaves like `true`: the channel
142145
* being ended is the last one open.
146+
* @param currentPortMaterialized whether the ended port is read from materialization, by a
147+
* reader thread that is never started here, so it has not
148+
* returned when END is processed.
143149
*/
144150
private class Fixture(
145151
executor: OperatorExecutor,
146-
otherInputPortCompleted: Boolean = true
152+
otherInputPortCompleted: Boolean = true,
153+
currentPortMaterialized: Boolean = false
147154
) {
148155
val sent: ArrayBuffer[WorkflowFIFOMessage] = ArrayBuffer()
149156

@@ -158,7 +165,16 @@ class EndChannelHandlerSpec extends AnyFlatSpec {
158165

159166
// Two input ports on two different channels; only one of them is the channel the ECM came in
160167
// on, so a handler that picked an arbitrary channel would read the wrong port.
161-
dp.inputManager.addPort(currentPortId, schema, List.empty, List.empty)
168+
if (currentPortMaterialized) {
169+
dp.inputManager.addPort(
170+
currentPortId,
171+
schema,
172+
List(new URI("file:///end-channel-spec")),
173+
List(BroadcastPartitioning(batchSize, Seq(currentChannelId)))
174+
)
175+
} else {
176+
dp.inputManager.addPort(currentPortId, schema, List.empty, List.empty)
177+
}
162178
dp.inputManager.addPort(otherPortId, schema, List.empty, List.empty)
163179
dp.inputGateway.getChannel(currentChannelId).setPortId(currentPortId)
164180
dp.inputGateway.getChannel(otherChannelId).setPortId(otherPortId)
@@ -427,6 +443,24 @@ class EndChannelHandlerSpec extends AnyFlatSpec {
427443
assert(executor.calls.map(c => (c._1, c._2)) == Seq(("state", 3), ("finish", 3)))
428444
}
429445

446+
it should "finalize the output when the last input port is read from materialization, before its reader thread has returned" in {
447+
val fixture = new Fixture(new RecordingExecutor(), currentPortMaterialized = true)
448+
val reader = fixture.dp.inputManager.getInputPortReaderThreads(currentPortId).head
449+
450+
fixture.endChannel()
451+
452+
// The DP thread can process END before the reader thread that enqueued it returns. Leaving
453+
// the output unfinalized then would hang the region: nothing ends it later.
454+
assert(reader.getState == Thread.State.NEW)
455+
assert(
456+
fixture.drainOutput() == List(
457+
FinalizePort(currentPortId, input = true),
458+
FinalizePort(outputPortId, input = false),
459+
FinalizeExecutor()
460+
)
461+
)
462+
}
463+
430464
it should "finalize the output once the last input port completes" in {
431465
val fixture = new Fixture(new RecordingExecutor(), otherInputPortCompleted = true)
432466

0 commit comments

Comments
 (0)