You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add end-to-end integration test for multi-region workflow execution #6133
Cross-region data delivery goes through input-port materialization reader threads (which replaced the cache-source operators in #3425). This path has no end-to-end coverage — it is exercised only by scheduling unit tests that count regions on a physical plan, so a regression in the materialization read/write path could pass CI unnoticed.
This matters right now because the now-dead cache-source plumbing is about to be removed (#6125 / #6126). The multi-region path — exactly what the removed cache sources used to serve — should get end-to-end coverage before that refactor.
Add an integration test that drives a big "X"-shaped workflow across multiple regions:
csvLeft ─┬─▶ join.build (port 0) ─┐
│ join ─▶ pythonSort ─▶ (terminal)
csvRight ─┼─▶ join.probe (port 1) ─┘
│
csvLeft ─┼─▶ union ───────────────────────────────────▶ (terminal)
csvRight ─┘ (two links fan into union's single port)
The hash join's probe-depends-on-build ordering forces the build input to be materialized, cutting the plan into ≥2 regions whose boundary is crossed by the reader-thread path. A Python sort UDF makes it a real end-to-end run with worker subprocesses.
Assertions:
Aspect
Expectation
Region count
plan spans ≥2 regions (via the same CostBasedScheduleGenerator the controller uses at runtime)
Join branch
self-join on the unique Order ID key → 100 rows, returned sorted by the Python UDF
Union branch
concatenation of both sources → 200 rows
Notes:
Python-tagged (@IntegrationTest) so it runs in the amber-integration CI job (which provisions Python deps); the lighter amber job excludes it.
Activity
test(amber): add X-shaped multi-region workflow e2e integration test (a…