Skip to content

Add end-to-end integration test for multi-region workflow execution #6133

Closed

Description

@aglinxinyuan

Task Summary

Cross-region data delivery goes through input-port materialization reader threads (which replaced the cache-source operators in #3425). This path has no end-to-end coverage — it is exercised only by scheduling unit tests that count regions on a physical plan, so a regression in the materialization read/write path could pass CI unnoticed.

This matters right now because the now-dead cache-source plumbing is about to be removed (#6125 / #6126). The multi-region path — exactly what the removed cache sources used to serve — should get end-to-end coverage before that refactor.

Add an integration test that drives a big "X"-shaped workflow across multiple regions:

csvLeft  ─┬─▶ join.build (port 0) ─┐
          │                         join ─▶ pythonSort ─▶ (terminal)
csvRight ─┼─▶ join.probe (port 1) ─┘
          │
csvLeft  ─┼─▶ union ───────────────────────────────────▶ (terminal)
csvRight ─┘   (two links fan into union's single port)

The hash join's probe-depends-on-build ordering forces the build input to be materialized, cutting the plan into ≥2 regions whose boundary is crossed by the reader-thread path. A Python sort UDF makes it a real end-to-end run with worker subprocesses.

Assertions:

Aspect Expectation
Region count plan spans ≥2 regions (via the same CostBasedScheduleGenerator the controller uses at runtime)
Join branch self-join on the unique Order ID key → 100 rows, returned sorted by the Python UDF
Union branch concatenation of both sources → 200 rows

Notes:

Task Type

  • Test / Coverage

Activity

  1. added theissue type on Jul 5, 2026
  2. added a commit that references this issue on Jul 5, 2026
    fd3fabc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions