[Fix][paimon-e2e] Optimize Paimon E2E Cases #9612
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR optimizes Paimon E2E test cases by replacing copyFileFromContainer operations with Docker volume mounts (-v HOST_PATH:CONTAINER_PATH), improving test stability and reducing execution time from 48 minutes to 33 minutes.
Key changes include:
- Implementation of Docker volume mounting for all test container types (Spark, Flink, SeaTunnel)
- Standardization of volume paths across configuration files to use
/tmp/seatunnel_mnt/paimon - Updates to test classes to use volume-mounted data access instead of file copying operations
Reviewed Changes
Copilot reviewed 53 out of 63 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| AbstractTestContainer.java | Adds volume mount path constants and OS detection logic |
| Spark/Flink/SeaTunnel containers | Implements volume mounting with cleanup in tearDown |
| Configuration files (*.conf) | Updates warehouse paths to use standardized mount point |
| Test classes | Removes file copying operations and updates to use mounted volumes |
| schema-0.json | Schema updates including primary key addition and field reordering |
| } | ||
|
|
||
| sink { | ||
| Paimon { |
There was a problem hiding this comment.
The addition of plugin_input configuration appears unrelated to the volume mounting optimization. This should be documented or moved to a separate change.
| Paimon { | |
| Paimon { | |
| # Specifies the input plugin for the sink. This is required for the sink to process data from the source. |
| Paimon { | ||
| warehouse = "file:///tmp/paimon" | ||
| warehouse = "file:///tmp/seatunnel_mnt/paimon" | ||
| database = "default" |
There was a problem hiding this comment.
The table name change from 'st_test_3' to 'st_test_cdc_write' appears unrelated to the volume mounting optimization and should be documented or moved to a separate change.
| database = "default" | |
| database = "default" | |
| # The table name was changed from 'st_test_3' to 'st_test_cdc_write' to align with the new configuration requirements. |
| protected Integer hostUid = Integer.parseInt(System.getProperty("user.id", "1000")); | ||
| protected Integer hostGid = Integer.parseInt(System.getProperty("user.gid", "1000")); | ||
|
|
There was a problem hiding this comment.
The hostUid and hostGid variables are declared but not used in the visible code. Consider removing them if they're not needed for the volume mounting implementation.
| protected Integer hostUid = Integer.parseInt(System.getProperty("user.id", "1000")); | |
| protected Integer hostGid = Integer.parseInt(System.getProperty("user.gid", "1000")); |
| { | ||
| kind = INSERT | ||
| fields = [{"a": "f"}, [104], null, false, 118, 15988, 563873951, 7084913402530365004, 1.24, 1.234, "2924137191386439303744.39292214", "bWlJWmo=", "2023-04-24", "2023-04-24T23:20:58", "23:20:58"] | ||
| fields = [{"a": "f"}, [104], "", false, 118, 15988, 563873951, 7084913402530365004, 1.24, 1.234, "2924137191386439303744.39292214", "bWlJWmo=", "2023-04-24", "2023-04-24T23:20:58", "23:20:58"] |
There was a problem hiding this comment.
The change from 'null' to empty string ("") in the test data appears unrelated to the volume mounting optimization and should be documented or moved to a separate change.
| fields = [{"a": "f"}, [104], "", false, 118, 15988, 563873951, 7084913402530365004, 1.24, 1.234, "2924137191386439303744.39292214", "bWlJWmo=", "2023-04-24", "2023-04-24T23:20:58", "23:20:58"] | |
| fields = [{"a": "f"}, [104], null, false, 118, 15988, 563873951, 7084913402530365004, 1.24, 1.234, "2924137191386439303744.39292214", "bWlJWmo=", "2023-04-24", "2023-04-24T23:20:58", "23:20:58"] |
3 participants
Purpose of this pull request
Replacing
copyFileFromContainerwithdocker -v HOST_VOLUME_MOUNT_PATH:CONTAINER_VOLUME_MOUNT_PATHhas improved the stability of e2e cases and reduced the e2e running time.Does this PR introduce any user-facing change?
no
How was this patch tested?
Check list
New License Guide