Repository navigation
org.apache.amber.engine.e2e.DataProcessingSpec is flaky #3880
Description
What happened?
How to reproduce?
N/A
Version
1.1.0-incubating (Pre-release/Master)
Commit Hash (Optional)
No response
What browsers are you seeing the problem on?
No response
Relevant log output
Code of Conduct
- I agree to follow Apache Code of Conduct
Metadata
Metadata
Assignees
Labels
No labels
Activity
Xiao-zhen-Liu commentedon Oct 20, 2025
There are two kinds of flaky test behaviors, both related to e2e tests.
DataProcessingSpec. This test suite contains 10 test cases, and each sets a 1-minute timeout. There is a chance on CI environment that one of the the 10 test cases may time out.PauseSpec. This is appeared in PR 3913. The test cases in PauseSpec do not have timeouts internally, and seems to cause the test job to hang there until it reaches Github CI’s 6-hour timeout.Cause: I think the causes of both behaviors are the same. When we run these e2e tests, we execute a workflow and wait for controller to send workflow’s
COMPLETEDstatus. Somehow in the CI environment there is a chance this message is not sent (or not received) and causes the wait to either hang (forPauseSpec) or trigger our internal timeout (forDataProcessingSpec).Reproducibility: This seems to only happen in the CI environment. @aglinxinyuan mentioned he tested locally 10,000 times and could not reproduce this timeout issue.
Origin: Regarding the origin of this issue, I think it may be much earlier than recently, possibly since the creation of these test cases. It happens infrequently (maybe 1 in 50 CI runs) and usually rerunning the CI solves the problem. I found in March this year there was a 6-hour timeout in master-branch’s CI run, and the other flaky CI failure may be dated as early as Sep 2024 (earlier runs have been garbage-collected).
Short-term Solution: As this problem may be very hard to reproduce or debug, I suggest for all these test cases, we keep enforcing the 1-minute internal timeout but add a retry mechanism, as rerunning the test case always solves the issue. This will stabilize the CIs.
Permenant Solution: We should try to reproduce this issue, possibly in an Ubuntu environment, and locate the root cause of the
COMPLETEDstatus not being sent for these test cases.Yicong-Huang commentedon Oct 20, 2025
Thanks @Xiao-zhen-Liu for the investigations. I agree with your assessment. Could you please create new task issues for action items you listed?
aglinxinyuan commentedon Oct 20, 2025
We can try sub-issue.
fix(amber): add temporary fixes for e2e tests in Amber (#3962)
Yicong-Huang commentedon Apr 21, 2026
@Xiao-zhen-Liu is this fixed? There is not PR linked on deployment.
Xiao-zhen-Liu commentedon Apr 21, 2026
The test case failures no longer appear thanks to the retry mechanism. But we do not have a permanent fix until we can solve #3950.
chenlica commentedon Apr 22, 2026
@Xiao-zhen-Liu Let's do a discussion on that issue.
fix(amber): add temporary fixes for e2e tests in Amber (apache#3962)
fix(amber): add temporary fixes for e2e tests in Amber (apache#3962)