Unit test case stabilization: Defect_MAGN_159/164#747
Closed
benglin wants to merge 2 commits intoDynamoDS:masterfrom
Closed
Unit test case stabilization: Defect_MAGN_159/164#747benglin wants to merge 2 commits intoDynamoDS:masterfrom
benglin wants to merge 2 commits intoDynamoDS:masterfrom
Conversation
… cases pass Conflicts: src/DynamoCoreUITests/RecordedTests.cs
Collaborator
|
Can we aim for a deterministic scenario instead where we hold the test open until the execution has completed. Anything to do with timing is a difficult tradeoff between the length of time to test, and reliability of the test set. I'd rather adjust the design of the software to make it easier to test deterministically. Did you explore that option? is it possible? |
This was referenced Jun 6, 2019
Closed
Merged
Closed
Closed
This was referenced Jun 21, 2019
Closed
Closed
Closed
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
As command framework plays back the commands, the view gets destroyed 10ms after the last command is played back. This leaves very little time for Dynamo's background thread (EvaluationThread in auto-run mode) to produce result of computation. It appears that if these two tests (Defect_MAGN_159/164) are placed at the very beginning of RecordedTests test suite (due to their names in alphabetical order), then they tend to fail. And since 159 is placed before 164, it is a lot more likely to fail as compared to 164.
Possible fixes
Even though it is possible to configure the shutdown time for each test case by altering the value of PauseAfterPlaybackInMs in recorded command XML file to a higher value, it has the effect of accumulating if most command XML increase the value. That results in longer overall unit test duration.
This solution attempts to add a constant upfront delay by having _SnowPeashooter test case which shuts down after 2 seconds, giving enough time for the system (Dynamo runtime) to "warm up", before proceeding to execute the next test case Defect_MAGN_159.
For observation
This seems to be working on ChocoButterUI from the look of it, but we won't know until we see more build results in near future. So the ShowPeashooter is just an experimental build fix that may or may not be feasible. If it is not helping for some scenario (where the test machine is heavily taxed), then we will fallback to using PauseAfterPlaybackInMs in those failing XML.