Skip to content

Add Escape Valve for check test results the old way#2992

Merged
alexperovich merged 4 commits intodotnet:masterfrom
alexperovich:fixForNonFlakyTests
Jun 7, 2019
Merged

Add Escape Valve for check test results the old way#2992
alexperovich merged 4 commits intodotnet:masterfrom
alexperovich:fixForNonFlakyTests

Conversation

@alexperovich
Copy link
Member

@RussKeldorph this will enable the old way to check results as an opt-in.

@alexperovich alexperovich self-assigned this Jun 6, 2019
<ItemGroup>
<AzurePipelinesExpectedTestFailure Include="PizzaTests.cheese"/>
<AzurePipelinesExpectedTestFailure Include="PizzaTests.cheese2"/>
<!-- <AzurePipelinesExpectedTestFailure Include="PizzaTests.cheese"/>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

? Commented out?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, I am gona revert this commit once I verify the change. This is to verify the thing I wrote will work in azdev.

@RussKeldorph
Copy link
Contributor

@BruceForstall @echesakovMSFT

@alexperovich alexperovich merged commit f6e72e7 into dotnet:master Jun 7, 2019
@alexperovich alexperovich deleted the fixForNonFlakyTests branch June 7, 2019 01:05
@RussKeldorph
Copy link
Contributor

I'm not extremely familiar with this issue, but, after talking to @BruceForstall and @echesakovMSFT, this new behavior of Azure Pipelines sounds broken. Are we tracking getting the AzP team to fix it?

@alexperovich @MattGal @chcosta

@alexperovich
Copy link
Member Author

I have a conversation ongoing with them. The problem is the api they pointed me too isn't run specific. If there isn't a run specific api then we will have to explore other options.

@echesakov
Copy link

@alexperovich I don't know much about the apis AzDO has but I am wondering whether it is possible to use the two apis:

  1. the old one to get all the test failures for given test run id;
  2. the new one for filtering out the flaky test based on the knowledge of all the failed non-flaky tests in a build;
    This could be a kind of intersection of the results of both queries?

@alexperovich
Copy link
Member Author

The new api is nothing more than the counts of test results per status. There isn't any documentation for it either so I don't know if there are other apis in the same set that I can use to get results for specific runs.

BruceForstall added a commit to BruceForstall/coreclr that referenced this pull request Jun 8, 2019
Fix bug where the first failure causes all other tests to fail.

See dotnet/arcade#2992
BruceForstall added a commit to BruceForstall/coreclr that referenced this pull request Jun 10, 2019
Fix bug where the first failure causes all other tests to fail.

See dotnet/arcade#2992
BruceForstall added a commit to BruceForstall/coreclr that referenced this pull request Jun 11, 2019
Fix bug where the first failure causes all other tests to fail.

See dotnet/arcade#2992
BruceForstall added a commit to BruceForstall/coreclr that referenced this pull request Jun 11, 2019
Fix bug where the first failure causes all other tests to fail.

See dotnet/arcade#2992
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants