Skip to content

TeamCity Reporter. Remove flow id from name of test suites.#3054

Merged
bradwilson merged 2 commits intoxunit:v2from
A6blpka:v2_fix_teamcity_history_work
Nov 8, 2024
Merged

TeamCity Reporter. Remove flow id from name of test suites.#3054
bradwilson merged 2 commits intoxunit:v2from
A6blpka:v2_fix_teamcity_history_work

Conversation

@A6blpka
Copy link

@A6blpka A6blpka commented Nov 7, 2024

Request fixes this issue xunit/visualstudio.xunit#394

The output name of the test suites does not need a flow id.
I can assume that during the development of xunit.runner.visualstudio 2.5 version, flow id generation for test suites was fixed, which caused test suites to stop being constant, which breaks the history of test execution on Teamcity.

@A6blpka
Copy link
Author

A6blpka commented Nov 7, 2024

@dotnet-policy-service agree

@bradwilson
Copy link
Member

bradwilson commented Nov 7, 2024

This was done purposefully.

Collection names are not required to be globally unique, so in order to differentiate test collections from one assembly from test collections with the same names in another assembly, the collection unique ID must be part of the name.

In addition, although you're trying to "unbreak" your own history, this change would break history for other people.

@bradwilson bradwilson added the By Design The issue is working as designed label Nov 7, 2024
@A6blpka
Copy link
Author

A6blpka commented Nov 7, 2024 •
edited
Loading

This was done purposefully.

Collection names are not required to be globally unique, so in order to differentiate test collections from one assembly from test collections with the same names in another assembly, the collection unique ID must be part of the name.

In addition, although you're trying to "unbreak" your own history, this change would break history for other people.

No, I'm not trying to break my history, it's already broken after the xunit.runner.visualstudio update, which was not immediately noticed.
At the moment each test is unique, because the GUID is used to create the collection name, which is new every time, and now for any test the history consists of one item.

@bradwilson
Copy link
Member

bradwilson commented Nov 7, 2024

Can you prove a repro project that illustrates this?

@bradwilson
Copy link
Member

bradwilson commented Nov 7, 2024 •
edited
Loading

Here is an example of running a few tests in a test project:

##teamcity[flowStarted timestamp='2024-11-07T19:05:53.271+0000' flowId='c9e3fe690c20413a042f5fbf49864d467a8e39c85ed2a27692d6e75ed67d56d1']
##teamcity[testSuiteStarted timestamp='2024-11-07T19:05:53.271+0000' flowId='c9e3fe690c20413a042f5fbf49864d467a8e39c85ed2a27692d6e75ed67d56d1' name='C:|0x005CDev|0x005Cxunit|0x005Cxunit|0x005Csrc|0x005Cxunit.v3.assert.tests|0x005Cbin|0x005CDebug|0x005Cnet472|0x005Cxunit.v3.assert.tests.exe']
##teamcity[flowStarted timestamp='2024-11-07T19:05:53.286+0000' flowId='5df1a65569b127dd4a2f5fc940183ac92a7bdc2cc9202f2e65146633b52629c0' parent='c9e3fe690c20413a042f5fbf49864d467a8e39c85ed2a27692d6e75ed67d56d1']
##teamcity[testSuiteStarted timestamp='2024-11-07T19:05:53.286+0000' flowId='5df1a65569b127dd4a2f5fc940183ac92a7bdc2cc9202f2e65146633b52629c0' name='Test collection for BooleanAssertsTests+False (id: cb82e8ecacbab286cb49cdb09257c46444784190d9768a2ed3feb31f55021add) (5df1a65569b127dd4a2f5fc940183ac92a7bdc2cc9202f2e65146633b52629c0)']
##teamcity[testStarted timestamp='2024-11-07T19:05:53.320+0000' flowId='5df1a65569b127dd4a2f5fc940183ac92a7bdc2cc9202f2e65146633b52629c0' name='BooleanAssertsTests+False.AssertFalse']
##teamcity[testFinished timestamp='2024-11-07T19:05:53.343+0000' flowId='5df1a65569b127dd4a2f5fc940183ac92a7bdc2cc9202f2e65146633b52629c0' name='BooleanAssertsTests+False.AssertFalse' duration='9']
##teamcity[testStarted timestamp='2024-11-07T19:05:53.345+0000' flowId='5df1a65569b127dd4a2f5fc940183ac92a7bdc2cc9202f2e65146633b52629c0' name='BooleanAssertsTests+False.UserSuppliedMessage']
##teamcity[testFinished timestamp='2024-11-07T19:05:53.346+0000' flowId='5df1a65569b127dd4a2f5fc940183ac92a7bdc2cc9202f2e65146633b52629c0' name='BooleanAssertsTests+False.UserSuppliedMessage' duration='2']
##teamcity[testStarted timestamp='2024-11-07T19:05:53.346+0000' flowId='5df1a65569b127dd4a2f5fc940183ac92a7bdc2cc9202f2e65146633b52629c0' name='BooleanAssertsTests+False.ThrowsExceptionWhenTrue']
##teamcity[testFinished timestamp='2024-11-07T19:05:53.347+0000' flowId='5df1a65569b127dd4a2f5fc940183ac92a7bdc2cc9202f2e65146633b52629c0' name='BooleanAssertsTests+False.ThrowsExceptionWhenTrue' duration='1']
##teamcity[testStarted timestamp='2024-11-07T19:05:53.347+0000' flowId='5df1a65569b127dd4a2f5fc940183ac92a7bdc2cc9202f2e65146633b52629c0' name='BooleanAssertsTests+False.ThrowsExceptionWhenNull']
##teamcity[testFinished timestamp='2024-11-07T19:05:53.348+0000' flowId='5df1a65569b127dd4a2f5fc940183ac92a7bdc2cc9202f2e65146633b52629c0' name='BooleanAssertsTests+False.ThrowsExceptionWhenNull' duration='0']
##teamcity[testSuiteFinished timestamp='2024-11-07T19:05:53.350+0000' flowId='5df1a65569b127dd4a2f5fc940183ac92a7bdc2cc9202f2e65146633b52629c0' name='Test collection for BooleanAssertsTests+False (id: cb82e8ecacbab286cb49cdb09257c46444784190d9768a2ed3feb31f55021add) (5df1a65569b127dd4a2f5fc940183ac92a7bdc2cc9202f2e65146633b52629c0)']
##teamcity[flowFinished timestamp='2024-11-07T19:05:53.350+0000' flowId='5df1a65569b127dd4a2f5fc940183ac92a7bdc2cc9202f2e65146633b52629c0']
##teamcity[testSuiteFinished timestamp='2024-11-07T19:05:53.356+0000' flowId='c9e3fe690c20413a042f5fbf49864d467a8e39c85ed2a27692d6e75ed67d56d1' name='C:|0x005CDev|0x005Cxunit|0x005Cxunit|0x005Csrc|0x005Cxunit.v3.assert.tests|0x005Cbin|0x005CDebug|0x005Cnet472|0x005Cxunit.v3.assert.tests.exe']
##teamcity[flowFinished timestamp='2024-11-07T19:05:53.356+0000' flowId='c9e3fe690c20413a042f5fbf49864d467a8e39c85ed2a27692d6e75ed67d56d1']

After a clean & rebuild, I get the exact same values for the flow IDs and collection unique ID.

@bradwilson
Copy link
Member

bradwilson commented Nov 7, 2024

Never mind, the ID stability is in the v3 tree, not the v2 tree. Sorry for the confusion.

@A6blpka
Copy link
Author

A6blpka commented Nov 7, 2024

Anything empty test

    public class UnitTest1
    {
        [Fact]
        public void Test1()
        {

        }
    }

VSTest version 17.11.1 (x64)

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.8.2+699d445a1a (64-bit .NET 6.0.35)
[xUnit.net 00:00:00.06]   Discovering: TeamcityXunitTestProject
[xUnit.net 00:00:00.09]   Discovered:  TeamcityXunitTestProject
[xUnit.net 00:00:00.09]   Starting:    TeamcityXunitTestProject
[xUnit.net 00:00:00.10] ##teamcity[flowStarted timestamp='2024-11-07T19:16:04.478+0000' flowId='TeamcityXunitTestProject.dll']
[xUnit.net 00:00:00.10] ##teamcity[testSuiteStarted timestamp='2024-11-07T19:16:04.478+0000' flowId='TeamcityXunitTestProject.dll' name='TeamcityXunitTestProject.dll']
[xUnit.net 00:00:00.11] ##teamcity[flowStarted timestamp='2024-11-07T19:16:04.485+0000' flowId='1173d19cf8844fb2b5aa19133ed4f22b' parent='TeamcityXunitTestProject.dll']
[xUnit.net 00:00:00.11] ##teamcity[testSuiteStarted timestamp='2024-11-07T19:16:04.485+0000' flowId='1173d19cf8844fb2b5aa19133ed4f22b' name='Test collection for TeamcityXunitTestProject.UnitTest1 (1173d19cf8844fb2b5aa19133ed4f22b)']
[xUnit.net 00:00:00.12] ##teamcity[testStarted timestamp='2024-11-07T19:16:04.502+0000' flowId='1173d19cf8844fb2b5aa19133ed4f22b' name='TeamcityXunitTestProject.UnitTest1.Test1']
[xUnit.net 00:00:00.14] ##teamcity[testFinished timestamp='2024-11-07T19:16:04.515+0000' flowId='1173d19cf8844fb2b5aa19133ed4f22b' name='TeamcityXunitTestProject.UnitTest1.Test1' duration='1']
[xUnit.net 00:00:00.14] ##teamcity[testSuiteFinished timestamp='2024-11-07T19:16:04.516+0000' flowId='1173d19cf8844fb2b5aa19133ed4f22b' name='Test collection for TeamcityXunitTestProject.UnitTest1 (1173d19cf8844fb2b5aa19133ed4f22b)']
[xUnit.net 00:00:00.14] ##teamcity[flowFinished timestamp='2024-11-07T19:16:04.516+0000' flowId='1173d19cf8844fb2b5aa19133ed4f22b']
[xUnit.net 00:00:00.14] ##teamcity[testSuiteFinished timestamp='2024-11-07T19:16:04.517+0000' flowId='TeamcityXunitTestProject.dll' name='TeamcityXunitTestProject.dll']
[xUnit.net 00:00:00.14] ##teamcity[flowFinished timestamp='2024-11-07T19:16:04.517+0000' flowId='TeamcityXunitTestProject.dll']
[xUnit.net 00:00:00.14]   Finished:    TeamcityXunitTestProject
  Passed TeamcityXunitTestProject.UnitTest1.Test1 [1 ms]

VSTest version 17.11.1 (x64)

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.8.2+699d445a1a (64-bit .NET 6.0.35)
[xUnit.net 00:00:00.06]   Discovering: TeamcityXunitTestProject
[xUnit.net 00:00:00.09]   Discovered:  TeamcityXunitTestProject
[xUnit.net 00:00:00.09]   Starting:    TeamcityXunitTestProject
[xUnit.net 00:00:00.10] ##teamcity[flowStarted timestamp='2024-11-07T19:16:00.145+0000' flowId='TeamcityXunitTestProject.dll']
[xUnit.net 00:00:00.10] ##teamcity[testSuiteStarted timestamp='2024-11-07T19:16:00.145+0000' flowId='TeamcityXunitTestProject.dll' name='TeamcityXunitTestProject.dll']
[xUnit.net 00:00:00.10] ##teamcity[flowStarted timestamp='2024-11-07T19:16:00.151+0000' flowId='8e5f00aefe904542bed7f3b4386e3b41' parent='TeamcityXunitTestProject.dll']
[xUnit.net 00:00:00.10] ##teamcity[testSuiteStarted timestamp='2024-11-07T19:16:00.151+0000' flowId='8e5f00aefe904542bed7f3b4386e3b41' name='Test collection for TeamcityXunitTestProject.UnitTest1 (8e5f00aefe904542bed7f3b4386e3b41)']
[xUnit.net 00:00:00.12] ##teamcity[testStarted timestamp='2024-11-07T19:16:00.168+0000' flowId='8e5f00aefe904542bed7f3b4386e3b41' name='TeamcityXunitTestProject.UnitTest1.Test1']
[xUnit.net 00:00:00.13] ##teamcity[testFinished timestamp='2024-11-07T19:16:00.181+0000' flowId='8e5f00aefe904542bed7f3b4386e3b41' name='TeamcityXunitTestProject.UnitTest1.Test1' duration='1']
[xUnit.net 00:00:00.13] ##teamcity[testSuiteFinished timestamp='2024-11-07T19:16:00.182+0000' flowId='8e5f00aefe904542bed7f3b4386e3b41' name='Test collection for TeamcityXunitTestProject.UnitTest1 (8e5f00aefe904542bed7f3b4386e3b41)']
[xUnit.net 00:00:00.13] ##teamcity[flowFinished timestamp='2024-11-07T19:16:00.182+0000' flowId='8e5f00aefe904542bed7f3b4386e3b41']
[xUnit.net 00:00:00.13] ##teamcity[testSuiteFinished timestamp='2024-11-07T19:16:00.184+0000' flowId='TeamcityXunitTestProject.dll' name='TeamcityXunitTestProject.dll']
[xUnit.net 00:00:00.13] ##teamcity[flowFinished timestamp='2024-11-07T19:16:00.184+0000' flowId='TeamcityXunitTestProject.dll']
[xUnit.net 00:00:00.13]   Finished:    TeamcityXunitTestProject
  Passed TeamcityXunitTestProject.UnitTest1.Test1 [1 ms]

@A6blpka
Copy link
Author

A6blpka commented Nov 7, 2024

Collection names are not required to be globally unique, so in order to differentiate test collections from one assembly from test collections with the same names in another assembly, the collection unique ID must be part of the name.

Don't think anything negative, but this seems like overengineering to me. The test collection name consists of namespace and class name, making different solutions with the same namespace is like shooting oneself in the foot.

@bradwilson
Copy link
Member

bradwilson commented Nov 7, 2024

Don't think anything negative, but this seems like overengineering to me. The test collection name consists of namespace and class name, making different solutions with the same namespace is like shooting oneself in the foot.

Collection names are user customizable.

@bradwilson
Copy link
Member

bradwilson commented Nov 7, 2024

There are tests that need to be updated. See the build failure: https://github.com/xunit/xunit/actions/runs/11725900453/job/32676586129?pr=3054

@bradwilson bradwilson removed the By Design The issue is working as designed label Nov 7, 2024
@A6blpka
Copy link
Author

A6blpka commented Nov 8, 2024

There are tests that need to be updated. See the build failure: https://github.com/xunit/xunit/actions/runs/11725900453/job/32676586129?pr=3054

Sorry about that, didn't immediately understand how to build and run testing locally. Anyway, in the local launch I couldn't get a full test pass because of the different locale from the English one. Hopefully now they will pass pepeline :)

@bradwilson bradwilson merged commit 6070070 into xunit:v2 Nov 8, 2024
@bradwilson
Copy link
Member

bradwilson commented Nov 8, 2024

Once the CI build is complete, this will be available in 2.9.3-pre.6. https://xunit.net/docs/using-ci-builds

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@A6blpka @bradwilson

Comments