{"id":19695,"date":"2016-08-05T23:32:53","date_gmt":"2016-08-06T06:32:53","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/visualstudioalm\/?p=19695"},"modified":"2019-02-14T17:33:46","modified_gmt":"2019-02-15T01:33:46","slug":"evolving-the-visual-studio-test-platform-part-2","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/devops\/evolving-the-visual-studio-test-platform-part-2\/","title":{"rendered":"Evolving the Visual Studio Test Platform \u2013 Part 2"},"content":{"rendered":"<p>As promised in the <a href=\"https:\/\/blogs.msdn.microsoft.com\/visualstudioalm\/2016\/07\/25\/evolving-the-visual-studio-test-platform-part-1\/\">previous post<\/a>, what follows is a recap of all of the Testing-related features implemented and delivered in the Visual Studio 2015 cycle until now. Each of the features might be relevant at a different stage in the lifecycle, but together they serve the single purpose of enabling efficiency &#8211; and overlaid on the lifecycle graphic, that constellation emerges.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/6\/2019\/05\/InvestmentsShipped.jpg\"><img decoding=\"async\" width=\"1233\" height=\"625\" class=\"alignnone size-full wp-image-19696\" alt=\"InvestmentsShipped\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2016\/08\/InvestmentsShipped.jpg\" \/><\/a><\/p>\n<p>Very briefly, the features may be described as follows:<\/p>\n<ul>\n<li>Invigorated, &#8220;<a href=\"https:\/\/blogs.msdn.microsoft.com\/visualstudioalm\/2016\/06\/17\/taking-the-mstest-framework-forward-with-mstest-v2\/\">MSTest V2<\/a>&#8221; <strong>(4)<\/strong> comes with long awaited feature-additions, and joins the list of third party test frameworks and adapters available on NuGet <strong>(1)<\/strong>, enabling it to have its own release\/update cadence, and making easy to search for, install, uninstall, and update it as a dependency to the code-under-test.<\/li>\n<li>The <a href=\"https:\/\/blogs.msdn.microsoft.com\/visualstudioalm\/2015\/03\/06\/creating-unit-test-method-stubs-with-create-unit-tests\/\">Create Unit Tests <\/a><strong>(2)<\/strong> wizard enables quick bootstrapping of tests while in the flow of coding by making it easy to configure a test project, and the test class and the test method stub therein.<\/li>\n<li>But we know that developing a suite of unit tests that exhaustively exercise and validate the logic of the code under test is a challenge \u2013 and even more so when the underlying code-under-test is constantly evolving. <a href=\"https:\/\/devblogs.microsoft.com\/visualstudio\/intellitest-for-net-test-more-with-less-effort\/\">IntelliTest<\/a> <strong>(3)<\/strong> helps overcome this daunting challenge, by automatically generating a suite of tests, and automatically keeping them updated.<\/li>\n<li><a href=\"https:\/\/blogs.msdn.microsoft.com\/visualstudioalm\/2016\/02\/08\/parallel-and-context-sensitive-test-execution-with-visual-studio-2015-update-1\/\">Incremental validation<\/a> <strong>(5)<\/strong> enables fast turnaround in the red-green-refactor &#8220;dev inner-loop&#8221;, and does so by selecting tests only from test containers touched by the build &#8211; the Test Explorer window\u2019s \u201cRun Test After Build\u201d control then runs only that selection.<\/li>\n<li>In Visual Studio 2015 Update 3, <em>test execution is faster<\/em><strong> (9)<\/strong> even if you don\u2019t use parallel execution, and <em>memory consumption is lower<\/em><strong> (6)<\/strong> during both discovery and execution of tests. These improvements are pronounced for large solutions containing thousands of tests spread over many test containers.<\/li>\n<li><a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/jj635153.aspx\">Assembly Resolution<\/a> <strong>(8)<\/strong> provides a convenient means to configure the locations from where to load the dependencies for a test during its execution &#8211; especially useful when you want to run tests in a lab environment against an installed product (i.e. when explicitly placing dependent DLLs next to the test DLL might be impractical).<\/li>\n<li><a href=\"https:\/\/blogs.msdn.microsoft.com\/visualstudioalm\/2016\/02\/08\/parallel-and-context-sensitive-test-execution-with-visual-studio-2015-update-1\/\">Parallel Test Execution<\/a> <strong>(10)<\/strong> is supported for all test adapters, using the test container (i.e. the file containing the tests to be executed) as the unit of parallelization, and a process as the unit of isolation. Within each container, the tests will be executed as per the semantics of the corresponding test framework. It can be conveniently enabled directly from the Test Explorer window, but additionally, the level of parallelism can be <a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/jj635153.aspx\">configured<\/a> <strong>(7)<\/strong> through the .runsettings file.<\/li>\n<\/ul>\n<p><strong>Visual Studio &#8220;15&#8221; Preview<\/strong>\nThere has been a lot of <a href=\"https:\/\/visualstudio.uservoice.com\/forums\/330519-team-services\/suggestions\/5126061-test-explorer-is-severely-limited\">feedback on the Test Explorer<\/a>. We have begun addressing this feedback from Visual Studio &#8220;15&#8221; Preview 2 onwards.<\/p>\n<ul>\n<li><a href=\"https:\/\/visualstudio.uservoice.com\/forums\/330519-team-services\/suggestions\/3090310-allow-tests-to-be-grouped-by-class-category-and\">Group by namespace<\/a> <strong>(11)<\/strong> has been added, and completes the complement of groupings available through the Test Explorer.<\/li>\n<li>On a build failure, the <a href=\"https:\/\/visualstudio.uservoice.com\/forums\/330519-team-services\/suggestions\/2754415-allow-to-disable-the-query-run-tests-from-the-las\">option to run tests from a prior build<\/a> <strong>(12)<\/strong> can now be controlled from Tools | Options | Project and Solutions | Build and Run hive.<\/li>\n<li>The <a href=\"https:\/\/visualstudio.uservoice.com\/forums\/330519-team-services\/suggestions\/4106946-add-option-to-show-test-explorer-automatically-whe\">Test Explorer shows up automatically<\/a> <strong>(13)<\/strong> when you run tests making it easy to know what tests are running.<\/li>\n<li>The new h:mm:ss:sss format is used to report test duration precisely <strong>(14)<\/strong>.<\/li>\n<li>The test duration is reported uniformly <strong>(15)<\/strong> on each test, in the Summary pane, in the Output window, and in the test report seen in CI.<\/li>\n<li>Furthermore, when grouped by duration, the tests within each group are sorted <strong>(16)<\/strong> such that the slower running tests do not hide, but instead float to the top.<\/li>\n<\/ul>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/6\/2019\/05\/GroupingAndDuration.jpg\"><img decoding=\"async\" width=\"896\" height=\"579\" class=\"alignnone size-full wp-image-19705\" alt=\"GroupingAndDuration\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2016\/08\/GroupingAndDuration.jpg\" \/><\/a><\/p>\n<p>Perusing through an output log file would abruptly end with the message <em>&#8220;!!LOG TRUCATED!! To get a complete log select \u201cCopy All\u201d and paste into some text editor like Notepad&#8221;<\/em>. Quickly wanting to look up say, an exception message in code was no easier \u2013 since select-and-copy was not supported in the Test Window. That is changed with Visual Studio &#8220;15&#8221;:<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/6\/2019\/05\/CopyFromOutputWindow.jpg\"><img decoding=\"async\" width=\"872\" height=\"579\" class=\"alignnone size-full wp-image-19715\" alt=\"CopyFromOutputWindow\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2016\/08\/CopyFromOutputWindow.jpg\" \/><\/a><\/p>\n<ul>\n<li><a href=\"https:\/\/visualstudio.uservoice.com\/forums\/330519-team-services\/suggestions\/2753655-allow-cut-and-paste-in-unit-test-output-window\">Select-and-copy is supported<\/a> <strong>(17)<\/strong> in the Summary pane and in the Test Output window.<\/li>\n<li>The <a href=\"https:\/\/visualstudio.uservoice.com\/forums\/330519-team-services\/suggestions\/9719772-allow-to-turn-off-the-log-truncated-feature-of-the\">logs are no longer truncated<\/a> <strong>(18)<\/strong>.<\/li>\n<\/ul>\n<p><strong>Summary<\/strong>\nThere, 18 features, with a common purpose: efficiency in getting from conceptualizing the tests, all the way to the point where they are generating reports that can inform business decisions.<\/p>\n<p>Each of these features merit a post of its own. But that will be for another day. In the meantime leave us your feedback. We are listening.<\/p>\n<p><strong>Next post &#8211; next steps for the vstest platform<\/strong>\nWhere is the test platform headed? What are the next steps in that direction?<\/p>\n<p>Stay tuned.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>As promised in the previous post, what follows is a recap of all of the Testing-related features implemented and delivered in the Visual Studio 2015 cycle until now. Each of the features might be relevant at a different stage in the lifecycle, but together they serve the single purpose of enabling efficiency &#8211; and overlaid [&hellip;]<\/p>\n","protected":false},"author":765,"featured_media":45953,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[226,1,252],"tags":[],"class_list":["post-19695","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ci","category-devops","category-testing"],"acf":[],"blog_post_summary":"<p>As promised in the previous post, what follows is a recap of all of the Testing-related features implemented and delivered in the Visual Studio 2015 cycle until now. Each of the features might be relevant at a different stage in the lifecycle, but together they serve the single purpose of enabling efficiency &#8211; and overlaid [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/19695","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/users\/765"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/comments?post=19695"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/19695\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media\/45953"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media?parent=19695"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/categories?post=19695"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/tags?post=19695"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}