<<\/p>\n
p>We (the RM team) used that as an opportunity to re-look at our RM usage, and to fix things up so that it becomes easier to use. Along the way, we fixed up some things in the product, and some things in the way we use the product. In this two-part blog, I will walk you through what we did. The first part covers the issues that we faced, and the fixes that have been rolled out. I will blog the second part when the remaining fixes have been rolled out (since they are still in-flight).<\/p>\n
The various things that didn\u2019t work very well in RM<\/h2>\n
<<\/p>\n
p>As we walk through the issues, we will use the release \u201cTFS \u2013 Prod Update 538\u201d as an example of what didn\u2019t work very well in RM:<\/p>\n
<<\/p>\n
p> <<\/p>\n p>Double-clicking on the release showed a pretty un-actionable set of Issues. We used to see this \u201cwrapper script\u201d text for all errors, and it wasn\u2019t very useful:<\/p>\n <<\/p>\n p> <<\/p>\n p>Further, the Release Summary indicated that the deployments to Ring 0, Ring 1, Ring 2, and Ring 3 succeeded. However, clicking on the Environments tab and looking closely at the number of tasks enabled in these rings told a different story.<\/p>\n <<\/p>\n p> <<\/p>\n p>There were, in fact, zero tasks enabled in Rings 0, 1, 2 and 3! The bits needed to be deployed only to Ring 4, but the release was taken through Rings 0, 1, 2 and 3 without doing anything meaningful in these environments. This used to mess up the traceability of the product, because RM thought that the current release on Ring 0 (and Rings 1, 2 and 3 also) was \u201cTFS \u2013 Prod Update 538\u201d (at least till the next release rolled out), whereas actually the bits on Ring 0 corresponded to an older release. The \u201ccurrent release in an environment\u201d concept is pretty important for RM: The product surfaces this in some of its views, and also uses this for some internal operations like release retention e.g. RM won\u2019t delete a release if it is the current release on an environment.<\/p>\n <<\/p>\n p>The above design \u2013 of dragging the release through Rings 0, 1, 2 and 3 unnecessarily \u2013 begs the question, \u201cWhy? Why couldn\u2019t the release be created so that zero environments started automatically \u2018after release creation\u2019, and then Ring 4 was started manually?\u201d The reason for this was the requirement that there should be only 1 approval for the entire release<\/strong>, across all the environments. When the release was created, the approver wanted to check that everything was in order across all the environments, and then didn\u2019t want to be asked to approve again. Therefore, this requirement was modeled as an approval on Ring 0, and every release had to go through this environment before it reached its real destination.<\/p>\n <<\/p>\n p> <<\/p>\n p>Moving on \u2026 going to the Logs tab showed that the log file corresponding to the failed task wasn\u2019t even available in the browser. This used to sometimes happen when the log file was very large. The workaround was that we used to have to log into the agent box and view the logs on the agent. <\/p>\n <<\/p>\n p> <<\/p>\n p>Once we got to the logs, we found out that the logs were not easy to understand. Reason: Each environment corresponded to multiple Scale Units (SUs) e.g. Ring 3 corresponds to three scale units (WEU2, SU6 and WUS22). So the logs for the three scale units were interspersed.<\/p>\n <<\/p>\n p>Finally, as part of aligning to Azure\u2019s Safe Deployment practices, there was a requirement to have each Ring \u201cbake\u201d for some time before proceeding to the next ring, so that issues were flushed out in the inner rings before moving to the outer (and more public) rings. The bake time was modeled as a \u201csleep\u201d task. This was less-than-ideal because it used up an agent unnecessarily while sleeping. <\/p>\n <<\/p>\n p> <<\/p>\n p>In the screenshot above, the Sleep task was disabled for this particular hotfix deployment, probably to get some fix out into prod early, but typically this task is enabled.<\/p>\n We fixed the following issues either by enhancing RM or by changing the way we used the product.<\/p>\n Problem statement<\/strong>: The Issues list in the release was un-actionable.<\/p>\n Solution<\/strong>: [Change in the way we use RM<\/font>] Changed the Powershell script so that it logged the inner exception as the Error, as opposed to the outer exception<\/p>\n <<\/p>\n p>Problem statement<\/strong>: Incorrect traceability in the product caused by taking each release through Ring 0 even if the bits were meant for a different Ring<\/p>\n <<\/p>\n p>Solution<\/strong>: [Enhanced RM<\/strong><\/font>] We enabled correct traceability in the product by adding support for a new feature \u201cRelease-level-approvals\u201d, and then used this in the \u201cTFS \u2013 Prod Update\u201d Release Definition. This feature ensures that approval is required only once in the release \u2013 no matter which Ring is deployed first \u2013 as long as the approvers for all the Rings are the same:<\/p>\n <<\/p>\n p> <<\/p>\n p>As you can see above, all Rings, except for the first Ring, have the following option selected: \u201cAutomatically approve auto-triggered deployments to this environment for users who have approved deployment to the previous environment\u201d. (That\u2019s quite a mouthful!) Ring 0 doesn\u2019t need to have this option selected since there is no \u201cprevious\u201d environment.<\/p>\n <<\/p>\n p>This ensures clean traceability of the release i.e. the bits are not unnecessarily dragged through rings where they are not meant to be deployed.<\/p>\n <<\/p>\n p> The issues in this section have not been completely addressed. Once we address them, I will write up the details in part 2 of this blog.<\/p>\n <<\/p>\n p>Problem statement<\/strong>: The log file was sometimes not available in the browser \u2013 typically when it was very large.<\/p>\n <<\/p>\n p>Solution<\/strong>: [TBD: Change<\/font> in the way we use RM<\/font>] We will fix the log file upload reliability issue by moving from the 1.x agent to the 2.x agent. There is a known reliability issue with the 1.x agent with respect to uploading large log files.<\/p>\n <<\/p>\n![\"image\"](\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/image_thumb720.png\" "\\"image\\"")
<\/a><\/p>\n![\"image\"](\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/image_thumb721.png\" "\\"image\\"")
<\/a><\/p>\n![\"image\"](\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/image_thumb722.png\" "\\"image\\"")
<\/a><\/p>\n![\"image\"](\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/image_thumb723.png\" "\\"image\\"")
<\/a><\/p>\n![\"image\"](\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/image_thumb724.png\" "\\"image\\"")
<\/a><\/p>\n![\"image\"](\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/image_thumb725.png\" "\\"image\\"")
<\/a> <\/p>\nFixes that we have rolled out<\/h2>\n
![\"image\"](\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/image_thumb726.png\" "\\"image\\"")
<\/a><\/p>\n![\"image\"](\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/image_thumb727.png\" "\\"image\\"")
<\/a><\/p>\n![\"image\"](\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/image_thumb728.png\" "\\"image\\"")
<\/a><\/p>\nFixes still to be rolled out<\/h2>\n
![\"image\"](\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/image_thumb729.png\" "\\"image\\"")
<\/a><\/p>\n![\"image\"](\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/image_thumb730.png\" "\\"image\\"")
<\/a><\/li>\n<\/ol>\n![\"image\"](\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/image_thumb731.png\" "\\"image\\"")
<\/a><\/p>\n