{"id":30605,"date":"2017-04-24T09:42:46","date_gmt":"2017-04-24T17:42:46","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/visualstudioalm\/?p=30605"},"modified":"2019-02-14T15:51:52","modified_gmt":"2019-02-14T23:51:52","slug":"configuring-your-release-pipelines-for-safe-deployments","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/devops\/configuring-your-release-pipelines-for-safe-deployments\/","title":{"rendered":"Configuring your release pipelines for safe deployments"},"content":{"rendered":"

For large and high scale applications, the promise of \u201centerprise grade\u201d availability and high reliability levels are key to customer confidence on the applications. Continuous delivery pipelines for such scaled out applications typically consist of multiple environments.<\/p>\n

DevOPS enables faster & automated delivery of changes, thereby helping customers with the most advanced set of features. In theory, any change to a production system has risks. Safe deployment guidelines help in managing this risk for large scaled out applications, thereby fulfilling the customer promise.<\/p>\n

In this blog post, we shall share the safe deployment guidelines that we follow in Microsoft and how do we configure the pipelines or release definitions<\/a> using Visual Studio Team Services<\/a> to enforce the guidelines.<\/p>\n

Gradual rollout to multiple environments<\/h2>\n

For applications under discussion, the production environment would comprise of multiple scale units (one per region).<\/p>\n

You may want to deploy the changes a test or staging environment before deploying to any of the production environments (as a final quality validation) and a canary environment that interacts with production environments for the dependent services serving some synthetic load.<\/p>\n

Also, it is recommended to not deploy to all production environments in one go, exposing all the customers to the changes. A gradual rollout that exposes the changes to customers over a period, thereby implicitly validating the changes in production with a smaller set of customers at a time.<\/p>\n

In effect, the deployment pipeline would look like the following:\n\"order<\/a>\nAs an example, for an application is deployed in 12 regions with US regions (4) having a high load, European regions (4) having a medium load and Asian regions (4) having a relatively lighter load, following would be the order of rollout.<\/p>\n\n\n\n\n\n\n\n\n\n
\u00a0 Regions<\/strong><\/td>\n\u00a0 Description<\/strong><\/td>\n<\/tr>\n
Test<\/td>\nRun final functional validation on the application<\/td>\n<\/tr>\n
Canary<\/td>\nProcess synthetic load on the application, interacting with production instances of dependent services<\/td>\n<\/tr>\n
Pilot customers<\/td>\nPilot customers (Internal and early adopter customers) are onboarded to a separate scale unit. Deploy after deployment to Canary succeeds.<\/td>\n<\/tr>\n
Asian regions 1, 2, 3 & 4<\/td>\nAsian regions have a lighter load. Deploy to all regions in parallel after deployment to Pilot succeeds.<\/td>\n<\/tr>\n
European regions 1, 2, 3 & 4<\/td>\nEuropean regions have medium load. Deploy to all regions in parallel after deployment to all Asian regions succeed.<\/td>\n<\/tr>\n
\u00a0US regions 1, 2, 3 & 4<\/td>\nUS regions have high load. Deploy to all regions in parallel after deployment to all European regions succeed.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

In a release definition, we use environment triggers<\/a> to configure the environment deployment conditions as follows.<\/p>\n

\"environment<\/a><\/p>\n

If required, you can configure to deploy to the four scale units in each region sequentially for additional level of control.<\/p>\n

Use of feature flags<\/span><\/h3>\n

There are circumstances in which we are not fully confident about a feature and are looking to have additional exposure control for the feature. This means that within a SU, we would like some customers to get to try the feature out and give us feedback before it is made broadly available to everybody.<\/span><\/span><\/p>\n

In such cases, we use \u201cFeature flags\u201d. Essentially these are controllable knobs that wrap the new features. When turned off, the applications skip executing the new code and hence do not impact the functionality of the application. Then turned on, the new code executes making the feature available for users. <\/span>Launch Darkly<\/span><\/a> provides a <\/span>feature flag management platform<\/span><\/a> that enables teams to have an even more gradual rollout of new features, and can easily be integrated with VSTS RM.<\/span><\/span><\/p>\n

Uniform deployment workflow for all environments<\/h2>\n

As discussed above, we are deploying the application to each scale unit independently. A deployment and validation process is defined for each of the scale units. As a best practice, you should follow the same procedure to deploy those bits\u00a0in every environment. The only thing that should change from one environment to the next is the configuration you want to apply and the validation steps to execute.<\/p>\n

To enforce the deployment procedures to be same across environments, we define a task group<\/a> for the deployment procedure and include the same in each of the environments. The different configurations are parameterized and the values are managed using environment variables<\/a> in the release definition.<\/p>\n

The deployment workflow for each of the environments in our release definitions looks like the following.<\/p>\n

\"Use<\/a><\/p>\n

<\/div>\n

Manual approval for rollouts<\/h2>\n

There are various reasons due to which you may not want the application to be updated at some points of time. It could be due to an upcoming major event for which you want to avoid all risks, or a known issue with a dependency that needs changes to your application to be deferred.<\/p>\n

Configuring manual approvals before the pipeline begins ensures that we get two pairs of eyes ensure the application is not going through these special circumstances and can be updated.<\/p>\n

Moreover, there might be urgent hotfixes or special changes that do not apply to all the scale units. For such changes, we need to bypass the pipeline and directly deploy to specific environments only. We still would like to get approvals in such scenarios. However, in case of a pipeline flow we do not want to get multiple approvals, one for each of the environments.<\/p>\n

So, in a nut shell, we are looking for one approval at the start of the deployment sequence. The sequence may or may not start from the Test environment.<\/p>\n

We configure approvals<\/a> for each of the environments in our release definitions to fulfil these requirements. The approvals for the production environments are configured like the following.\n\"approvals<\/a><\/p>\n

Segregation of roles<\/h2>\n

As we discussed above, we would like to have two people analyze every deployment and ensure that all\u2019s well for deploying to the environment.<\/p>\n

It is possible that the approver mentioned in the release definition is same as the person requesting the deployment. In such a scenario, the requirement of segregating the roles of deployment submitter and approver between two different users does not get fulfilled, thereby risking a wrong deployment due to manual oversight.\nRelease management provides an option to deny the submitter from approving a deployment, thereby helping enforce segregation of roles.<\/p>\n

\"segregationofroles\"<\/a><\/p>\n

Health check during roll out<\/h2>\n

With the above environment and approval settings, validation phase for the environments play a key role in ensuring the environments are healthy after the deployments. It may always not be possible to fully automate the validation and health monitoring of environments.<\/p>\n

In such circumstances, adopting a \u201cWait and auto-promote\u201d criteria for the production environments are recommended. The pipeline is paused for a certain duration of time, during which team members monitor various health indicators of the service and can abort the pipeline if it is not appropriate to continue. Users can manually re-start the pipeline from the next environment once the issue is analyzed.<\/p>\n

Including a manual intervention<\/a> task in the validation workflow for production environments helps us configure the release definitions to be in \u201cwait and auto-promote\u201d mode with a 24 hours wait between environments.\n\"manualintervention\"<\/a><\/p>\n

Branch filters for deployments<\/h2>\n

With extensive use of Git as the version control system for development, developers commit changes in various branches. All the changes come together in master or release branches. To ensure completeness of features being deployed, it is recommended to restrict deployments of artifacts generated from these branches to the production environments.<\/p>\n

In order to achieve this, you can configure release triggers<\/a> to create the release only when the build is produced by compiling code from certain branches<\/strong> or when the build has certain tags<\/strong>.\nVSTS RM would soon have an\u00a0enhancement\u00a0to environment triggers<\/a>\u00a0to restrict\u00a0deployments to an environment to start automatically when the build is from certain branches or when the build has certain tags. With this, you’ll be able to take every build to the Test environment but have builds from master or release branches only deployed on the production environments.<\/p>\n

Secure the pipelines<\/h2>\n

We have now configured our pipelines that ensure we safely deploy the changes to all the environments. Team members can now create releases and start the deployments. To avoid any issues, we need to ensure that the configurations are not disturbed and the checks out in place are not by passed by users.<\/p>\n

We configure Release permissions<\/a> on the release definitions to avoid any unwanted changes. Specifically, we control the users who are allowed \u201cAdminister release permissions\u201d, \u201cDelete release definition\u201d, \u201cDelete release environment\u201d, \u201cDelete releases\u201d, \u201cEdit release definition\u201d, \u201cEdit release environment\u201d, \u201cManage release approvers\u201d and \u201cManage releases\u201d permissions.<\/p>\n","protected":false},"excerpt":{"rendered":"

For large and high scale applications, the promise of \u201centerprise grade\u201d availability and high reliability levels are key to customer confidence on the applications. Continuous delivery pipelines for such scaled out applications typically consist of multiple environments. DevOPS enables faster & automated delivery of changes, thereby helping customers with the most advanced set of features. […]<\/p>\n","protected":false},"author":200,"featured_media":45953,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[226,1],"tags":[],"class_list":["post-30605","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ci","category-devops"],"acf":[],"blog_post_summary":"

For large and high scale applications, the promise of \u201centerprise grade\u201d availability and high reliability levels are key to customer confidence on the applications. Continuous delivery pipelines for such scaled out applications typically consist of multiple environments. DevOPS enables faster & automated delivery of changes, thereby helping customers with the most advanced set of features. […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/30605","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/users\/200"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/comments?post=30605"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/30605\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media\/45953"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media?parent=30605"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/categories?post=30605"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/tags?post=30605"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}