{"@attributes":{"version":"2.0"},"channel":{"title":"Lalit Adithya - Coder, Blogger, Architect, and a Photographer","description":{},"link":"https:\/\/lalitadithya.com\/","pubDate":"Sat, 20 Feb 2021 14:52:14 +0000","lastBuildDate":"Sat, 20 Feb 2021 14:52:14 +0000","generator":"Jekyll v4.0.1","item":[{"title":"OTA updates for .NET Applications","description":"<p>Let us see how we can build a .NET core console application that can upgrade itself using the Cincinnati update protocol. The Cincinnati protocol will serve OTA updates to the client. Cincinnati is an update protocol designed by Openshift as a successor to Google\u2019s Omaha update protocol. It describes a method for representing transitions between releases and allowing a client to perform automatic updates between these releases.<\/p>\n\n<!--more-->\n\n<p>Diagrammatically the solution is<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/auto-update-dot-net-applications\/auto-update.webp\" data-fancybox=\"\" data-title=\"Architecture Diagram\">\n    <img src=\"\/images\/auto-update-dot-net-applications\/auto-update.webp\" alt=\"Architecture Diagram\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Architecture Diagram<\/figcaption>\n  \n<\/figure>\n\n<p>The client application checks for updates by reaching out to a central update server (Cincinnati Server shown in green above). Cincinnati pulls the metadata stored in a docker registry, builds, and returns a DAG (directed acyclic graph) repesenting the upgrade paths along with metadata. The client application (shown in orange) uses this graph to determine if an upgrade is available. Using the upgrade metadata, the client downloads and launches the installer to start the upgrade process.<\/p>\n\n<h1 id=\"what-constitutes-a-release\">What constitutes a release?<\/h1>\n\n<p>A release consists of two parts:<\/p>\n<ol>\n  <li><em>Installer<\/em> - The installer should be able to upgrade the application with minimal user input.<\/li>\n  <li><em>Release metadata<\/em> - Release metadata is a JSON document that contains:\n    <ul>\n      <li>Channel of the release, i.e., stable, beta, alpha, fast ring etc<\/li>\n      <li>Platform<\/li>\n      <li>Weblink to the installer<\/li>\n      <li>The version of the installer<\/li>\n      <li>Upgrade paths<\/li>\n      <li>Any other metadata needed for the upgrade process such as \u201chave the T&amp;C changed requiring the user consent\u201d, \u201ctype of release\u201d and so on<\/li>\n    <\/ul>\n  <\/li>\n<\/ol>\n\n<p>An example release metadata for version 1.1.0 of an application will be:<\/p>\n<div class=\"language-json highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"p\">{<\/span><span class=\"w\">\n\t<\/span><span class=\"nl\">\"kind\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"cincinnati-metadata-v0\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n\t<\/span><span class=\"nl\">\"version\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"1.1.0\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n\t<\/span><span class=\"nl\">\"previous\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s2\">\"1.0.0\"<\/span><span class=\"p\">],<\/span><span class=\"w\">\n\t<\/span><span class=\"nl\">\"metadata\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"graph.release.channels\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"stable\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"graph.release.arch\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"amd64\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n\t\t<\/span><span class=\"nl\">\"kind\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"release\"<\/span><span class=\"w\">\n\t<\/span><span class=\"p\">}<\/span><span class=\"w\">\n<\/span><span class=\"p\">}<\/span><span class=\"w\">\n<\/span><\/code><\/pre><\/div><\/div>\n<p>The \u201cprevious\u201d field in the JSON document will determine the versions of the applications that can upgrade to version 1.1.0. The upgrade paths can be controlled using the previous field or the next field as explained <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/github.com\/openshift\/cincinnati\/blob\/master\/docs\/design\/cincinnati.md\" target=\"_blank\">here<\/a>.<\/p>\n\n<p>\u201cgraph.release.channels\u201d is used to indicate the channel (alpha, beta, stable etc) for this release. \u201cgraph.release.arch\u201d is used to indicate the architecture for this release, i.e., amd64, arm, x86 etc. Cincinnati uses these two fields to filter the graph, i.e., a client in the stable channel with amd64 architecture will not receive the upgrade paths for other channels and architectures. You can add more metadata inside the \u201cmetadata\u201d object.<\/p>\n\n<p>For the sake of simplicity, we will be making use of a single channel. Let us now look at the various components and how we can build\/deploy each.<\/p>\n\n<h1 id=\"part-1--cincinnati-server\">Part 1 \u2013 Cincinnati Server<\/h1>\n\n<p>You can find the source code for the Cincinnati server <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/github.com\/lalitadithya\/cincinnati\/tree\/dev\" target=\"_blank\">here<\/a> and the docker image <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/hub.docker.com\/r\/lalitadithya\/cincinnati\" target=\"_blank\">here<\/a>. This fork of Cincinnati removes OpenShift specific code.<\/p>\n\n<p>To run the Cincinnati server, we need a docker repository to store the release metadata. Create a blank repository in the docker registry of your choice and then open a terminal and run the following command to start the Cincinnati server on port 9000.<\/p>\n<div class=\"highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>docker run -p 9000:9000 -d -e PORT=9000 -e REGISTRY=&lt;&lt;registory-url&gt;&gt; -e REPOSITORY=&lt;&lt;username&gt;&gt;\/&lt;&lt;repository-name&gt;&gt; lalitadithya\/Cincinnati\n<\/code><\/pre><\/div><\/div>\n\n<h1 id=\"part-2--the-net-core-application\">Part 2 \u2013 The .NET Core Application<\/h1>\n\n<p>The application should upgrade itself after the user has closed the application, similar to Slack and Visual Studio Code. In .NET, we can register an event handler for <code class=\"highlighter-rouge\">AppDomain.CurrentDomain.ProcessExit<\/code> event, and then we can use the <code class=\"highlighter-rouge\">Process<\/code> library to start the installer as shown in the snippet below:<\/p>\n<div class=\"language-csharp highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"k\">static<\/span> <span class=\"kt\">bool<\/span> <span class=\"n\">isUpdateAvailable<\/span> <span class=\"p\">=<\/span> <span class=\"k\">false<\/span><span class=\"p\">;<\/span>\n\n<span class=\"k\">static<\/span> <span class=\"k\">void<\/span> <span class=\"nf\">Main<\/span><span class=\"p\">(<\/span><span class=\"kt\">string<\/span><span class=\"p\">[]<\/span> <span class=\"n\">args<\/span><span class=\"p\">)<\/span>\n<span class=\"p\">{<\/span>\n    <span class=\"n\">AppDomain<\/span><span class=\"p\">.<\/span><span class=\"n\">CurrentDomain<\/span><span class=\"p\">.<\/span><span class=\"n\">ProcessExit<\/span> <span class=\"p\">+=<\/span> <span class=\"n\">CurrentDomain_ProcessExit<\/span><span class=\"p\">;<\/span>\n    <span class=\"c1\">\/\/ process user request and check for updates in the background and download it if it is available<\/span>\n    <span class=\"n\">isUpdateAvailable<\/span> <span class=\"p\">=<\/span> <span class=\"k\">true<\/span><span class=\"p\">;<\/span>\n<span class=\"p\">}<\/span>\n\n<span class=\"k\">private<\/span> <span class=\"k\">static<\/span> <span class=\"k\">void<\/span> <span class=\"nf\">CurrentDomain_ProcessExit<\/span><span class=\"p\">(<\/span><span class=\"kt\">object<\/span> <span class=\"n\">sender<\/span><span class=\"p\">,<\/span> <span class=\"n\">EventArgs<\/span> <span class=\"n\">e<\/span><span class=\"p\">)<\/span>\n<span class=\"p\">{<\/span>\n    <span class=\"k\">if<\/span><span class=\"p\">(<\/span><span class=\"n\">isUpdateAvailable<\/span><span class=\"p\">)<\/span>\n    <span class=\"p\">{<\/span>\n        <span class=\"n\">Process<\/span><span class=\"p\">.<\/span><span class=\"nf\">Start<\/span><span class=\"p\">(<\/span><span class=\"k\">new<\/span> <span class=\"nf\">ProcessStartInfo<\/span><span class=\"p\">(<\/span><span class=\"n\">installerLocation<\/span><span class=\"p\">)<\/span>\n        <span class=\"p\">{<\/span>\n            <span class=\"n\">UseShellExecute<\/span> <span class=\"p\">=<\/span> <span class=\"k\">true<\/span>\n        <span class=\"p\">});<\/span>\n    <span class=\"p\">}<\/span>\n<span class=\"p\">}<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>To interact with the Cincinnati server, we can make use of a NuGet package called \u201ccincinnati-client-net\u201d (<a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/github.com\/lalitadithya\/cincinnati-client-net\" target=\"_blank\">GitHub<\/a> and <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/www.nuget.org\/packages\/CincinnatiClient\/\" target=\"_blank\">nuget<\/a>). This library queries Cincinnati for upgrade paths, and it returns a list of the possible upgrades along with the metadata as a dictionary. The metadata is fetched from the release metadata that we will create in the next step.<\/p>\n<div class=\"language-csharp highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"k\">static<\/span> <span class=\"kt\">string<\/span> <span class=\"n\">updateServer<\/span> <span class=\"p\">=<\/span> <span class=\"s\">\"localhost:9000\"<\/span><span class=\"p\">;<\/span>\n\n<span class=\"k\">static<\/span> <span class=\"k\">void<\/span> <span class=\"nf\">Main<\/span><span class=\"p\">(<\/span><span class=\"kt\">string<\/span><span class=\"p\">[]<\/span> <span class=\"n\">args<\/span><span class=\"p\">)<\/span>\n<span class=\"p\">{<\/span>\n    <span class=\"n\">Version<\/span> <span class=\"n\">version<\/span> <span class=\"p\">=<\/span> <span class=\"n\">Assembly<\/span><span class=\"p\">.<\/span><span class=\"nf\">GetExecutingAssembly<\/span><span class=\"p\">().<\/span><span class=\"nf\">GetName<\/span><span class=\"p\">().<\/span><span class=\"n\">Version<\/span><span class=\"p\">;<\/span>\n    <span class=\"kt\">string<\/span> <span class=\"n\">currentVersion<\/span> <span class=\"p\">=<\/span> <span class=\"s\">$\"<\/span><span class=\"p\">{<\/span><span class=\"n\">version<\/span><span class=\"p\">.<\/span><span class=\"n\">Major<\/span><span class=\"p\">}<\/span><span class=\"s\">.<\/span><span class=\"p\">{<\/span><span class=\"n\">version<\/span><span class=\"p\">.<\/span><span class=\"n\">Minor<\/span><span class=\"p\">}<\/span><span class=\"s\">.<\/span><span class=\"p\">{<\/span><span class=\"n\">version<\/span><span class=\"p\">.<\/span><span class=\"n\">Revision<\/span><span class=\"p\">}<\/span><span class=\"s\">\"<\/span><span class=\"p\">;<\/span>\n\n    <span class=\"c1\">\/\/ check for updates<\/span>\n    <span class=\"n\">CincinnatiClient<\/span> <span class=\"n\">cincinnatiClient<\/span> <span class=\"p\">=<\/span> <span class=\"n\">CincinnatiClientBuilder<\/span><span class=\"p\">.<\/span><span class=\"nf\">GetBuilder<\/span><span class=\"p\">()<\/span>\n                                            <span class=\"p\">.<\/span><span class=\"nf\">WithServerUrl<\/span><span class=\"p\">(<\/span><span class=\"n\">updateServer<\/span><span class=\"p\">)<\/span>\n                                            <span class=\"p\">.<\/span><span class=\"nf\">WithReleaseChannel<\/span><span class=\"p\">(<\/span><span class=\"s\">\"stable\"<\/span><span class=\"p\">)<\/span>\n                                            <span class=\"p\">.<\/span><span class=\"nf\">Build<\/span><span class=\"p\">(<\/span><span class=\"k\">new<\/span> <span class=\"nf\">HttpClient<\/span><span class=\"p\">());<\/span>\n    <span class=\"kt\">var<\/span> <span class=\"n\">availableUpgrades<\/span> <span class=\"p\">=<\/span> <span class=\"n\">cincinnatiClient<\/span><span class=\"p\">.<\/span><span class=\"nf\">GetNextApplicationVersions<\/span><span class=\"p\">(<\/span><span class=\"n\">currentVersion<\/span><span class=\"p\">).<\/span><span class=\"nf\">GetAwaiter<\/span><span class=\"p\">().<\/span><span class=\"nf\">GetResult<\/span><span class=\"p\">();<\/span>\n    <span class=\"k\">if<\/span><span class=\"p\">(<\/span><span class=\"n\">availableUpgrades<\/span><span class=\"p\">.<\/span><span class=\"n\">Count<\/span> <span class=\"p\">==<\/span> <span class=\"m\">1<\/span><span class=\"p\">)<\/span>\n    <span class=\"p\">{<\/span>\n        <span class=\"kt\">string<\/span> <span class=\"n\">newVersion<\/span> <span class=\"p\">=<\/span> <span class=\"n\">availableUpgrades<\/span><span class=\"p\">[<\/span><span class=\"m\">0<\/span><span class=\"p\">].<\/span><span class=\"n\">Version<\/span><span class=\"p\">;<\/span>\n        <span class=\"kt\">string<\/span> <span class=\"n\">installerWebLocation<\/span> <span class=\"p\">=<\/span> <span class=\"n\">availableUpgrades<\/span><span class=\"p\">[<\/span><span class=\"m\">0<\/span><span class=\"p\">].<\/span><span class=\"n\">Metadata<\/span><span class=\"p\">[<\/span><span class=\"s\">\"installer\"<\/span><span class=\"p\">];<\/span>\n        <span class=\"n\">Console<\/span><span class=\"p\">.<\/span><span class=\"nf\">WriteLine<\/span><span class=\"p\">(<\/span><span class=\"s\">\"Downloading update for version \"<\/span> <span class=\"p\">+<\/span> <span class=\"n\">newVersion<\/span><span class=\"p\">);<\/span>\n        <span class=\"n\">installerLocation<\/span> <span class=\"p\">=<\/span> <span class=\"n\">Path<\/span><span class=\"p\">.<\/span><span class=\"nf\">Join<\/span><span class=\"p\">(<\/span><span class=\"n\">Path<\/span><span class=\"p\">.<\/span><span class=\"nf\">GetTempPath<\/span><span class=\"p\">(),<\/span> <span class=\"s\">$\"AutoUpdateSample.<\/span><span class=\"p\">{<\/span><span class=\"n\">newVersion<\/span><span class=\"p\">}<\/span><span class=\"s\">.msi\"<\/span><span class=\"p\">);<\/span>\n        <span class=\"k\">using<\/span> <span class=\"nn\">var<\/span> <span class=\"n\">client<\/span> <span class=\"p\">=<\/span> <span class=\"k\">new<\/span> <span class=\"nf\">WebClient<\/span><span class=\"p\">();<\/span>\n        <span class=\"n\">client<\/span><span class=\"p\">.<\/span><span class=\"nf\">DownloadFile<\/span><span class=\"p\">(<\/span><span class=\"n\">installerWebLocation<\/span><span class=\"p\">,<\/span> <span class=\"n\">installerLocation<\/span><span class=\"p\">);<\/span>\n    <span class=\"p\">}<\/span>\n<span class=\"p\">}<\/span>\n<\/code><\/pre><\/div><\/div>\n<p>This snippet uses the library to fetch the possible upgrades and then downloads the installer for the upgrade. The download location for the installer is fetched from the metadata.<\/p>\n\n<p>I have created two versions of a sample application. You can find the source code and the installers below:<\/p>\n<ol>\n  <li><strong>Version 1.0.0<\/strong>\n    <ul>\n      <li><em>Source<\/em>: <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/github.com\/lalitadithya\/auto-update-sample\/tree\/1.0.0\/AutoUpdateSample\" target=\"_blank\">GitHub<\/a><\/li>\n      <li><em>Installer<\/em>: <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/github.com\/lalitadithya\/auto-update-sample\/releases\/tag\/1.0.0\" target=\"_blank\">GitHub Releases<\/a><\/li>\n    <\/ul>\n  <\/li>\n  <li><strong>Version 1.1.0<\/strong>\n    <ul>\n      <li><em>Source<\/em>: <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/github.com\/lalitadithya\/auto-update-sample\/tree\/1.1.0\/AutoUpdateSample\" target=\"_blank\">GitHub<\/a><\/li>\n      <li><em>Installer<\/em>: <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/github.com\/lalitadithya\/auto-update-sample\/releases\/tag\/1.1.0\" target=\"_blank\">GitHub Releases<\/a><\/li>\n    <\/ul>\n  <\/li>\n<\/ol>\n\n<h1 id=\"part-3--the-release-metadata\">Part 3 \u2013 The Release Metadata<\/h1>\n\n<p>Now that we have two versions of the application (along with an installer) and the Cincinnati server running locally, we can create the release metadata and push it to the docker repository that we created in part 1.<\/p>\n\n<p>Create a JSON file called \u201c1.0.0.json\u201d with the following contents. This is the release metadata for version 1.0.0. Notice that the download link for the installer is present in the \u201cmetadata\u201d object. This value is used by the application to download the installer<\/p>\n<div class=\"language-json highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"p\">{<\/span><span class=\"w\">\n  <\/span><span class=\"nl\">\"kind\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"cincinnati-metadata-v0\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n  <\/span><span class=\"nl\">\"version\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"1.0.0\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n  <\/span><span class=\"nl\">\"previous\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[],<\/span><span class=\"w\">\n  <\/span><span class=\"nl\">\"metadata\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"graph.release.channels\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"stable\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"installer\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"https:\/\/github.com\/lalitadithya\/auto-update-sample\/releases\/download\/1.0.0\/AutoUpdateSample.1.0.0.msi\"<\/span><span class=\"w\">\n  <\/span><span class=\"p\">}<\/span><span class=\"w\">\n<\/span><span class=\"p\">}<\/span><span class=\"w\">\n<\/span><\/code><\/pre><\/div><\/div>\n\n<p>Package this release metadata file into a docker container using the docker file shown below:<\/p>\n<div class=\"language-dockerfile highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"k\">FROM<\/span><span class=\"s\"> scratch<\/span>\n<span class=\"k\">COPY<\/span><span class=\"s\"> 1.0.0.json release-manifests\/release-metadata<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>Build and push the release metadata using the following commands:<\/p>\n<div class=\"highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>docker build -t &lt;&lt;username&gt;&gt;\/&lt;&lt;repository-name&gt;&gt;:1.0.0 .\ndocker push &lt;&lt;username&gt;&gt;\/&lt;&lt;repository-name&gt;&gt;:1.0.0\n<\/code><\/pre><\/div><\/div>\n\n<p>Repeat this process for version 1.1.0. Make sure to use tag 1.1.0 while building the docker container and not tag 1.0.0. The metadata for version 1.1.0 will be<\/p>\n<div class=\"language-json highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"p\">{<\/span><span class=\"w\">\n  <\/span><span class=\"nl\">\"kind\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"cincinnati-metadata-v0\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n  <\/span><span class=\"nl\">\"version\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"1.1.0\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n  <\/span><span class=\"nl\">\"previous\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"1.0.0\"<\/span><span class=\"w\"> <\/span><span class=\"p\">],<\/span><span class=\"w\">\n  <\/span><span class=\"nl\">\"metadata\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"graph.release.channels\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"stable\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"installer\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"https:\/\/github.com\/lalitadithya\/auto-update-sample\/releases\/download\/1.1.0\/AutoUpdateSample.1.1.0.msi\"<\/span><span class=\"w\">\n  <\/span><span class=\"p\">}<\/span><span class=\"w\">\n<\/span><span class=\"p\">}<\/span><span class=\"w\">\n<\/span><\/code><\/pre><\/div><\/div>\n\n<h1 id=\"part-4--final-result\">Part 4 \u2013 Final Result<\/h1>\n\n<p>Restart the Cincinnati server. Install and run version 1.0.0 of the application using the installer, and you should see the application auto-upgrade to 1.1.0.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/auto-update-dot-net-applications\/demo.gif\" data-fancybox=\"\" data-title=\"Demo\">\n    <img src=\"\/images\/auto-update-dot-net-applications\/demo.gif\" alt=\"Demo\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Demo<\/figcaption>\n  \n<\/figure>\n","pubDate":"Sat, 20 Feb 2021 00:00:00 +0000","link":"https:\/\/lalitadithya.com\/blog\/ota-updates-for-dot-net-applications\/","guid":"https:\/\/lalitadithya.com\/blog\/ota-updates-for-dot-net-applications\/","category":["openshift","auto-update","ota",".NET","continuous-delivery"]},{"title":"Git-based CMS for Jekyll using GitHub Actions \u2013 Part 3","description":"<p>In the previous <a href=\"\/blog\/git-based-cms-for-jekyll-using-github-actions-part-2\/\">post<\/a>, we manually deployed our website to Azure and we enabled CDN with a custom domain and HTTPS. In this part, we will be automating deployments and we will create staging websites.<\/p>\n\n<p>We will add a GitHub action that will deploy the production website when changes are pushed to the master\/main branch. We will also add one GitHub action that will deploy a staging website when a pull request to master is opened. We will add one last GitHub action that will automatically tear down the staging website when the pull request is either merged or closed. These actions will allow us to leverage git branches to work on multiple posts or website changes without one affecting the other.<\/p>\n\n<!--more-->\n\n<p>The publishing workflow will be:<\/p>\n<ol>\n  <li>Create a new branch for every post or change, commit changes, and push the changes to GitHub<\/li>\n  <li>Open a pull request to the master branch and wait for the bot to comment the URL of the staging website. If you are working in a team, you can share the URL of the PR for review<\/li>\n  <li>If you are happy with the way your changes turned out, then merge the pull request. The production website will be updated, and the staging site will be torn down<\/li>\n  <li>If you aren\u2019t happy with the way your changes turned out, make some modifications, and push them to the branch. The GitHub action will update the staging site. If you are happy with the changes, merge the pull request and the production website will be updated and the staging site will be torn down<\/li>\n<\/ol>\n\n<p>This approach is similar to deploying a website to a deployment slot, testing it out, and then swapping out the slot with production when everything is good.<\/p>\n\n<h1 id=\"part-1--github-action-to-deploy-the-master-branch\">Part 1 \u2013 GitHub action to deploy the master branch<\/h1>\n\n<p>In this section, we will build a continuous delivery pipeline that will build and deploy the website to Azure. The pipeline will:<\/p>\n<ol>\n  <li>Install dependencies<\/li>\n  <li>Build the production version of the website<\/li>\n  <li>Upload the artifact \u2013 This might come in handy if we want to manually deploy a specific version of the website<\/li>\n  <li>Login to Azure<\/li>\n  <li>Sync the build directory with the storage account<\/li>\n  <li>Purge the CDN cache<\/li>\n<\/ol>\n\n<h2 id=\"step-1--prerequisites\">Step 1 \u2013 Prerequisites<\/h2>\n\n<p>Create a free GitHub account <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/github.com\/join\" target=\"_blank\">here<\/a>.<\/p>\n\n<p>Fetch the connection string for the storage account you created and keep it handy. If you don\u2019t remember how to get the connection string for your storage account, you can find the instructions in <a href=\"\/blog\/git-based-cms-for-jekyll-using-github-actions-part-2\/\">part 2<\/a> of this series.<\/p>\n\n<h2 id=\"step-2--push-your-repository-to-github\">Step 2 \u2013 Push your repository to GitHub<\/h2>\n\n<p>Navigate to this <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/github.com\/new\" target=\"_blank\">link<\/a> in your browser to create a new repository. Select the \u201cPrivate\u201d option and be sure to uncheck all the options under \u201cInitialize this repository with\u201d heading. Give it a name and click on \u201cCreate repository\u201d when you are done.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-3\/create-a-git-repo-00.webp\" data-fancybox=\"\" data-title=\"Creating a GitHub repository\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-3\/create-a-git-repo-00.webp\" alt=\"Creating a GitHub repository\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Creating a GitHub repository<\/figcaption>\n  \n<\/figure>\n\n<p>It should take a few seconds and when it is complete, you should see something like this. Copy the URL to your repository.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-3\/create-a-git-repo-01.webp\" data-fancybox=\"\" data-title=\"GitHub repository created\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-3\/create-a-git-repo-01.webp\" alt=\"GitHub repository created\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">GitHub repository created<\/figcaption>\n  \n<\/figure>\n\n<p>Open Visual Studio Code and click on the \u201cSource control\u201d link in the left pane. Click on the three dots on the top, select \u201cRemote\u201d and then select \u201cAdd Remote\u201d<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-3\/create-a-git-repo-02.webp\" data-fancybox=\"\" data-title=\"Adding the remote for the repository using Visual Studio Code\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-3\/create-a-git-repo-02.webp\" alt=\"Adding the remote for the repository using Visual Studio Code\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Adding the remote for the repository using Visual Studio Code<\/figcaption>\n  \n<\/figure>\n\n<p>In the box that appears, paste the URL you copied and hit enter. When it asks you for a remote name enter \u201corigin\u201d and hit enter one more time.<\/p>\n\n<p>Push your changes to GitHub by clicking on the three dots and then selecting \u201cPull, Push\u201d and then selecting \u201cSync\u201d. The sync option will ensure that it first fetches any changes in the remote, merges them with local, and then pushes the result to the remote. This operation might take a minute or two depending on your connection speed and the amount of data that needs to be synced.<\/p>\n\n<p>Navigate to your browser and you should be able to view your code on GitHub. Since you created a private repository, only you will be able to view it and it can\u2019t be indexed by search engines.<\/p>\n\n<h2 id=\"step-3--add-secrets\">Step 3 \u2013 Add secrets<\/h2>\n\n<p>The action needs to talk to the storage account so that it can deploy your website and it needs to talk to your Azure account to purge the CDN cache. To establish this communication, the pipeline will need the connection string for the storage account and a service principle. Both items are like your password and you should not share it with anyone. To keep these items safe we will use GitHub secrets.<\/p>\n\n<p>To add a GitHub secret, navigate to your GitHub repository in your browser, click on the \u201cSettings\u201d tab in the top navigation bar. Select \u201cSecrets\u201d in the left navigation bar that appears and then click on \u201cNew Secret\u201d.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-3\/add-github-secret-00.webp\" data-fancybox=\"\" data-title=\"Adding a new GitHub secret\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-3\/add-github-secret-00.webp\" alt=\"Adding a new GitHub secret\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Adding a new GitHub secret<\/figcaption>\n  \n<\/figure>\n\n<p>In the page that appears, you need to enter a name for your secret and the secret itself in the \u201cvalue\u201d text field and then click on \u201cAdd secret\u201d.<\/p>\n\n<p>Add the first secret with name \u201cPROD_BLOB_CONNECTION_STRING\u201d and the value will be the connection string for your storage account.<\/p>\n\n<p>Let us create a service principle with contributor access to the resource group that contains our storage account. Start by finding your subscription id by running \u201caz account list\u201d.<\/p>\n\n<pre>\n&gt; <b>az account list<\/b>\n[\n  {\n    \"cloudName\": \"AzureCloud\",\n    \"homeTenantId\": \"...\",\n    <b>\"id\": \"...\",<\/b>\n    \"isDefault\": true,\n    \"managedByTenants\": [...],\n    \"name\": \"...\",\n    \"state\": \"...\",\n    \"tenantId\": \"...\",\n    \"user\": {\n      \"name\": \"...\",\n      \"type\": \"...\"\n    }\n  }\n]\n<\/pre>\n\n<p>The command will output a JSON string that contains the details of your account. Your subscription id will be the value in the \u201cid\u201d field (bolded above).<\/p>\n\n<p>Create a service principle by entering \u201caz ad sp create-for-rbac \u2013name \u201csp-blog-deploy\u201d \u2013role contributor \u2013scopes subscriptions\/&lt;&lt;subscription-id&gt;&gt;\/resourceGroups\/&lt;&lt;resource-group-name&gt;&gt; \u2013sdk-auth\u201d (replace the &lt;&lt;subscription-id&gt;&gt; with the id you copied in the previous step; replace &lt;&lt;resource-group-name&gt;&gt; with the name of your resource group) in your terminal.<\/p>\n\n<pre>\n&gt; <b>az ad sp create-for-rbac --name \"sp-blog-deploy\" --role contributor --scopes subscriptions\/subscription-id\/resourceGroups\/resource-group-name --sdk-auth<\/b>\n\nChanging \"sp-blog-deploy\" to a valid URI of \"http:\/\/sp-blog-deploy\", which is the required format used for service principal names\nCreating a role assignment under the scope of \"\/subscriptions\/...\/resourceGroups\/rg-lalit_blog-prod-001\"\n<b>{\n\"clientId\": \"...\",\n\"clientSecret\": \"...\",\n\"subscriptionId\": \"...\",\n\"tenantId\": \"...\",\n\"activeDirectoryEndpointUrl\": \"...\",\n\"resourceManagerEndpointUrl\": \"...\",\n\"activeDirectoryGraphResourceId\": \"...\",\n\"sqlManagementEndpointUrl\": \"...\",\n\"galleryEndpointUrl\": \"...\",\n\"managementEndpointUrl\": \"...\"\n}<\/b>\n<\/pre>\n\n<p>Add one more secret with name \u201cAZURE_CREDENTIALS\u201d. The value will be the JSON that is output by the command.<\/p>\n\n<h2 id=\"step-4--add-the-github-action-for-continuous-delivery\">Step 4 \u2013 Add the GitHub action for continuous delivery<\/h2>\n\n<p>Navigate to your GitHub repository and create an empty action by clicking on the \u201cActions\u201d tab and then clicking on the \u201cset up a workflow yourself\u201d link.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-3\/github-actions-00.webp\" data-fancybox=\"\" data-title=\"Adding a new GitHub Action\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-3\/github-actions-00.webp\" alt=\"Adding a new GitHub Action\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Adding a new GitHub Action<\/figcaption>\n  \n<\/figure>\n\n<p>In the editor that appears, paste the below YAML definition for the GitHub workflow. Replace the placeholders represented by &lt;&lt;&gt;&gt; with the appropriate values and when you are done commit the file to the master branch<\/p>\n\n<div class=\"language-yaml highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Build &amp; Deploy Master<\/span>\n\n<span class=\"na\">on<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">push<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">branches<\/span><span class=\"pi\">:<\/span> <span class=\"pi\">[<\/span> <span class=\"nv\">master<\/span> <span class=\"pi\">]<\/span>\n\n<span class=\"na\">jobs<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">build<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">runs-on<\/span><span class=\"pi\">:<\/span> <span class=\"s\">ubuntu-latest<\/span>\n    <span class=\"na\">steps<\/span><span class=\"pi\">:<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Checkout<\/span>\n        <span class=\"na\">uses<\/span><span class=\"pi\">:<\/span> <span class=\"s\">actions\/checkout@v2<\/span>\n      \n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Set up cache<\/span>\n        <span class=\"na\">uses<\/span><span class=\"pi\">:<\/span> <span class=\"s\">actions\/cache@v2<\/span>\n        <span class=\"na\">with<\/span><span class=\"pi\">:<\/span>\n          <span class=\"na\">path<\/span><span class=\"pi\">:<\/span> <span class=\"s\">vendor\/bundle<\/span>\n          <span class=\"na\">key<\/span><span class=\"pi\">:<\/span> <span class=\"s\">${{ runner.os }}-gems-${{ hashFiles('**\/Gemfile.lock') }}<\/span>\n          <span class=\"na\">restore-keys<\/span><span class=\"pi\">:<\/span> <span class=\"pi\">|<\/span>\n            <span class=\"s\">${{ runner.os }}-gems-<\/span>\n\n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Set up Ruby<\/span>\n        <span class=\"na\">uses<\/span><span class=\"pi\">:<\/span> <span class=\"s\">ruby\/setup-ruby@v1<\/span>\n        <span class=\"na\">with<\/span><span class=\"pi\">:<\/span>\n          <span class=\"na\">ruby-version<\/span><span class=\"pi\">:<\/span> <span class=\"m\">2.6<\/span>\n\n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Install dependencies<\/span>\n        <span class=\"na\">run<\/span><span class=\"pi\">:<\/span> <span class=\"pi\">|<\/span>\n          <span class=\"s\">bundle config path vendor\/bundle<\/span>\n          <span class=\"s\">bundle install --jobs 4 --retry 3<\/span>\n\n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Build<\/span>\n        <span class=\"na\">run<\/span><span class=\"pi\">:<\/span> <span class=\"s\">JEKYLL_ENV=production bundle exec jekyll build<\/span>\n      \n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Upload site<\/span>\n        <span class=\"na\">uses<\/span><span class=\"pi\">:<\/span> <span class=\"s\">actions\/upload-artifact@v2.1.4<\/span>\n        <span class=\"na\">with<\/span><span class=\"pi\">:<\/span>\n          <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">site<\/span>\n          <span class=\"na\">path<\/span><span class=\"pi\">:<\/span> <span class=\"s\">_site\/*<\/span>\n          <span class=\"na\">if-no-files-found<\/span><span class=\"pi\">:<\/span> <span class=\"s\">error<\/span>\n\n  <span class=\"na\">deploy<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">runs-on<\/span><span class=\"pi\">:<\/span> <span class=\"s\">ubuntu-latest<\/span>\n    <span class=\"na\">needs<\/span><span class=\"pi\">:<\/span> <span class=\"pi\">[<\/span><span class=\"nv\">build<\/span><span class=\"pi\">]<\/span>\n    <span class=\"na\">steps<\/span><span class=\"pi\">:<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Download site<\/span>\n        <span class=\"na\">uses<\/span><span class=\"pi\">:<\/span> <span class=\"s\">actions\/download-artifact@v2.0.5<\/span>\n        <span class=\"na\">with<\/span><span class=\"pi\">:<\/span>\n          <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">site<\/span>\n          <span class=\"na\">path<\/span><span class=\"pi\">:<\/span> <span class=\"s\">_site<\/span>\n\n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Azure Login<\/span>\n        <span class=\"na\">uses<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Azure\/login@v1.1<\/span>\n        <span class=\"na\">with<\/span><span class=\"pi\">:<\/span>\n          <span class=\"na\">creds<\/span><span class=\"pi\">:<\/span> <span class=\"s\">${{ secrets.AZURE_CREDENTIALS }}<\/span>\n          \n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Azure CLI Action<\/span>\n        <span class=\"na\">uses<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Azure\/cli@v1.0.0<\/span>\n        <span class=\"na\">with<\/span><span class=\"pi\">:<\/span>\n          <span class=\"na\">inlineScript<\/span><span class=\"pi\">:<\/span> <span class=\"pi\">|<\/span>\n            <span class=\"s\">az storage blob sync -c '$web' -s '_site' --connection-string '${{ secrets.PROD_BLOB_CONNECTION_STRING }}'<\/span>\n            <span class=\"s\">az cdn endpoint purge -g &lt;&lt;resource-group-name&gt;&gt; -n &lt;&lt;cdn-endpoint-name&gt;&gt; --profile-name &lt;&lt;cdn-profile-name&gt;&gt; --content-paths '\/*' --no-wait<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>Once you commit the file, it should kick off the build and deploy process. It might take approximately 5 minutes to complete (subsequent runs will be significantly faster). While that is happening let us understand what the action is doing. The GitHub action runs on every push to the master branch and it has two jobs:<\/p>\n<ol>\n  <li><strong><em>Build<\/em><\/strong> \u2013 This job will build the production version of our website and it will upload the website as an artifact. The steps in this job are:\n    <ul>\n      <li><strong><em>Checkout<\/em><\/strong> \u2013 This step will checkout the code from the main branch<\/li>\n      <li><strong><em>Set up cache<\/em><\/strong> \u2013 This step sets up a cache for the build dependencies. The \u201cInstall dependencies\u201d step is a time-consuming process, so we cache the output of the step to ensure that subsequent runs are faster<\/li>\n      <li><strong><em>Set up Ruby<\/em><\/strong> \u2013 This step sets up ruby version 2.6<\/li>\n      <li><strong><em>Install dependencies<\/em><\/strong> \u2013 This step installs the dependencies (including Jekyll) that are needed to build the website. Any plugins that you have enabled will be installed during this stage<\/li>\n      <li><strong><em>Build<\/em><\/strong> \u2013 This step will build a release\/production version of our website using Jekyll<\/li>\n      <li><strong><em>Upload site<\/em><\/strong> \u2013 This step will upload the site as an artifact<\/li>\n    <\/ul>\n  <\/li>\n  <li><strong><em>Deploy<\/em><\/strong> \u2013 This job will launch only after the previous job has completed and it will deploy our website and purge the CDN cache to ensure users get the latest version of the website. The steps in this job are:\n    <ul>\n      <li><strong><em>Download site<\/em><\/strong> \u2013 This step downloads the site that was uploaded by the previous job<\/li>\n      <li><strong><em>Azure Login<\/em><\/strong> \u2013 This step logins to Azure using the service principle provided<\/li>\n      <li><strong><em>Azure CLI Action<\/em><\/strong> \u2013 This step deploys our website and purges the CDN cache using the cli<\/li>\n    <\/ul>\n  <\/li>\n<\/ol>\n\n<p>Once the pipeline is complete, go to your website and ensure that a bad build hasn\u2019t been deployed. Go ahead, make a change to your website, and push the change to master. The pipeline should kick-off and you should see your updated website live in a couple of minutes.<\/p>\n\n<h1 id=\"part-2--github-actions-to-deploy-a-staging-website-branch\">Part 2 \u2013 GitHub actions to deploy a staging website branch<\/h1>\n\n<p>In this part we will create two GitHub actions:<\/p>\n<ol>\n  <li>GitHub action to deploy a staging website \u2013 This action will be triggered when a pull request is opened, reopened, or updated. It will build the website but unlike the previous action, it will also include draft and future posts in the website. Once the build is complete, it will deploy the website<\/li>\n  <li>GitHub action to tear down the staging website \u2013 This action will be triggered when a pull request is closed. It will delete all the files from the storage account rendering the staging website inaccessible<\/li>\n<\/ol>\n\n<h2 id=\"step-1--prerequisites-1\">Step 1 \u2013 Prerequisites<\/h2>\n\n<p>Create a new storage account and enable the static website feature as demonstrated in <a href=\"\/blog\/git-based-cms-for-jekyll-using-github-actions-part-2\/\">part 2<\/a>. This storage account will be used for the staging websites. You don\u2019t have to enable the CDN or a custom domain for the staging website (you can if you want to). The GitHub action will create a new folder\/directory for every pull request and then deploy the website for that pull request in that folder. The name of the folder will be the pull request number. This will allow us to have multiple versions of the website in staging. This means that we don\u2019t have to create a separate storage account for each pull request and delete them after the pull request is closed or merged.<\/p>\n\n<p>Once the storage account has been created, create a new GitHub secret with name \u201cSTAGE_BLOB_CONNECTION_STRING\u201d and the value as the connection string for the staging storage account.<\/p>\n\n<h2 id=\"step-2--add-github-actions-to-deploy-the-staging-website\">Step 2 \u2013 Add GitHub Actions to deploy the staging website<\/h2>\n\n<p>Create a new blank GitHub action as you did earlier and paste the following YAML in the editor. Replace the items in \u00ab\u00bb with the appropriate values.<\/p>\n\n<div class=\"language-yaml highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Build &amp; Deploy staging website<\/span>\n\n<span class=\"na\">on<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">pull_request<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">branches<\/span><span class=\"pi\">:<\/span> <span class=\"pi\">[<\/span> <span class=\"nv\">master<\/span> <span class=\"pi\">]<\/span>\n\n<span class=\"na\">env<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">stage_directory<\/span><span class=\"pi\">:<\/span> <span class=\"s\">${{ github.event.number }}<\/span>\n  <span class=\"na\">base_url<\/span><span class=\"pi\">:<\/span> <span class=\"s\">&lt;&lt;primary-endpoint-url&gt;&gt;<\/span>\n\n<span class=\"na\">jobs<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">build<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">runs-on<\/span><span class=\"pi\">:<\/span> <span class=\"s\">ubuntu-latest<\/span>\n    <span class=\"na\">steps<\/span><span class=\"pi\">:<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Checkout<\/span>\n        <span class=\"na\">uses<\/span><span class=\"pi\">:<\/span> <span class=\"s\">actions\/checkout@v2<\/span>\n      \n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Set up cache<\/span>\n        <span class=\"na\">uses<\/span><span class=\"pi\">:<\/span> <span class=\"s\">actions\/cache@v2<\/span>\n        <span class=\"na\">with<\/span><span class=\"pi\">:<\/span>\n          <span class=\"na\">path<\/span><span class=\"pi\">:<\/span> <span class=\"s\">vendor\/bundle<\/span>\n          <span class=\"na\">key<\/span><span class=\"pi\">:<\/span> <span class=\"s\">${{ runner.os }}-gems-${{ hashFiles('**\/Gemfile.lock') }}<\/span>\n          <span class=\"na\">restore-keys<\/span><span class=\"pi\">:<\/span> <span class=\"pi\">|<\/span>\n            <span class=\"s\">${{ runner.os }}-gems-<\/span>\n\n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Set up Ruby<\/span>\n        <span class=\"na\">uses<\/span><span class=\"pi\">:<\/span> <span class=\"s\">ruby\/setup-ruby@v1<\/span>\n        <span class=\"na\">with<\/span><span class=\"pi\">:<\/span>\n          <span class=\"na\">ruby-version<\/span><span class=\"pi\">:<\/span> <span class=\"m\">2.6<\/span>\n\n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Install dependencies<\/span>\n        <span class=\"na\">run<\/span><span class=\"pi\">:<\/span> <span class=\"pi\">|<\/span>\n          <span class=\"s\">bundle config path vendor\/bundle<\/span>\n          <span class=\"s\">bundle install --jobs 4 --retry 3<\/span>\n\n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Build<\/span>\n        <span class=\"na\">run<\/span><span class=\"pi\">:<\/span> <span class=\"s\">JEKYLL_ENV=production bundle exec jekyll build --baseurl '${{ env.stage_directory }}' --future --drafts<\/span> \n      \n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Upload site<\/span>\n        <span class=\"na\">uses<\/span><span class=\"pi\">:<\/span> <span class=\"s\">actions\/upload-artifact@v2.1.4<\/span>\n        <span class=\"na\">with<\/span><span class=\"pi\">:<\/span>\n          <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">site<\/span>\n          <span class=\"na\">path<\/span><span class=\"pi\">:<\/span> <span class=\"s\">_site\/*<\/span>\n          <span class=\"na\">if-no-files-found<\/span><span class=\"pi\">:<\/span> <span class=\"s\">error<\/span>\n\n  <span class=\"na\">deploy<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">runs-on<\/span><span class=\"pi\">:<\/span> <span class=\"s\">ubuntu-latest<\/span>\n    <span class=\"na\">needs<\/span><span class=\"pi\">:<\/span> <span class=\"pi\">[<\/span><span class=\"nv\">build<\/span><span class=\"pi\">]<\/span>\n    <span class=\"na\">steps<\/span><span class=\"pi\">:<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Download site<\/span>\n        <span class=\"na\">uses<\/span><span class=\"pi\">:<\/span> <span class=\"s\">actions\/download-artifact@v2.0.5<\/span>\n        <span class=\"na\">with<\/span><span class=\"pi\">:<\/span>\n          <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">site<\/span>\n          <span class=\"na\">path<\/span><span class=\"pi\">:<\/span> <span class=\"s\">_site<\/span>\n\n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Azure Login<\/span>\n        <span class=\"na\">uses<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Azure\/login@v1.1<\/span>\n        <span class=\"na\">with<\/span><span class=\"pi\">:<\/span>\n          <span class=\"na\">creds<\/span><span class=\"pi\">:<\/span> <span class=\"s\">${{ secrets.AZURE_CREDENTIALS }}<\/span>\n          \n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Azure CLI Action<\/span>\n        <span class=\"na\">uses<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Azure\/cli@v1.0.0<\/span>\n        <span class=\"na\">with<\/span><span class=\"pi\">:<\/span>\n          <span class=\"na\">inlineScript<\/span><span class=\"pi\">:<\/span> <span class=\"pi\">|<\/span>\n            <span class=\"s\">az storage blob sync -c '$web\/${{ env.stage_directory }}' -s '_site' --connection-string '${{ secrets.STAGE_BLOB_CONNECTION_STRING }}'<\/span>\n      \n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Comment with URL for the staging website<\/span>\n        <span class=\"na\">uses<\/span><span class=\"pi\">:<\/span> <span class=\"s\">mshick\/add-pr-comment@v1<\/span>\n        <span class=\"na\">with<\/span><span class=\"pi\">:<\/span>\n          <span class=\"na\">message<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Website has been deployed to ${{ env.base_url }}\/${{ env.stage_directory }}<\/span>\n          <span class=\"na\">repo-token<\/span><span class=\"pi\">:<\/span> <span class=\"s\">${{ secrets.GITHUB_TOKEN }}<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>This GitHub action is extremely similar to the one we used for deploying the production version of the website, but it has a few key differences:<\/p>\n<ol>\n  <li>It declares an environment variable called \u201cstage_directory\u201d and it holds the pull request number<\/li>\n  <li>The Jekyll build makes use of the base URL command line flag to set the base URL for the entire website. It makes use of the \u201cstage_directory\u201d environment variable to determine the base URL. This is necessary because each pull request will deploy the staging copy of the website to a separate folder, i.e., if you have a pull request with numbers 1 and 2, then the action for pull request 1 will create a folder called \u20181\u2019 in the root of the storage account and it will deploy its version to that folder. The action for pull request 2 will create a folder called \u20182\u2019 and it will deploy its version to that folder. Since each folder has a website, we make use of the base URL to distinguish the websites. If the domain for staging is \u201cstage.lalitadithya.com\u201d then to view the changes made in pull request 1 we will have to navigate to \u201cstage.lalitadithya.com\/1\u201d and to view the changes made in pull request 2, we have to navigate to \u201cstage.lalitadithya.com\/2\u201d<\/li>\n  <li>The Jekyll build makes use of the \u201cfuture\u201d and the \u201cdrafts\u201d flag to include all posts with a future publish date and posts marked as a draft in the website that will be published<\/li>\n  <li>The sync command deploys the website to the directory based on the pull request number and not the root directory<\/li>\n  <li>The last step adds a comment with the URL for the staging website for easy access<\/li>\n<\/ol>\n\n<p>Commit this action to the master branch, and add one more GitHub action with the YAML below:<\/p>\n\n<div class=\"language-yaml highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Clean up stage website<\/span>\n\n<span class=\"na\">on<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">pull_request<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">branches<\/span><span class=\"pi\">:<\/span> <span class=\"pi\">[<\/span> <span class=\"nv\">master<\/span> <span class=\"pi\">]<\/span>\n    <span class=\"na\">types<\/span><span class=\"pi\">:<\/span> <span class=\"pi\">[<\/span> <span class=\"nv\">closed<\/span> <span class=\"pi\">]<\/span>\n\n<span class=\"na\">env<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">stage_directory<\/span><span class=\"pi\">:<\/span> <span class=\"s\">${{ github.event.number }}<\/span>\n\n<span class=\"na\">jobs<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">clean-up<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">runs-on<\/span><span class=\"pi\">:<\/span> <span class=\"s\">ubuntu-latest<\/span>\n    <span class=\"na\">steps<\/span><span class=\"pi\">:<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Azure Login<\/span>\n        <span class=\"na\">uses<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Azure\/login@v1.1<\/span>\n        <span class=\"na\">with<\/span><span class=\"pi\">:<\/span>\n          <span class=\"na\">creds<\/span><span class=\"pi\">:<\/span> <span class=\"s\">${{ secrets.AZURE_CREDENTIALS }}<\/span>\n          \n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Azure CLI Action<\/span>\n        <span class=\"na\">uses<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Azure\/cli@v1.0.0<\/span>\n        <span class=\"na\">with<\/span><span class=\"pi\">:<\/span>\n          <span class=\"na\">inlineScript<\/span><span class=\"pi\">:<\/span> <span class=\"pi\">|<\/span>\n            <span class=\"s\">az extension add --name storage-preview<\/span>\n            <span class=\"s\">az storage blob directory delete -c '$web' -d '${{ env.stage_directory }}' --connection-string '${{ secrets.STAGE_BLOB_CONNECTION_STRING }}' --recursive<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>This action will make use of the storage preview extension of the az CLI to recursively delete the folder created by the previous action.<\/p>\n\n<p>Go ahead create a branch, make some changes, and open a pull request. Give it a few minutes and then you should see a comment with a URL. Navigate to the URL and you should be able to see the changes deployed to a staging website. Merge the pull request to see your production copy of the website updated and the stage website deleted.<\/p>\n\n<h1 id=\"conclusion\">Conclusion<\/h1>\n\n<p>In this multi-series post, we saw how we can effectively make use of GitHub actions to develop a git-based CMS system for a Jekyll website.<\/p>\n","pubDate":"Sun, 06 Sep 2020 00:00:00 +0000","link":"https:\/\/lalitadithya.com\/blog\/git-based-cms-for-jekyll-using-github-actions-part-3\/","guid":"https:\/\/lalitadithya.com\/blog\/git-based-cms-for-jekyll-using-github-actions-part-3\/","category":["cms","jekyll","git","github-actions","continuous-delivery"]},{"title":"Git-based CMS for Jekyll using GitHub Actions \u2013 Part 2","description":"<p>In the previous post, we looked at what a git-based CMS is, and we set up a Jekyll website on our local machine. I recommend that you read the previous post <a href=\"\/blog\/git-based-cms-for-jekyll-using-github-actions-part-1\/\">here<\/a> before proceeding.<\/p>\n\n<p>In this post, we are going to be creating the necessary resources on Azure and we will deploy our website. If you don\u2019t want to use Azure, you can use any other public cloud provider such as AWS. Most cloud providers have similar offerings to host a website along with a CDN, custom domain, and HTTPS.<\/p>\n\n<!--more-->\n\n<h1 id=\"lets-deploy-our-website-to-the-cloud\">Let\u2019s deploy our website to the cloud!<\/h1>\n\n<p>We are going to be using an object store for hosting and we will use a CDN to reduce page load time for our users across the world. We will configure a custom domain and HTTPS on the CDN and not on the object store.<\/p>\n\n<p><em>Side note: If the images in this post are not visible clearly, click\/tap on them to zoom in.<\/em><\/p>\n\n<h2 id=\"step-1---prerequisites\">Step 1 - Prerequisites<\/h2>\n\n<p>Create a free Azure account if you don\u2019t have one by following the instructions provided <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/azure.microsoft.com\/en-in\/free\/\" target=\"_blank\">here<\/a>. Next, install Azure CLI for your operating system by following the instructions provided <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/docs.microsoft.com\/en-us\/cli\/azure\/install-azure-cli?view=azure-cli-latest\" target=\"_blank\">here<\/a>. Verify that the CLI has been installed correctly by running \u201c<strong>az \u2013version<\/strong>\u201d in your terminal. If it is installed correctly it should say \u201cYour CLI is up-to-date\u201d as shown below<\/p>\n\n<pre>\n&gt; az --version\nazure-cli 2.10.1\ncommand-modules-nspkg 2.0.3\ncore 2.10.1\nnspkg 3.0.4\ntelemetry 1.0.4\nPython location 'C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\python.exe\\'\nExtensions directory 'C:\\Users\\lalit\\.azure\\cliextensions\\'\nPython (Windows) 3.6.8 (tags\/v3.6.8:3c6b436a57, Dec 23 2018, 23:31:17) [MSC v.1916 32 bit (Intel)]\nLegal docs and information: aka.ms\/AzureCliLegal\n<b>Your CLI is up-to-date.<\/b>\nPlease let us know how we are doing: https:\/\/aka.ms\/azureclihats\nand let us know if you're interested in trying out our newest features: https:\/\/aka.ms\/CLIUXstudy\n<\/pre>\n\n<p>Login to Azure using the CLI by running \u201c<strong>az login<\/strong>\u201d in your terminal and following the onscreen instructions.<\/p>\n\n<h2 id=\"step-2---create-an-azure-storage-account\">Step 2 - Create an Azure Storage Account<\/h2>\n\n<p>Open the Azure portal by visiting \u201cportal.azure.com\u201d in your web browser. Create a new storage account by searching for \u201cStorage Accounts\u201d in the search bar on the top of the screen and then clicking on the \u201cAdd\u201d button as shown.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/create-storage-account-00.webp\" data-fancybox=\"\" data-title=\"Creating a new storage account\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/create-storage-account-00.webp\" alt=\"Creating a new storage account\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Creating a new storage account<\/figcaption>\n  \n<\/figure>\n\n<p>In the multi-page form that appears, select a resource group if you already have one or create a new resource group by clicking on the \u201cCreate new\u201d button. Select \u201cGeneral Purpose V2\u201d for kind; \u201cHot\u201d for access tier and \u201cGeo-redundant storage (GRS)\u201d for replication. Give your storage account a name, pick one of the recommended locations and click on \u201cReview + create\u201d. The location of the storage account does not matter because we will be using a CDN which will cache our website across the globe.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/create-storage-account-01.webp\" data-fancybox=\"\" data-title=\"Selecting options on first tab for storage account creation\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/create-storage-account-01.webp\" alt=\"Selecting options on first tab for storage account creation\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Selecting options on first tab for storage account creation<\/figcaption>\n  \n<\/figure>\n\n<p>Give Azure a few seconds to run some validations. When it is done, click on the \u201cCreate\u201d button as shown.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/create-storage-account-02.webp\" data-fancybox=\"\" data-title=\"Azure validating storage account\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/create-storage-account-02.webp\" alt=\"Azure validating storage account\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Azure validating storage account<\/figcaption>\n  \n<\/figure>\n\n<p>Azure will now start deploying your request and you should see something like this<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/create-storage-account-03.webp\" data-fancybox=\"\" data-title=\"Azure deploying storage account\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/create-storage-account-03.webp\" alt=\"Azure deploying storage account\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Azure deploying storage account<\/figcaption>\n  \n<\/figure>\n\n<p>While the deployment is in progress, let us take a closer look at the options that we selected:<\/p>\n<ol>\n  <li><strong><em>Resource Group<\/em><\/strong> - A resource group is a container for your resources. It is recommended that you create one resource group per project or resource type.<\/li>\n  <li><strong><em>Storage Account Name<\/em><\/strong> - It is a globally unique name that is used to identify your storage account.<\/li>\n  <li><strong><em>Replication<\/em><\/strong> - Redundancy ensures that your storage account meets the Service-Level Agreement (SLA). We picked GRS which means that Azure will copy our data synchronously three times in the primary region using LRS. It then copies our data asynchronously to the secondary region which will then be replicated using LRS. This will ensure that our website will be up even if an entire Azure region or datacenter is down<\/li>\n  <li><strong><em>Access Tier<\/em><\/strong> - We picked the \u201chot\u201d tier as our data will be frequently accessed<\/li>\n<\/ol>\n\n<p>Once the deployment is complete, you should see something like this. Click on \u201cGo to resource\u201d to go to your storage account.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/create-storage-account-04.webp\" data-fancybox=\"\" data-title=\"Azure storage account deployment complete\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/create-storage-account-04.webp\" alt=\"Azure storage account deployment complete\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Azure storage account deployment complete<\/figcaption>\n  \n<\/figure>\n\n<p>Enable the static website feature by clicking on the \u201cStatic website\u201d link in the left navigation bar and then toggling the switch to \u201cEnabled\u201d. In the \u201cIndex document name\u201d enter \u201cindex.html\u201d and in the \u201cError document name\u201d enter \u201c404.html\u201d and then hit \u201cSave\u201d.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/enable-static-website-00.webp\" data-fancybox=\"\" data-title=\"Enabling static website in the storage account\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/enable-static-website-00.webp\" alt=\"Enabling static website in the storage account\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Enabling static website in the storage account<\/figcaption>\n  \n<\/figure>\n\n<p>Once you hit save you should see something like this.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/enable-static-website-01.webp\" data-fancybox=\"\" data-title=\"Static website being enabled\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/enable-static-website-01.webp\" alt=\"Static website being enabled\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Static website being enabled<\/figcaption>\n  \n<\/figure>\n\n<p>While that is happening, let us understand what we just did. In the previous <a href=\"\/blog\/git-based-cms-for-jekyll-using-github-actions-part-1\/\">part<\/a> of this series, we saw that Jekyll generates HTML and CSS for our website and it will place them in the \u201c_site\u201d folder. The static website feature allows us to upload these HTML, CSS, JS, and image files to a special container called \u201c$web\u201d and Azure will take care of serving these files to our users. All the complex server maintenance and scaling are taken care by Azure and we don\u2019t have to worry about it.<\/p>\n\n<p>Once the deployment is done, you should see<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/enable-static-website-02.webp\" data-fancybox=\"\" data-title=\"Static website enabled\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/enable-static-website-02.webp\" alt=\"Static website enabled\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Static website enabled<\/figcaption>\n  \n<\/figure>\n\n<p>Note down the \u201cPrimary Endpoint\u201d someplace safe. It will come in very handy later. This is the Microsoft provided URL for our website. If you open a new tab in your browser and navigate to the endpoint now you will see an error page, i.e., a not found page. This is because we have not deployed our website to the \u201c$web\u201d container yet.<\/p>\n\n<h2 id=\"step-3---deploy\">Step 3 - Deploy!<\/h2>\n\n<p>To deploy your website, you will need the connection string for your storage account. Get the connection string by clicking on \u201cAccess Keys\u201d in the left navigation pane and then copy the connection string (under key 1) as shown. <strong><em>Note: The connection string is a password that provides complete access (root access) to all the files inside in your storage account. Do not share it with anybody.<\/em><\/strong><\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/connection-string.webp\" data-fancybox=\"\" data-title=\"Fetching the connection string for the storage account\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/connection-string.webp\" alt=\"Fetching the connection string for the storage account\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Fetching the connection string for the storage account<\/figcaption>\n  \n<\/figure>\n\n<p>Navigate to the directory where you stored your website. Build your website by running \u201c<strong><em>JEKYLL_ENV=production bundle exec jekyll build<\/em><\/strong>\u201d in your terminal. When the build completes successfully, enter the command \u201c<strong><em>az storage blob sync -c \u2018$web\u2019 -s \u2018_site\u2019 \u2013connection-string \u2018\u00abBLOB_CONNECTION_STRING\u00bb\u2018<\/em><\/strong>\u201d in your terminal to deploy your website to your storage account.<\/p>\n\n<div class=\"highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>&gt; cd .\\blog\\my-awesome-site\\\n&gt; JEKYLL_ENV=production bundle exec jekyll build\nConfiguration file: \/mnt\/c\/Users\/lalit\/blog\/my-awesome-site\/_config.yml\n            Source: \/mnt\/c\/Users\/lalit\/blog\/my-awesome-site\n       Destination: \/mnt\/c\/Users\/lalit\/blog\/my-awesome-site\/_site\n Incremental build: disabled. Enable with --incremental\n      Generating...\n                    done in 7.172 seconds.\nAuto-regeneration: disabled. Use --watch to enable.\n&gt; az storage blob sync -c '$web' -s '_site' --connection-string '&lt;&lt;connection string&gt;'\nThis command is in preview. It may be changed\/removed in a future release.\nAzcopy command: [...]\nINFO: Any empty folders will not be processed, because source and\/or destination doesn't have full folder support\nJob 7dbd6075-cb6a-8c40-5d69-811004e79f3d has started\nLog file is located at: C:\\Users\\lalit\\.azcopy\\7dbd6075-cb6a-8c40-5d69-811004e79f3d.log\nINFO: azcopy.exe: A newer version 10.6.0 is available to download\n0.0 %, 0 Done, 0 Failed, 8 Pending, 8 Total, 2-sec Throughput (Mb\/s): 0\nJob 7dbd6075-cb6a-8c40-5d69-811004e79f3d Summary\nFiles Scanned at Source: 8\nFiles Scanned at Destination: 0\nElapsed Time (Minutes): 0.0668\nNumber of Copy Transfers for Files: 8\nNumber of Copy Transfers for Folder Properties: 0\nTotal Number Of Copy Transfers: 8\nNumber of Copy Transfers Completed: 8\nNumber of Copy Transfers Failed: 0\nNumber of Deletions at Destination: 0\nTotal Number of Bytes Transferred: 69702\nTotal Number of Bytes Enumerated: 69702\nFinal Job Status: Completed\n<\/code><\/pre><\/div><\/div>\n\n<p>The \u201caz storage blob sync \u2026\u201d will \u201csync\u201d all the files that are present in the \u201c_site\u201d folder to the \u201c$web\u201d container. As we saw in <a href=\"\/blog\/git-based-cms-for-jekyll-using-github-actions-part-1\/\">part 1<\/a> of this series, the \u201c_site\u201d folder contains the HTML and CSS files Jekyll generated for your site. We perform a sync operation and not a batch because if there are files in the \u201c$web\u201d container that are not in the \u201c_site\u201d folder, they must be deleted. The sync command will do that, but the batch upload command will not. This will ensure that if you remove an image or post on your website it will be removed from your storage account as well.<\/p>\n\n<p>Now open a new tab in your browser and navigate to the \u201cPrimary endpoint\u201d that you saved in the previous step and you should be able to see your website.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/first-time-website-live.webp\" data-fancybox=\"\" data-title=\"Website hosted and live from the storage account\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/first-time-website-live.webp\" alt=\"Website hosted and live from the storage account\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Website hosted and live from the storage account<\/figcaption>\n  \n<\/figure>\n\n<h2 id=\"step-4---cdn\">Step 4 - CDN!<\/h2>\n\n<p>Let us add a content delivery network (CDN). A CDN will cache our website across the world in its points of presence (POPs). This will ensure quick transfer of our HTML, CSS, JS, and images to our users\u2019 devices.<\/p>\n\n<p>To add a CDN, click on the \u201cAzure CDN\u201d in the left navigation and in the form that appears, enter a \u201cCDN profile name\u201d in the first text box. Select \u201cStandard Microsoft\u201d for the pricing tier and provide a memorable URL for the \u201cCDN endpoint name\u201d. Ensure that in the \u201cOrigin hostname\u201d the static website option is selected and then hit \u201cCreate\u201d<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/create-cdn-00.webp\" data-fancybox=\"\" data-title=\"Add a CDN for the website\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/create-cdn-00.webp\" alt=\"Add a CDN for the website\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Add a CDN for the website<\/figcaption>\n  \n<\/figure>\n\n<p>You should now see something like this<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/create-cdn-01.webp\" data-fancybox=\"\" data-title=\"CDN being deployed\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/create-cdn-01.webp\" alt=\"CDN being deployed\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">CDN being deployed<\/figcaption>\n  \n<\/figure>\n\n<p>Give it a couple of minutes to complete the configuration. When it is complete, you should see something like this<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/create-cdn-02.webp\" data-fancybox=\"\" data-title=\"CDN deployed successfully\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/create-cdn-02.webp\" alt=\"CDN deployed successfully\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">CDN deployed successfully<\/figcaption>\n  \n<\/figure>\n\n<p>Now navigate to CDN endpoint name followed by \u201c.azureedge.net\u201d in a new tab. In my case the CDN endpoint name was \u201clalitblog\u201d, hence I will navigate to \nlalitblog.azureedge.net\u201d. You should see something like this<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/create-cdn-03.webp\" data-fancybox=\"\" data-title=\"CDN not working? Something wrong?\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/create-cdn-03.webp\" alt=\"CDN not working? Something wrong?\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">CDN not working? Something wrong?<\/figcaption>\n  \n<\/figure>\n\n<p>Did we mess something up? No. It can take up to 30 minutes to update the CDN POPs around the world. Please be patient and try again after some time. When I refreshed the page after 20 minutes, I was able to see my website.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/create-cdn-04.webp\" data-fancybox=\"\" data-title=\"Website being served using CDN\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/create-cdn-04.webp\" alt=\"Website being served using CDN\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Website being served using CDN<\/figcaption>\n  \n<\/figure>\n\n<p>We now have a CDN for our website. Notice that the website loads significantly faster. This is because the website is being fetched from the location that closest to you and not from the origin. In my case, my website was being fetched from the Indian CDN POP and not from US East where my website is hosted.<\/p>\n\n<h2 id=\"step-5---custom-domain-and-https\">Step 5 - Custom domain and HTTPS!<\/h2>\n\n<p>Let us now add a custom domain to our website. I am going to be adding \u201cblog.lalitadithya.com\u201d as my custom domain. To add a custom domain we need to add a CNAME pointing to the CDN endpoint.<\/p>\n\n<p>Login to your DNS provider and add a CNAME for the custom domain of your choice pointing to your CDN endpoint. In my case, I logged in to Azure DNS and then added a CNAME for \u201cblog.lalitadithya.com\u201d pointing to \u201clalitblog.azureedge.net\u201d. DNS propagation might take some time, so wait at least about 5 to 10 minutes before proceeding further.<\/p>\n\n<p>Switch back to your Azure portal and then click on your hostname under the endpoints subsection as shown.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/custom-domain-01.webp\" data-fancybox=\"\" data-title=\"Navigating to the CDN endpoint\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/custom-domain-01.webp\" alt=\"Navigating to the CDN endpoint\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Navigating to the CDN endpoint<\/figcaption>\n  \n<\/figure>\n\n<p>Click on the \u201c+ Custom domain\u201d button in the top navigation bar.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/custom-domain-02.webp\" data-fancybox=\"\" data-title=\"Adding a custom domain - step 1\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/custom-domain-02.webp\" alt=\"Adding a custom domain - step 1\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Adding a custom domain - step 1<\/figcaption>\n  \n<\/figure>\n\n<p>In the fly-out menu, enter your custom domain name and click on the \u201cAdd\u201d button. In my case, I will enter \u201cblog.lalitadithya.com\u201d<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/custom-domain-03.webp\" data-fancybox=\"\" data-title=\"Adding a custom domain - step 2\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/custom-domain-03.webp\" alt=\"Adding a custom domain - step 2\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Adding a custom domain - step 2<\/figcaption>\n  \n<\/figure>\n\n<p>You should see:<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/custom-domain-04.webp\" data-fancybox=\"\" data-title=\"Custom domain being configured\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/custom-domain-04.webp\" alt=\"Custom domain being configured\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Custom domain being configured<\/figcaption>\n  \n<\/figure>\n\n<p>Give it a few minutes to complete. Once it is complete, you should see something like this. To enable HTTPS, click on your custom domain as shown.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/custom-domain-05.webp\" data-fancybox=\"\" data-title=\"Custom domain configured successfully\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/custom-domain-05.webp\" alt=\"Custom domain configured successfully\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Custom domain configured successfully<\/figcaption>\n  \n<\/figure>\n\n<p>Flip the toggle switch to \u201cOn\u201d and then click \u201cSave\u201d.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/https-01.webp\" data-fancybox=\"\" data-title=\"Enabling HTTPS\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/https-01.webp\" alt=\"Enabling HTTPS\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Enabling HTTPS<\/figcaption>\n  \n<\/figure>\n\n<p>You should now see something like this<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/https-02.webp\" data-fancybox=\"\" data-title=\"HTTPS being enabled\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/https-02.webp\" alt=\"HTTPS being enabled\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">HTTPS being enabled<\/figcaption>\n  \n<\/figure>\n\n<p>After about a minute, the deployment would have completed, and you should see something like this<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/https-03.webp\" data-fancybox=\"\" data-title=\"HTTPS propagation started\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/https-03.webp\" alt=\"HTTPS propagation started\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">HTTPS propagation started<\/figcaption>\n  \n<\/figure>\n\n<p>The HTTPS propagation process can take at least half-hour to at most 2 hours, so please be patient. The page will update itself automatically when a step is complete. Once all the steps are complete, you should see something like this.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/https-04.webp\" data-fancybox=\"\" data-title=\"HTTPS propagation completed\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/https-04.webp\" alt=\"HTTPS propagation completed\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">HTTPS propagation completed<\/figcaption>\n  \n<\/figure>\n\n<p>Now open one more tab and navigate to the custom domain. In my case, I will navigate to \u201cblog.lalitadithya.com\u201d. You should now see your website with your custom domain with HTTPS enabled.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/final-website-with-a-custom-domain.webp\" data-fancybox=\"\" data-title=\"Website deployed with HTTPS and a custom domain\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-2\/final-website-with-a-custom-domain.webp\" alt=\"Website deployed with HTTPS and a custom domain\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Website deployed with HTTPS and a custom domain<\/figcaption>\n  \n<\/figure>\n\n<h1 id=\"conclusion\">Conclusion<\/h1>\n\n<p>In this post, we created all the necessary resources on Azure and deployed our website manually. We configured a CDN, added a custom domain, and enabled HTTPS. In the next post we will look at how we can build a continuous deployment pipeline using GitHub actions.<\/p>\n","pubDate":"Sat, 29 Aug 2020 00:00:00 +0000","link":"https:\/\/lalitadithya.com\/blog\/git-based-cms-for-jekyll-using-github-actions-part-2\/","guid":"https:\/\/lalitadithya.com\/blog\/git-based-cms-for-jekyll-using-github-actions-part-2\/","category":["cms","jekyll","git","github-actions","continuous-delivery"]},{"title":"Git based CMS for Jekyll using GitHub Actions \u2013 Part 1","description":"<p>In this multi post series we are going to be building a git-based headless CMS. We will be using Jekyll to build our website, GitHub actions for building a continuous delivery pipeline that will deploy our website to Azure. We will leverage the infinite scalability of Azure storage account\u2019s blob storage along with Azure CDN to reduce the load time for our users around the world.<\/p>\n\n<!--more-->\n\n<h1 id=\"what-is-a-git-based-cms\">What is a Git-based CMS?<\/h1>\n\n<p>A git-based CMS allows users to leverage git as a single source of truth for all their content, i.e., blog posts, website theme, plugins, assets etc. Every time a user pushes a change (a change can be anything from adding a new post to updating a older post to adding a plugin or changing the theme) to the master\/main branch, a pipeline will be triggered that will build and deploy the website.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-1\/git-based-publishing-flow.webp\" data-fancybox=\"\" data-title=\"Workflow for a git based CMS\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-1\/git-based-publishing-flow.webp\" alt=\"Workflow for a git based CMS\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Workflow for a git based CMS<\/figcaption>\n  \n<\/figure>\n\n<h2 id=\"advantages-of-a-git-based-cms\">Advantages of a git-based CMS<\/h2>\n<ol>\n  <li><strong>Version control for everything<\/strong> \u2013 Images, scripts, plugins, content, theme etc are version controlled out of box<\/li>\n  <li><strong>Easy rollback<\/strong> \u2013 Just rollback the bad commit and the pipeline will take care of the rest<\/li>\n  <li><strong>Ability to leverage git branches to work on multiple things at the same time<\/strong> - When one post or update is complete, just merge that branch to master\/main and the pipeline will take care of the rest<\/li>\n  <li><strong>Ability to leverage the pull request feature for an easy review process<\/strong> \u2013 A pipeline will deploy the changes made in the pull request to a non-public website to make it easy for reviewers to view changes<\/li>\n  <li><strong>No vendor lock in<\/strong> \u2013 Don\u2019t want to use GitHub? You can use GitLab or an on-premises git server. Don\u2019t want to use Azure? You can make use of AWS or GCP or IBM cloud. Even switching from one cloud provider to the other is as simple as updating the pipeline(s)<\/li>\n<\/ol>\n\n<h2 id=\"disadvantages-of-git-based-cms\">Disadvantages of git-based CMS<\/h2>\n<ol>\n  <li>Not easy to use for non-technical users<\/li>\n  <li>Might not be the best if your content is changing rapidly<\/li>\n<\/ol>\n\n<h1 id=\"high-level-architecture\">High level architecture<\/h1>\n\n<p>Given below is a diagrammatic representation of what we are going to be building.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-1\/high-level-architecture.webp\" data-fancybox=\"\" data-title=\"High level architecture of what we will be building\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-1\/high-level-architecture.webp\" alt=\"High level architecture of what we will be building\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">High level architecture of what we will be building<\/figcaption>\n  \n<\/figure>\n\n<p>Let us jump right into it shall we?<\/p>\n\n<h1 id=\"what-is-jekyll\">What is Jekyll?<\/h1>\n\n<p>Jekyll is a simple, blog-aware, static site generator. It is written in Ruby and it is distributed under the open source MIT license.<\/p>\n\n<blockquote>\n  <p>Jekyll does what you tell it to do \u2014 no more, no less. It doesn\u2019t try to outsmart users by making bold assumptions, nor does it burden them with needless complexity and configuration. Put simply, Jekyll gets out of your way and allows you to concentrate on what truly matters: your content.<\/p>\n\n  <p>\u2013 <cite>Jekyll readme file<\/cite><\/p>\n<\/blockquote>\n\n<h2 id=\"why-not-wordpress-like-38-of-the-internet\">Why not WordPress like 38% of the internet?<\/h2>\n\n<p>Unlike WordPress that needs to be run on servers (that we must manage, scale, maintain, and secure), Jekyll generates good old HTML and CSS files which can be hosted on any object store such as Azure Storage Accounts (or Amazon Web Services\u2019 S3). We do not have to manage, scale, maintain and secure any compute infrastructure or databases.<\/p>\n\n<p>We can leverage the \u201conly pay for what you use\u201d model to lower our costs. We only need to pay for the storage we use and the amount of data that we transfer to our users. In the case of WordPress, we will need pay a fixed monthly cost irrespective of the amount of data that we store or the amount of data we serve to our users.<\/p>\n\n<h1 id=\"lets-build-our-website\">Let\u2019s build our website<\/h1>\n\n<p>I will be using Ubuntu running in WSL2 on Windows. You will be able to follow along using any operating system, as all the tools we will be using are available for all the major operating systems.<\/p>\n\n<h2 id=\"step-1--prerequisites\">Step 1 \u2013 Prerequisites<\/h2>\n\n<p>Install Jekyll for your operating system by following the instructions provided <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/jekyllrb.com\/docs\/installation\/\" target=\"_blank\">here<\/a>. Verify that you have installed Jekyll correct by typing \u201c<strong>jekyll -v<\/strong>\u201d in your command line. You should see something like this:<\/p>\n\n<pre>\n$ <strong>jekyll -v<\/strong>\njekyll 4.1.1\n<\/pre>\n\n<p>Install git for your operating system by following the instructions provided <a href=\"https:\/\/git-scm.com\/book\/en\/v2\/Getting-Started-Installing-Git\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">here<\/a>. Verify that you have installed git successfully by typing \u201c<strong>git \u2013version<\/strong>\u201d in your command line. You should see something like this:<\/p>\n\n<pre>\n$ <strong>git --version<\/strong>\ngit version 2.25.1\n<\/pre>\n\n<p>Install Visual Studio Code by following the instructions provided <a href=\"https:\/\/code.visualstudio.com\/download\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">here<\/a>.<\/p>\n\n<h2 id=\"step-2--create-a-website\">Step 2 \u2013 Create a website<\/h2>\n\n<p>Navigate to the directory where you want to save your blog and run \u201c<strong>sudo jekyll new my-awesome-site<\/strong>\u201d to create a new Jekyll site. You can replace \u201cmy-awesome-site\u201d with any name you choose. Jekyll will create a folder called \u201cmy-awesome-site\u201d and it will scaffold a basic website for you inside that folder.<\/p>\n\n<pre>\n$ <strong>sudo jekyll new my-awesome-site<\/strong>\n[sudo] password for lalit:\nRunning bundle install in \/mnt\/c\/Users\/lalit\/blog\/my-awesome-site...\n\u00a0 Bundler: Fetching gem metadata from https:\/\/rubygems.org\/..........\n\u00a0 Bundler: Fetching gem metadata from https:\/\/rubygems.org\/.\n\u00a0 Bundler: Resolving dependencies...\n\u00a0 Bundler: Using public_suffix 4.0.5\n\u00a0 Bundler: Using addressable 2.7.0\n\u2026\n\u2026\nNew jekyll site installed in \/mnt\/c\/Users\/lalit\/blog\/my-awesome-site.\n<\/pre>\n\n<p>Now navigate to the \u201cmy-awesome-site\u201d directory and run \u201c<strong>sudo bundle exec jekyll serve<\/strong>\u201d to start serving your website on localhost<\/p>\n\n<pre>\n$ <strong>cd my-awesome-site\/<\/strong>\n$ <strong>sudo bundle exec jekyll serve<\/strong>\nConfiguration file: \/mnt\/c\/Users\/lalit\/blog\/my-awesome-site\/_config.yml\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Source: \/mnt\/c\/Users\/lalit\/blog\/my-awesome-site\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Destination: \/mnt\/c\/Users\/lalit\/blog\/my-awesome-site\/_site\n\u00a0Incremental build: disabled. Enable with --incremental\n\u00a0\u00a0\u00a0\u00a0\u00a0 Generating...\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Jekyll Feed: Generating feed for posts\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 done in 1.399 seconds.\n\u00a0\u00a0\u00a0 Server address: http:\/\/127.0.0.1:4000\/\n\u00a0 Server running... press ctrl-c to stop.\n<\/pre>\n\n<p>Open your favourite web browser and navigate to <strong>http:\/\/127.0.0.1:4000\/<\/strong>. You should be able to see your website and it should look something like this<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-1\/first-website.webp\" data-fancybox=\"\" data-title=\"\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-1\/first-website.webp\" alt=\"A simple Jekyll website running on local host\" \/>\n  <\/a>\n  \n<\/figure>\n\n<p>Open the \u201cmy-awesome-site\u201d folder in Visual Studio Code. Let us examine the contents of that folder to better understand what Jekyll has created for us.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-1\/jekyll-folder-structure.webp\" data-fancybox=\"\" data-title=\"\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-1\/jekyll-folder-structure.webp\" alt=\"The Jekyll folder structure\" \/>\n  <\/a>\n  \n<\/figure>\n\n<ul>\n  <li><strong>_posts<\/strong> \u2013 This folder will contain all your blog posts. Right now, we have only one blog post that is titled \u201cWelcome to Jekyll\u201d. The naming convention of these files are important. All files in the posts folder must follow the naming convention - YEAR-MONTH-DAY-title.markdown<\/li>\n  <li><strong>_site<\/strong> \u2013 This folder contains that the generated HTML and CSS files. We will be uploading the contents of this directory to our hosting provider.<\/li>\n  <li><strong>_config.yml<\/strong> \u2013 This file contains the configuration information about the site. Configuration items include the title of your website, your email address, the theme you are using, the plugins you have enabled etc.<\/li>\n  <li><strong>404.html<\/strong> \u2013 This is the file that will be rendered when a user tries to visit a link that does exist on your website<\/li>\n  <li><strong>about.markdown<\/strong> \u2013 This file contains the contents for the \u201cabout\u201d page<\/li>\n  <li><strong>index.markdown<\/strong> \u2013 This file contains the contents for your website\u2019s homepage<\/li>\n  <li><strong>Gemfile<\/strong> \u2013 Jekyll is written in Ruby, therefore, all your themes, plugins are ruby gems. The Gemfile tells ruby what gems to install for your website can to work as expected<\/li>\n<\/ul>\n\n<h2 id=\"step-3--create-a-git-repository\">Step 3 \u2013 Create a git repository<\/h2>\n\n<p>Now that we have a basic understanding of the folder structure, let us \u201csave\u201d our progress so far by creating a git repo and committing our files. To do that, click on the \u201cSource Control\u201d icon in the left navigation bar in Visual Studio Code and click on initialize repository.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-1\/create-a-git-repo.webp\" data-fancybox=\"\" data-title=\"\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-1\/create-a-git-repo.webp\" alt=\"Create a git repo using Visual Studio Code\" \/>\n  <\/a>\n  \n<\/figure>\n\n<p>Give it a few seconds for it initialize the git repository and once it is done, you should see a text box asking for a message. Give a message like \u201cInitial commit\u201d and then click on the check mark as shown.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-1\/initial-commit.webp\" data-fancybox=\"\" data-title=\"\">\n    <img src=\"\/images\/git-based-cms-for-jekyll-using-github-actions-part-1\/initial-commit.webp\" alt=\"Commit using Visual Studio Code\" \/>\n  <\/a>\n  \n<\/figure>\n\n<h1 id=\"conclusion\">Conclusion<\/h1>\n\n<p>In this post we learned about git-based CMS and then we created a basic website using Jekyll. In the upcoming posts, we will set up the necessary resource on Azure and host our website.<\/p>\n","pubDate":"Sun, 23 Aug 2020 00:00:00 +0000","link":"https:\/\/lalitadithya.com\/blog\/git-based-cms-for-jekyll-using-github-actions-part-1\/","guid":"https:\/\/lalitadithya.com\/blog\/git-based-cms-for-jekyll-using-github-actions-part-1\/","category":["cms","jekyll","git","github-actions","continuous-delivery"]},{"title":"Building a Cloud Native Task Execution Framework on Azure","description":"<p>In this post we are going to be looking at how to build a consumption-based task execution framework that can scale elastically as demand rises and falls.<\/p>\n\n<!--more-->\n\n<p>The architecture diagram for the complete framework is shown below:<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/building-a-cloud-native-task-execution-framework-on-azure\/architecture-diagram.webp\" data-fancybox=\"\" data-title=\"Architecture Diagram\">\n    <img src=\"\/images\/building-a-cloud-native-task-execution-framework-on-azure\/architecture-diagram.webp\" alt=\"Architecture Diagram\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Architecture Diagram<\/figcaption>\n  \n<\/figure>\n\n<p>In this post we will only be exploring the core components of the framework, namely:<\/p>\n<ol>\n  <li>Task Runners \u2013 Docker containers running on Azure Container Instances<\/li>\n  <li>Task Orchestrator \u2013 Azure Functions App<\/li>\n<\/ol>\n\n<h1 id=\"task-runners\">Task Runners<\/h1>\n\n<p>At a very high level, a task runner is a docker container that will download and execute a task.<\/p>\n\n<h2 id=\"what-is-a-task\">What is a task?<\/h2>\n\n<p>A task can be anything ranging from powering off a virtual machine to deploying a new service to a Kubernetes cluster.<\/p>\n\n<p>For our framework, a task is any *nix executable that is stored in an Azure storage container. This executable will read input from environment variables.<\/p>\n\n<p>To keep things simple, let us take the example of turning off a virtual machine as our task. The bash script for this task shown below:<\/p>\n\n<div class=\"language-bash highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"c\">#! \/bin\/bash<\/span>\naz login <span class=\"nt\">--identity<\/span>\naz vm stop <span class=\"nt\">--resource-group<\/span> <span class=\"s2\">\"<\/span><span class=\"k\">${<\/span><span class=\"nv\">ResourceGroupName<\/span><span class=\"k\">}<\/span><span class=\"s2\">\"<\/span> <span class=\"nt\">--name<\/span> <span class=\"s2\">\"<\/span><span class=\"k\">${<\/span><span class=\"nv\">VMName<\/span><span class=\"k\">}<\/span><span class=\"s2\">\"<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>The script logs in to Azure using the assigned identity and uses the \u201caz vm stop\u201d command to stop the virtual machine. For this script to run, we will need the Azure CLI.<\/p>\n\n<h2 id=\"what-are-task-runners\">What are Task Runners?<\/h2>\n\n<p>A task runner is a docker container that contains all the dependencies needed for a task to execute and it will download and launch the task executable.<\/p>\n\n<p>For our example of turning off a virtual machine, the only dependency we have is the Azure CLI. Let us create a docker image that contains the Azure CLI and call it \u201clalitadithya\/az-cli-image\u201d.<\/p>\n\n<p>The Dockerfile for this image is shown below:<\/p>\n\n<div class=\"language-dockerfile highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"k\">FROM<\/span><span class=\"s\"> mcr.microsoft.com\/azure-cli<\/span>\n<span class=\"k\">WORKDIR<\/span><span class=\"s\"> \/scripts<\/span>\n<span class=\"k\">COPY<\/span><span class=\"s\"> script-runner.sh .<\/span>\n<span class=\"k\">ENV<\/span><span class=\"s\"> PATH=\"\/scripts:${PATH}\"<\/span>\n<span class=\"k\">RUN <\/span><span class=\"nb\">chmod <\/span>a+x script-runner.sh\n<span class=\"k\">CMD<\/span><span class=\"s\"> [\".\/script-runner.sh\", \"\"]<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>The docker file copies a bash script called \u201cscript-runner\u201d from the host, sets the executable permissions for the script and then executes the script. The \u201cscript-runner\u201d is shown below:<\/p>\n\n<div class=\"language-bash highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"c\">#! \/bin\/bash<\/span>\nwget <span class=\"nt\">-O<\/span> script.sh <span class=\"nv\">$1<\/span>\n<span class=\"nb\">chmod <\/span>a+x script.sh\n.\/script.sh\n<\/code><\/pre><\/div><\/div>\n\n<p>The script runner downloads the task script\/executable from the URL provided in the command line and executes the task script\/executable.<\/p>\n\n<h2 id=\"what-are-task-definitions\">What are Task Definitions?<\/h2>\n\n<p>A task definition is a JSON document that contains metadata associated with a task such as the name of the executable for the task, the docker image that should be used to execute the task and the list of input parameters for the task. The task definitions will be stored along with the task executables.<\/p>\n\n<p>The task definition for our turning off a virtual machine task is shown below:<\/p>\n\n<div class=\"language-json highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"p\">{<\/span><span class=\"w\">\n  <\/span><span class=\"nl\">\"taskName\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"StopVM\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n  <\/span><span class=\"nl\">\"dockerImage\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"lalitadithya\/az-cli-image:latest\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n  <\/span><span class=\"nl\">\"scriptName\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"stop-vm-azure.sh\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n  <\/span><span class=\"nl\">\"scriptParameters\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"w\">\n    <\/span><span class=\"p\">{<\/span><span class=\"w\">\n      <\/span><span class=\"nl\">\"parameterName\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"VMName\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n      <\/span><span class=\"nl\">\"parameterType\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"string\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n      <\/span><span class=\"nl\">\"required\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"kc\">true<\/span><span class=\"w\">\n    <\/span><span class=\"p\">},<\/span><span class=\"w\">\n    <\/span><span class=\"p\">{<\/span><span class=\"w\">\n      <\/span><span class=\"nl\">\"parameterName\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"ResourceGroupName\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n      <\/span><span class=\"nl\">\"parameterType\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"string\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n      <\/span><span class=\"nl\">\"required\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"kc\">true<\/span><span class=\"w\">\n    <\/span><span class=\"p\">}<\/span><span class=\"w\">\n  <\/span><span class=\"p\">]<\/span><span class=\"w\">\n<\/span><span class=\"p\">}<\/span><span class=\"w\">\n<\/span><\/code><\/pre><\/div><\/div>\n\n<h1 id=\"task-orchestrator\">Task Orchestrator<\/h1>\n\n<p>Let us now look at how we can make use of an Azure Functions app to automatically spawn Docker containers to execute tasks.<\/p>\n\n<p>Here are the tasks that needs to be accomplished by our task orchestrator for every request<\/p>\n<ol>\n  <li>Determine the container that needs to be spawned using the task definition<\/li>\n  <li>Create a SAS for the task executable<\/li>\n  <li>Spawn the docker container<\/li>\n<\/ol>\n\n<p>The task orchestrator will contain one POST endpoint that will be used to perform the above-mentioned tasks. The request body will contain the name of the task and the input parameters for the task as shown below:<\/p>\n\n<div class=\"language-json highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"p\">{<\/span><span class=\"w\">\n  <\/span><span class=\"nl\">\"taskName\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"StopVM\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n  <\/span><span class=\"nl\">\"parameters\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"w\">\n    <\/span><span class=\"p\">{<\/span><span class=\"w\">\n      <\/span><span class=\"nl\">\"name\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"VMName\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n      <\/span><span class=\"nl\">\"value\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"a3etest\"<\/span><span class=\"w\">\n    <\/span><span class=\"p\">},<\/span><span class=\"w\">\n    <\/span><span class=\"p\">{<\/span><span class=\"w\">\n      <\/span><span class=\"nl\">\"name\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"ResourceGroupName\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n      <\/span><span class=\"nl\">\"value\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"rg-a3e-test_resources-dev\"<\/span><span class=\"w\">\n    <\/span><span class=\"p\">}<\/span><span class=\"w\">\n  <\/span><span class=\"p\">]<\/span><span class=\"w\">\n<\/span><span class=\"p\">}<\/span><span class=\"w\">\n<\/span><\/code><\/pre><\/div><\/div>\n\n<h2 id=\"determine-the-container-that-needs-to-be-spawned-using-the-task-definition\">Determine the container that needs to be spawned using the task definition<\/h2>\n\n<p>To determine the container that needs to be spawned we will have to pull the task definition from the storage account.<\/p>\n\n<p>The code for pulling the task definition from storage account is shown below:<\/p>\n\n<div class=\"language-csharp highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"kt\">var<\/span> <span class=\"n\">cloudStorageAccount<\/span> <span class=\"p\">=<\/span> <span class=\"n\">CloudStorageAccount<\/span><span class=\"p\">.<\/span><span class=\"nf\">Parse<\/span><span class=\"p\">(<\/span><span class=\"n\">configuration<\/span><span class=\"p\">[<\/span><span class=\"s\">\"StorageConnetionString\"<\/span><span class=\"p\">])<\/span>\n<span class=\"kt\">var<\/span> <span class=\"n\">blobClient<\/span> <span class=\"p\">=<\/span> <span class=\"n\">cloudStorageAccount<\/span><span class=\"p\">.<\/span><span class=\"nf\">CreateCloudBlobClient<\/span><span class=\"p\">();<\/span>\n<span class=\"kt\">var<\/span> <span class=\"n\">storageContainer<\/span> <span class=\"p\">=<\/span> <span class=\"n\">blobClient<\/span><span class=\"p\">.<\/span><span class=\"nf\">GetContainerReference<\/span><span class=\"p\">(<\/span><span class=\"err\">\u201c<\/span><span class=\"n\">task<\/span><span class=\"p\">-<\/span><span class=\"n\">container<\/span><span class=\"err\">\u201d<\/span><span class=\"p\">);<\/span>\n<span class=\"kt\">var<\/span> <span class=\"n\">blob<\/span> <span class=\"p\">=<\/span> <span class=\"n\">storageContainer<\/span><span class=\"p\">.<\/span><span class=\"nf\">GetBlobReference<\/span><span class=\"p\">(<\/span><span class=\"err\">\u201c<\/span><span class=\"n\">stop<\/span><span class=\"p\">-<\/span><span class=\"n\">vm<\/span><span class=\"p\">.<\/span><span class=\"n\">json<\/span><span class=\"err\">\u201d<\/span><span class=\"p\">);<\/span> \n\n<span class=\"k\">using<\/span> <span class=\"nn\">MemoryStream<\/span> <span class=\"n\">blobContents<\/span> <span class=\"p\">=<\/span> <span class=\"k\">new<\/span> <span class=\"nf\">MemoryStream<\/span><span class=\"p\">();<\/span>\n<span class=\"k\">await<\/span> <span class=\"n\">blob<\/span><span class=\"p\">.<\/span><span class=\"nf\">DownloadToStreamAsync<\/span><span class=\"p\">(<\/span><span class=\"n\">blobContents<\/span><span class=\"p\">);<\/span>\n<span class=\"kt\">string<\/span> <span class=\"n\">jsonTaskDefinition<\/span> <span class=\"p\">=<\/span> <span class=\"n\">Encoding<\/span><span class=\"p\">.<\/span><span class=\"n\">UTF8<\/span><span class=\"p\">.<\/span><span class=\"nf\">GetString<\/span><span class=\"p\">(<\/span><span class=\"n\">blobContents<\/span><span class=\"p\">.<\/span><span class=\"nf\">ToArray<\/span><span class=\"p\">());<\/span>\n\n<span class=\"n\">TaskDefinition<\/span> <span class=\"n\">taskDefinition<\/span> <span class=\"p\">=<\/span> <span class=\"n\">JsonConvert<\/span><span class=\"p\">.<\/span><span class=\"n\">DeserializeObject<\/span><span class=\"p\">&lt;<\/span><span class=\"n\">TaskDefinition<\/span><span class=\"p\">&gt;(<\/span><span class=\"n\">jsonTaskDefinition<\/span><span class=\"p\">);<\/span>\n<span class=\"kt\">string<\/span> <span class=\"n\">containerImage<\/span> <span class=\"p\">=<\/span> <span class=\"n\">taskDefinition<\/span><span class=\"p\">.<\/span><span class=\"n\">DockerImage<\/span>\n<span class=\"kt\">string<\/span> <span class=\"n\">taskExecutableName<\/span> <span class=\"p\">=<\/span> <span class=\"n\">taskDefinition<\/span><span class=\"p\">.<\/span><span class=\"n\">ScriptName<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<h2 id=\"create-a-sas-for-the-task-executable\">Create a SAS for the task executable<\/h2>\n\n<p>In the previous step we found the name of task executable, now let us create a SAS for the executable so that we can pass the URI to task runner so that it can download the executable.<\/p>\n\n<p>The code for generating a SAS is shown below:<\/p>\n\n<div class=\"language-csharp highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"kt\">var<\/span> <span class=\"n\">cloudStorageAccount<\/span> <span class=\"p\">=<\/span> <span class=\"n\">CloudStorageAccount<\/span><span class=\"p\">.<\/span><span class=\"nf\">Parse<\/span><span class=\"p\">(<\/span><span class=\"n\">configuration<\/span><span class=\"p\">[<\/span><span class=\"s\">\"StorageConnetionString\"<\/span><span class=\"p\">])<\/span>\n<span class=\"kt\">var<\/span> <span class=\"n\">blobClient<\/span> <span class=\"p\">=<\/span> <span class=\"n\">cloudStorageAccount<\/span><span class=\"p\">.<\/span><span class=\"nf\">CreateCloudBlobClient<\/span><span class=\"p\">();<\/span>\n<span class=\"kt\">var<\/span> <span class=\"n\">storageContainer<\/span> <span class=\"p\">=<\/span> <span class=\"n\">blobClient<\/span><span class=\"p\">.<\/span><span class=\"nf\">GetContainerReference<\/span><span class=\"p\">(<\/span><span class=\"err\">\u201c<\/span><span class=\"n\">task<\/span><span class=\"p\">-<\/span><span class=\"n\">container<\/span><span class=\"err\">\u201d<\/span><span class=\"p\">);<\/span>\n<span class=\"kt\">var<\/span> <span class=\"n\">blob<\/span> <span class=\"p\">=<\/span> <span class=\"n\">storageContainer<\/span><span class=\"p\">.<\/span><span class=\"nf\">GetBlobReference<\/span><span class=\"p\">(<\/span><span class=\"n\">taskDefinition<\/span><span class=\"p\">.<\/span><span class=\"n\">ScriptName<\/span><span class=\"p\">);<\/span>\n\n<span class=\"n\">SharedAccessBlobPolicy<\/span> <span class=\"n\">sharedAccessBlobPolicy<\/span> <span class=\"p\">=<\/span> <span class=\"k\">new<\/span> <span class=\"n\">SharedAccessBlobPolicy<\/span>\n<span class=\"p\">{<\/span>\n        <span class=\"n\">SharedAccessExpiryTime<\/span> <span class=\"p\">=<\/span> <span class=\"n\">DateTime<\/span><span class=\"p\">.<\/span><span class=\"n\">UtcNow<\/span><span class=\"p\">.<\/span><span class=\"nf\">AddHours<\/span><span class=\"p\">(<\/span><span class=\"m\">1<\/span><span class=\"p\">),<\/span>\n        <span class=\"n\">Permissions<\/span> <span class=\"p\">=<\/span> <span class=\"n\">SharedAccessBlobPermissions<\/span><span class=\"p\">.<\/span><span class=\"n\">Read<\/span>\n<span class=\"p\">};<\/span>\n<span class=\"kt\">var<\/span> <span class=\"n\">blobToken<\/span> <span class=\"p\">=<\/span> <span class=\"n\">blob<\/span><span class=\"p\">.<\/span><span class=\"nf\">GetSharedAccessSignature<\/span><span class=\"p\">(<\/span><span class=\"n\">sharedAccessBlobPolicy<\/span><span class=\"p\">);<\/span>\n<span class=\"kt\">string<\/span> <span class=\"n\">uri<\/span> <span class=\"p\">=<\/span> <span class=\"n\">blob<\/span><span class=\"p\">.<\/span><span class=\"n\">Uri<\/span> <span class=\"p\">+<\/span> <span class=\"n\">blobToken<\/span><span class=\"p\">;<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<h2 id=\"spawn-the-docker-container\">Spawn the docker container<\/h2>\n\n<p>Now that we have a SAS URI for the task executable, let us now spawn a docker container on Azure Container Instances.<\/p>\n\n<p>The code for this is shown below:<\/p>\n\n<div class=\"language-csharp highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"kt\">var<\/span> <span class=\"n\">environmentVariables<\/span> <span class=\"p\">=<\/span> <span class=\"n\">taskDefinition<\/span><span class=\"p\">.<\/span><span class=\"n\">ScriptParameters<\/span>\n                                                <span class=\"p\">.<\/span><span class=\"nf\">ToDictionary<\/span><span class=\"p\">(<\/span><span class=\"n\">x<\/span> <span class=\"p\">=&gt;<\/span> <span class=\"n\">x<\/span><span class=\"p\">.<\/span><span class=\"n\">ParameterName<\/span><span class=\"p\">,<\/span>\n                                                                <span class=\"n\">x<\/span> <span class=\"p\">=&gt;<\/span> <span class=\"n\">task<\/span><span class=\"p\">.<\/span><span class=\"n\">Parameters<\/span><span class=\"p\">.<\/span><span class=\"nf\">FirstOrDefault<\/span><span class=\"p\">(<\/span><span class=\"n\">y<\/span> <span class=\"p\">=&gt;<\/span> <span class=\"n\">x<\/span><span class=\"p\">.<\/span><span class=\"n\">ParameterName<\/span> <span class=\"p\">==<\/span> <span class=\"n\">y<\/span><span class=\"p\">.<\/span><span class=\"n\">Name<\/span><span class=\"p\">)?.<\/span><span class=\"n\">Value<\/span><span class=\"p\">);<\/span>\n\n<span class=\"k\">await<\/span> <span class=\"n\">azure<\/span><span class=\"p\">.<\/span><span class=\"n\">ContainerGroups<\/span>\n<span class=\"p\">.<\/span><span class=\"nf\">Define<\/span><span class=\"p\">(<\/span><span class=\"n\">SdkContext<\/span><span class=\"p\">.<\/span><span class=\"nf\">RandomResourceName<\/span><span class=\"p\">(<\/span><span class=\"n\">configuration<\/span><span class=\"p\">[<\/span><span class=\"s\">\"ContainerGroupNamePrefix\"<\/span><span class=\"p\">],<\/span> <span class=\"m\">20<\/span><span class=\"p\">))<\/span>\n                                <span class=\"p\">.<\/span><span class=\"nf\">WithRegion<\/span><span class=\"p\">(<\/span><span class=\"n\">Region<\/span><span class=\"p\">.<\/span><span class=\"n\">USWest<\/span><span class=\"p\">)<\/span>\n                                <span class=\"p\">.<\/span><span class=\"nf\">WithExistingResourceGroup<\/span><span class=\"p\">(<\/span><span class=\"n\">configuration<\/span><span class=\"p\">[<\/span><span class=\"s\">\"ResourceGroupName\"<\/span><span class=\"p\">])<\/span>\n                                <span class=\"p\">.<\/span><span class=\"nf\">WithLinux<\/span><span class=\"p\">()<\/span>\n                                <span class=\"p\">.<\/span><span class=\"nf\">WithPublicImageRegistryOnly<\/span><span class=\"p\">()<\/span>\n                                <span class=\"p\">.<\/span><span class=\"nf\">WithoutVolume<\/span><span class=\"p\">()<\/span>\n<span class=\"p\">.<\/span><span class=\"nf\">DefineContainerInstance<\/span><span class=\"p\">(<\/span><span class=\"n\">SdkContext<\/span><span class=\"p\">.<\/span><span class=\"nf\">RandomResourceName<\/span><span class=\"p\">(<\/span><span class=\"n\">configuration<\/span><span class=\"p\">[<\/span><span class=\"s\">\"ContainerInstanceNamePrefix\"<\/span><span class=\"p\">],<\/span> <span class=\"m\">20<\/span><span class=\"p\">))<\/span>\n                                    <span class=\"p\">.<\/span><span class=\"nf\">WithImage<\/span><span class=\"p\">(<\/span><span class=\"n\">containerImage<\/span><span class=\"p\">)<\/span>\n                                    <span class=\"p\">.<\/span><span class=\"nf\">WithoutPorts<\/span><span class=\"p\">()<\/span>\n                                    <span class=\"p\">.<\/span><span class=\"nf\">WithCpuCoreCount<\/span><span class=\"p\">(<\/span><span class=\"m\">1<\/span><span class=\"p\">)<\/span>\n                                    <span class=\"p\">.<\/span><span class=\"nf\">WithMemorySizeInGB<\/span><span class=\"p\">(<\/span><span class=\"m\">1<\/span><span class=\"p\">)<\/span>\n                                    <span class=\"p\">.<\/span><span class=\"nf\">WithEnvironmentVariables<\/span><span class=\"p\">(<\/span><span class=\"n\">environmentVariables<\/span><span class=\"p\">)<\/span>\n                                    <span class=\"p\">.<\/span><span class=\"nf\">WithStartingCommandLine<\/span><span class=\"p\">(<\/span><span class=\"s\">\"script-runner.sh\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">scriptLocation<\/span><span class=\"p\">)<\/span>\n                                    <span class=\"p\">.<\/span><span class=\"nf\">Attach<\/span><span class=\"p\">()<\/span>\n                                <span class=\"p\">.<\/span><span class=\"nf\">WithExistingUserAssignedManagedServiceIdentity<\/span><span class=\"p\">(<\/span><span class=\"k\">await<\/span> <span class=\"n\">azure<\/span><span class=\"p\">.<\/span><span class=\"n\">Identities<\/span><span class=\"p\">.<\/span><span class=\"nf\">GetByIdAsync<\/span><span class=\"p\">(<\/span><span class=\"n\">configuration<\/span><span class=\"p\">[<\/span><span class=\"s\">\"UserAssignedManagedServiceIdentityId\"<\/span><span class=\"p\">]))<\/span>\n                                <span class=\"p\">.<\/span><span class=\"nf\">WithRestartPolicy<\/span><span class=\"p\">(<\/span><span class=\"n\">ContainerGroupRestartPolicy<\/span><span class=\"p\">.<\/span><span class=\"n\">Never<\/span><span class=\"p\">)<\/span>\n                                <span class=\"p\">.<\/span><span class=\"nf\">CreateAsync<\/span><span class=\"p\">();<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<h1 id=\"recap\">Recap<\/h1>\n\n<p>In this post we looked at how we can build a consumption-based task execution framework on Azure using Azure Container Instances and Azure Functions.<\/p>\n\n<p>You can view the source for all the components explained at my GitHub repository <a href=\"https:\/\/github.com\/lalitadithya\/automate-anything-and-everything\">here<\/a>&lt;\/a&gt;.<\/p>\n","pubDate":"Mon, 15 Jun 2020 00:00:00 +0000","link":"https:\/\/lalitadithya.com\/blog\/building-a-cloud-native-task-execution-framework-on-azure\/","guid":"https:\/\/lalitadithya.com\/blog\/building-a-cloud-native-task-execution-framework-on-azure\/","category":["automation","azure","azure-container-instances","azure-functions","cloud-native"]},{"title":"How to Save Money by \u201cTurning Off\u201d your Azure PaaS Resources","description":"<p>Azure resources can be broadly divided into the following categories:<\/p>\n<ol>\n  <li>Platform as a Services (PaaS)\n    <ul>\n      <li><strong>Consumption based<\/strong> \u2013 e.g., Azure Functions, Azure Logic Apps, Azure Container Instances etc<\/li>\n      <li><strong>Hour based<\/strong> \u2013 e.g., Azure SQL Server, Azure Database for Postgres, Azure Kubernetes Service etc<\/li>\n    <\/ul>\n  <\/li>\n  <li>Infrastructure as a Service (IaaS) \u2013 e.g., Azure Virtual Machines<\/li>\n  <li>Software as a Service (SaaS) \u2013 e.g., Microsoft 365<\/li>\n<\/ol>\n\n<p>In an ideal world, all our workloads will be entirely consumption based and we only pay for what we use. Often that is not the case our workloads consist of a mix of consumption-based PaaS, hour-based PaaS and sometimes sprinkled with IaaS for good measure. One of the major downsides of the hour-based PaaS is that they do not have a \u201cstop\u201d or a \u201cstart\u201d button like IaaS resources generally do. In this post we are going to be looking at how we can save more than just a few bucks on our Azure bill by turning off the hour-based PaaS services when we are not using them.<\/p>\n\n<!--more-->\n\n<p>A lot of folks save on their Azure bill by opting for the 1-year reserved or the 3-year reserved option for production use, but that kind of commitment might not be needed for dev\/test or side projects. In most cases the only option we have would be to delete the resources after we are done with them and then re-create them when we need them. That is a viable approach, but we lose the \u201cstate\u201d of our resources. \u201cState\u201d of a resource means many things. It can be the tables and\/or stored procedures in case of a database, or it could be deployments in case of Kubernetes.<\/p>\n\n<p>We can export the state before deleting the resource and then delete the resource. When we want to start the resource back up, we can create the resource and import the state back. This is exactly what I used to do.<\/p>\n\n<p>This got me thinking: the delete\/create is an extremely redundant and boring process that I do at least twice a week. Can this be automated? Afterall a wise man once said<\/p>\n\n<blockquote>\n  <p>If you must do something more than twice, stop and automate it<\/p>\n\n  <p>\u2013 <cite>A wise man<\/cite><\/p>\n<\/blockquote>\n\n<h1 id=\"ferb-i-know-what-were-gonna-do-today\">Ferb, I know what we\u2019re gonna do today<\/h1>\n\n<p>We are going to be building two docker containers that will give us the ability to \u201cturn off\u201d PaaS resources when we are not using them.<\/p>\n\n<p>We will be making use Azure Container Instances to run these containers and we will use Azure Logic Apps to trigger the docker containers. We will also be making use of a storage media such as Azure Storage to store state and finally we will leverage Azure\u2019s ARM templates to create our PaaS resources on demand.<\/p>\n\n<h4 id=\"side-note\">Side note<\/h4>\n\n<p>I am taking Azure SQL server as an example for this post, but the method described here is extremely generic and it can be used for any PaaS resource on Azure. I also make use of Azure storage and Azure container registry for storing state as I have those running 24*7, but you can make use of any other storage media like Google Drive or OneDrive and you can make use of Docker Hub for storing your container images if you don\u2019t want to use Azure container registry.<\/p>\n\n<h1 id=\"azure-container-instances\">Azure Container Instances<\/h1>\n\n<p>First, let us talk about Azure\u2019s container instances (ACI). ACI lets you build container-based event driven applications. If you want a short-lived container to run in response to (sporadic) events you can do that with ACI. The icing on the cake is that you will only be billed for the time your container is running. Pretty cool, right?<\/p>\n\n<p>Wait, why do we need ACI? I can do almost anything with Azure Functions and\/or Azure Logic Apps, right?<\/p>\n\n<p>Yes and no.<\/p>\n\n<p>If you just want to destroy\/create resources you can probably do that using the Azure management SDK and Azure functions, but if you want to make use of other tools like sqlcmd, kubectl etc for saving\/restoring your resource\u2019s state you most likely can not use Azure Functions or Azure Logic Apps.<\/p>\n\n<h1 id=\"let-us-drive-right-into-it-shall-we\">Let us drive right into it, shall we?<\/h1>\n\n<p>Let us start by dividing our resources into three separate resources groups, namely:<\/p>\n<ol>\n  <li><strong>rg-acme-rebootable<\/strong> \u2013 This resource group will contain the PaaS resources that we would like to \u201cstart\u201d and \u201cstop\u201d<\/li>\n  <li><strong>rg-acme-non-rebootable<\/strong> \u2013 This resource group will contain all our other resources that we need to keep running for our application and are consumption based<\/li>\n  <li><strong>rg-acme-automation<\/strong> \u2013 This resource group will contain all the resources that we will be using for automation<\/li>\n<\/ol>\n\n<p>If we have an Azure Functions app that connects to an Azure SQL backend. We should put the functions app in the \u201crg-acme-non-rebootable\u201d and the SQL server in the \u201crg-acme-rebootable\u201d resource group.<\/p>\n\n<p>To \u201cstart\u201d and \u201cstop\u201d the SQL server we are going to be making use of PowerShell scripts which will be run inside docker containers on ACI.<\/p>\n\n<p>Let us now look at the scripts.<\/p>\n\n<h2 id=\"stoping-the-sql-database\">\u201cStop\u201ding the SQL database<\/h2>\n\n<p>To stop the database, we can make use of a super cool tool called <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/github.com\/microsoft\/mssql-scripter\" target=\"_blank\">mssql-scripter<\/a>. For those of you who are familiar with SQL server you know that SSMS provides an option called \u201cgenerate scripts\u201d which can be used to export schema and data to an SQL file. mssql-scripter is the command line version of that option.<\/p>\n\n<p>A snippet from the script is shown below. You can find the complete script <a href=\"https:\/\/github.com\/lalitadithya\/AzureAutomation\/blob\/master\/StopDatabase\/stop-database.ps1\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">here<\/a>.<\/p>\n\n<div class=\"language-powershell highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"nf\">Write-Host<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"Attempting Azure login\"<\/span><span class=\"w\">\n<\/span><span class=\"nf\">az<\/span><span class=\"w\"> <\/span><span class=\"nx\">login<\/span><span class=\"w\"> <\/span><span class=\"nt\">--identity<\/span><span class=\"w\">\n<\/span><span class=\"nf\">Write-Host<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"Database export started\"<\/span><span class=\"w\">\n<\/span><span class=\"nf\">mssql-scripter<\/span><span class=\"w\"> <\/span><span class=\"nt\">-S<\/span><span class=\"w\"> <\/span><span class=\"nv\">$DatabaseServerName<\/span><span class=\"w\"> <\/span><span class=\"nt\">-d<\/span><span class=\"w\"> <\/span><span class=\"nv\">$DatabaseName<\/span><span class=\"w\"> <\/span><span class=\"nt\">-U<\/span><span class=\"w\"> <\/span><span class=\"nv\">$AdminUsername<\/span><span class=\"w\"> <\/span><span class=\"nt\">-P<\/span><span class=\"w\"> <\/span><span class=\"nv\">$AdminPassword<\/span><span class=\"w\"> <\/span><span class=\"nt\">--schema<\/span><span class=\"o\">-and<\/span><span class=\"nt\">-data<\/span><span class=\"w\"> <\/span><span class=\"nt\">--target-server-version<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"AzureDB\"<\/span><span class=\"w\"> <\/span><span class=\"nt\">--display-progress<\/span><span class=\"w\"> <\/span><span class=\"err\">&gt;<\/span><span class=\"w\"> <\/span><span class=\"nx\">backup.sql<\/span><span class=\"w\">\n<\/span><span class=\"nf\">Write-Host<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"Uploading script to blob\"<\/span><span class=\"w\">\n<\/span><span class=\"nf\">az<\/span><span class=\"w\"> <\/span><span class=\"nx\">storage<\/span><span class=\"w\"> <\/span><span class=\"nx\">blob<\/span><span class=\"w\"> <\/span><span class=\"nx\">upload<\/span><span class=\"w\"> <\/span><span class=\"nt\">-f<\/span><span class=\"w\"> <\/span><span class=\"nx\">backup.sql<\/span><span class=\"w\"> <\/span><span class=\"nt\">-c<\/span><span class=\"w\"> <\/span><span class=\"nv\">$StorageContainerName<\/span><span class=\"w\"> <\/span><span class=\"nt\">-n<\/span><span class=\"w\"> <\/span><span class=\"nx\">backup.sql<\/span><span class=\"w\"> <\/span><span class=\"nt\">--connection-string<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"<\/span><span class=\"nv\">$StorageConnectionString<\/span><span class=\"s2\">\"<\/span><span class=\"w\">\n<\/span><span class=\"nf\">Write-Host<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"Stopping database in resource group <\/span><span class=\"nv\">$ResourceGroupName<\/span><span class=\"s2\">\"<\/span><span class=\"w\">\n<\/span><span class=\"nf\">az<\/span><span class=\"w\"> <\/span><span class=\"nx\">group<\/span><span class=\"w\"> <\/span><span class=\"nx\">delete<\/span><span class=\"w\"> <\/span><span class=\"nt\">-n<\/span><span class=\"w\"> <\/span><span class=\"nv\">$ResourceGroupName<\/span><span class=\"w\"> <\/span><span class=\"nt\">--yes<\/span><span class=\"w\">\n<\/span><span class=\"nf\">Write-Host<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"Database stopped\"<\/span><span class=\"w\">\n<\/span><\/code><\/pre><\/div><\/div>\n\n<p>The script first logs in to Azure using the \u201caz login\u201d command. It then makes use of the mssql-scripter tool to export the database to an SQL file. After the export is complete, it uploads the file to Azure storage and then it deletes the resource group. You can customize the upload target to any other cloud storage by making use of a utility like <a href=\"https:\/\/rclone.org\/\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">rclone<\/a>.<\/p>\n\n<p>As mentioned earlier, the script will be running inside a docker container. The DockerFile for the container is shown below:<\/p>\n\n<div class=\"language-dockerfile highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"k\">FROM<\/span><span class=\"s\"> ubuntu:18.04<\/span>\n<span class=\"c\"># install azure cli<\/span>\n<span class=\"k\">RUN <\/span>apt-get update  <span class=\"se\">\\\n<\/span>    apt-get <span class=\"nb\">install<\/span> <span class=\"nt\">-y<\/span> ca-certificates curl apt-transport-https lsb-release gnupg  <span class=\"se\">\\\n<\/span>    curl <span class=\"nt\">-sL<\/span> https:\/\/packages.microsoft.com\/keys\/microsoft.asc | gpg <span class=\"nt\">--dearmor<\/span> | <span class=\"nb\">tee<\/span> \/etc\/apt\/trusted.gpg.d\/microsoft.asc.gpg <span class=\"o\">&gt;<\/span> \/dev\/null  <span class=\"se\">\\\n<\/span>    <span class=\"nv\">AZ_REPO<\/span><span class=\"o\">=<\/span><span class=\"si\">$(<\/span>lsb_release <span class=\"nt\">-cs<\/span><span class=\"si\">)<\/span>  <span class=\"se\">\\\n<\/span>    <span class=\"nb\">echo<\/span> <span class=\"s2\">\"deb [arch=amd64] https:\/\/packages.microsoft.com\/repos\/azure-cli\/ <\/span><span class=\"nv\">$AZ_REPO<\/span><span class=\"s2\"> main\"<\/span> | <span class=\"nb\">tee<\/span> \/etc\/apt\/sources.list.d\/azure-cli.list  <span class=\"se\">\\\n<\/span>    apt-get update  <span class=\"se\">\\\n<\/span>    apt-get <span class=\"nb\">install<\/span> <span class=\"nt\">-y<\/span> azure-cli\n<span class=\"c\"># install pwsh<\/span>\n<span class=\"k\">RUN <\/span>apt-get <span class=\"nt\">-y<\/span> <span class=\"nb\">install <\/span>wget software-properties-common  <span class=\"se\">\\\n<\/span>    wget <span class=\"nt\">-q<\/span> https:\/\/packages.microsoft.com\/config\/ubuntu\/18.04\/packages-microsoft-prod.deb  <span class=\"se\">\\\n<\/span>    dpkg <span class=\"nt\">-i<\/span> packages-microsoft-prod.deb  <span class=\"se\">\\\n<\/span>    apt-get update  <span class=\"se\">\\\n<\/span>    add-apt-repository universe  <span class=\"se\">\\\n<\/span>    apt-get <span class=\"nb\">install<\/span> <span class=\"nt\">-y<\/span> powershell\n<span class=\"c\"># install mssql-scripter<\/span>\n<span class=\"k\">RUN <\/span>apt-get <span class=\"nt\">-y<\/span> <span class=\"nb\">install <\/span>python python-pip  <span class=\"se\">\\\n<\/span>    pip <span class=\"nb\">install <\/span>mssql-scripter\n<span class=\"k\">COPY<\/span><span class=\"s\"> stop-database.ps1 \/stop-database.ps1<\/span>\n<span class=\"k\">CMD<\/span><span class=\"s\"> [\"pwsh\" ,\".\\\\stop-database.ps1\", \\<\/span>\n    \"-ResourceGroupName\", \"\", \\\n    \"-DatabaseServerName\", \"\", \\\n    \"-DatabaseName\", \"\", \\\n    \"-AdminPassword\", \"\", \\ \n    \"-AdminUsername\", \"\", \\ \n    \"-StorageConnectionString\", \"\\\"&lt;&lt;connection string goes here&gt;&gt;\\\"\", \\ \n    \"-StorageContainerName\", \"\"]\n<\/code><\/pre><\/div><\/div>\n\n<p>In the DockerFile, we first install Azure CLI, PowerShell and mssql-scripter. The script shown above is copied into the container and the script is executed using the \u201cCMD\u201d command. This container will run the script and then it will terminate.<\/p>\n\n<p>You can find the DockerFile <a rel=\"noreferrer noopener nofollow\" href=\"https:\/\/github.com\/lalitadithya\/AzureAutomation\/blob\/master\/StopDatabase\/DockerFile\" target=\"_blank\">here<\/a>.<\/p>\n\n<h2 id=\"starting-the-database\">\u201cStart\u201ding the database<\/h2>\n\n<p>To start the database, we make use of Azure\u2019s ARM template deployment model to deploy SQL server and then we can make use of Microsoft\u2019s sqlcmd to run the exported SQL file after downloading it from the storage media.<\/p>\n\n<p>A snippet from the script is shown below. You can find the complete version of the script <a href=\"https:\/\/github.com\/lalitadithya\/AzureAutomation\/blob\/master\/StartDatabase\/start-database.ps1\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">here<\/a>.<\/p>\n\n<div class=\"language-powershell highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"nf\">Write-Host<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"Attempting Azure login\"<\/span><span class=\"w\">\n<\/span><span class=\"nf\">az<\/span><span class=\"w\"> <\/span><span class=\"nx\">login<\/span><span class=\"w\"> <\/span><span class=\"nt\">--identity<\/span><span class=\"w\">\n<\/span><span class=\"nf\">Write-Host<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"Fetching ARM template\"<\/span><span class=\"w\">\n<\/span><span class=\"nf\">az<\/span><span class=\"w\"> <\/span><span class=\"nx\">storage<\/span><span class=\"w\"> <\/span><span class=\"nx\">blob<\/span><span class=\"w\"> <\/span><span class=\"nx\">download<\/span><span class=\"w\"> <\/span><span class=\"nt\">-c<\/span><span class=\"w\"> <\/span><span class=\"nv\">$TemplateStorageContainerName<\/span><span class=\"w\"> <\/span><span class=\"nt\">--file<\/span><span class=\"w\"> <\/span><span class=\"nx\">template.json<\/span><span class=\"w\"> <\/span><span class=\"nt\">--name<\/span><span class=\"w\"> <\/span><span class=\"nx\">template.json<\/span><span class=\"w\">  <\/span><span class=\"nt\">--connection-string<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"<\/span><span class=\"nv\">$StorageConnectionString<\/span><span class=\"s2\">\"<\/span><span class=\"w\">\n<\/span><span class=\"nf\">Write-Host<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"Fetching template parameters\"<\/span><span class=\"w\">\n<\/span><span class=\"nf\">az<\/span><span class=\"w\"> <\/span><span class=\"nx\">storage<\/span><span class=\"w\"> <\/span><span class=\"nx\">blob<\/span><span class=\"w\"> <\/span><span class=\"nx\">download<\/span><span class=\"w\"> <\/span><span class=\"nt\">-c<\/span><span class=\"w\"> <\/span><span class=\"nv\">$TemplateStorageContainerName<\/span><span class=\"w\"> <\/span><span class=\"nt\">--file<\/span><span class=\"w\"> <\/span><span class=\"nx\">parameters.json<\/span><span class=\"w\"> <\/span><span class=\"nt\">--name<\/span><span class=\"w\"> <\/span><span class=\"nx\">parameters.json<\/span><span class=\"w\">  <\/span><span class=\"nt\">--connection-string<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"<\/span><span class=\"nv\">$StorageConnectionString<\/span><span class=\"s2\">\"<\/span><span class=\"w\">\n<\/span><span class=\"nf\">Write-Host<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"Creating resource group\"<\/span><span class=\"w\">\n<\/span><span class=\"nf\">az<\/span><span class=\"w\"> <\/span><span class=\"nx\">group<\/span><span class=\"w\"> <\/span><span class=\"nx\">create<\/span><span class=\"w\"> <\/span><span class=\"nt\">--name<\/span><span class=\"w\"> <\/span><span class=\"nv\">$ResourceGroupName<\/span><span class=\"w\"> <\/span><span class=\"nt\">--location<\/span><span class=\"w\"> <\/span><span class=\"nx\">eastus<\/span><span class=\"w\">\n<\/span><span class=\"nf\">Write-Host<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"Database create started\"<\/span><span class=\"w\">\n<\/span><span class=\"nv\">$deploymentName<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"nf\">Get-Date<\/span><span class=\"w\"> <\/span><span class=\"nt\">-Format<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"MMddyyyyHHmmss\"<\/span><span class=\"w\">\n<\/span><span class=\"nf\">az<\/span><span class=\"w\"> <\/span><span class=\"nx\">group<\/span><span class=\"w\"> <\/span><span class=\"nx\">deployment<\/span><span class=\"w\"> <\/span><span class=\"nx\">create<\/span><span class=\"w\"> <\/span><span class=\"nt\">--name<\/span><span class=\"w\"> <\/span><span class=\"nv\">$deploymentName<\/span><span class=\"w\"> <\/span><span class=\"nt\">--resource-group<\/span><span class=\"w\"> <\/span><span class=\"nv\">$ResourceGroupName<\/span><span class=\"w\"> <\/span><span class=\"nt\">--template-file<\/span><span class=\"w\"> <\/span><span class=\"nx\">template.json<\/span><span class=\"w\"> <\/span><span class=\"nt\">--parameters<\/span><span class=\"w\"> <\/span><span class=\"se\">`@<\/span><span class=\"nx\">parameters.json<\/span><span class=\"w\">\n<\/span><span class=\"nf\">Write-Host<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"Database restore started\"<\/span><span class=\"w\">\n<\/span><span class=\"nf\">az<\/span><span class=\"w\"> <\/span><span class=\"nx\">storage<\/span><span class=\"w\"> <\/span><span class=\"nx\">blob<\/span><span class=\"w\"> <\/span><span class=\"nx\">download<\/span><span class=\"w\"> <\/span><span class=\"nt\">-c<\/span><span class=\"w\"> <\/span><span class=\"nv\">$DatabaseStorageContainerName<\/span><span class=\"w\"> <\/span><span class=\"nt\">--file<\/span><span class=\"w\"> <\/span><span class=\"nx\">backup.sql<\/span><span class=\"w\"> <\/span><span class=\"nt\">--name<\/span><span class=\"w\"> <\/span><span class=\"nx\">backup.sql<\/span><span class=\"w\">  <\/span><span class=\"nt\">--connection-string<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"<\/span><span class=\"nv\">$StorageConnectionString<\/span><span class=\"s2\">\"<\/span><span class=\"w\">\n<\/span><span class=\"nv\">$content<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"nf\">Get-Content<\/span><span class=\"w\"> <\/span><span class=\"s1\">'.\\backup.sql'<\/span><span class=\"p\">)<\/span><span class=\"w\">\n<\/span><span class=\"nv\">$content<\/span><span class=\"p\">[<\/span><span class=\"mi\">105<\/span><span class=\"o\">..<\/span><span class=\"p\">(<\/span><span class=\"nv\">$content<\/span><span class=\"o\">.<\/span><span class=\"nf\">Length-3<\/span><span class=\"p\">)]<\/span><span class=\"w\"> <\/span><span class=\"o\">|<\/span><span class=\"w\"> <\/span><span class=\"nf\">Set-Content<\/span><span class=\"w\"> <\/span><span class=\"nx\">backup.sql<\/span><span class=\"w\">\n<\/span><span class=\"nf\">\/opt\/mssql-tools\/bin\/sqlcmd<\/span><span class=\"w\"> <\/span><span class=\"nt\">-U<\/span><span class=\"w\"> <\/span><span class=\"nv\">$AdminUsername<\/span><span class=\"w\"> <\/span><span class=\"nt\">-S<\/span><span class=\"w\"> <\/span><span class=\"nv\">$DatabaseServerName<\/span><span class=\"w\"> <\/span><span class=\"nt\">-P<\/span><span class=\"w\"> <\/span><span class=\"nv\">$AdminPassword<\/span><span class=\"w\"> <\/span><span class=\"nt\">-i<\/span><span class=\"w\"> <\/span><span class=\"s2\">\".\/backup.sql\"<\/span><span class=\"w\"> <\/span><span class=\"nt\">-d<\/span><span class=\"w\"> <\/span><span class=\"nv\">$DatabaseName<\/span><span class=\"w\">\n<\/span><span class=\"nf\">Write-Host<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"Deleting backup\"<\/span><span class=\"w\">\n<\/span><span class=\"nf\">az<\/span><span class=\"w\"> <\/span><span class=\"nx\">storage<\/span><span class=\"w\"> <\/span><span class=\"nx\">blob<\/span><span class=\"w\"> <\/span><span class=\"nx\">delete<\/span><span class=\"w\"> <\/span><span class=\"nt\">-c<\/span><span class=\"w\"> <\/span><span class=\"nv\">$DatabaseStorageContainerName<\/span><span class=\"w\"> <\/span><span class=\"nt\">--name<\/span><span class=\"w\"> <\/span><span class=\"nx\">backup.sql<\/span><span class=\"w\"> <\/span><span class=\"nt\">--connection-string<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"<\/span><span class=\"nv\">$StorageConnectionString<\/span><span class=\"s2\">\"<\/span><span class=\"w\">\n<\/span><\/code><\/pre><\/div><\/div>\n\n<p>The script logs in to Azure using the \u201caz login\u201d command. It creates a resource group for the resources and deploys the SQL server from an ARM template. After the deployment is complete, the script fetches the SQL export from the storage media and makes use of the sqlcmd command to run the script. Once done, the script deletes the SQL file from the storage media. As mentioned previously, you can make use of any cloud storage media of your choice you do not have to stick to Azure Storage.<\/p>\n\n<p>This script will also be run in a Docker container like the previous one. The DockerFile for this one is extremely similar to the previous script and it can be found <a href=\"https:\/\/github.com\/lalitadithya\/AzureAutomation\/blob\/master\/StartDatabase\/DockerFile\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">here<\/a>.<\/p>\n\n<p>Now that we know the scripts that will be used to \u201cstart\u201d and \u201cstop\u201d the SQL server, let us now look at how we can put all of this together.<\/p>\n\n<h2 id=\"putting-it-all-together\">Putting it all together<\/h2>\n\n<p>Push the two docker images to a container registry of your choice and create a two ACI instances from the Azure portal as shown in the screenshots below. Make sure to clear out the \u201cCommand Override\u201d text box in the advanced tab and set the \u201cRestart Policy\u201d to never.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/how-to-save-money-by-turning-off-your-azure-paas-resources\/aci-page1.webp\" data-fancybox=\"\" data-title=\"ACI Page 1\">\n    <img src=\"\/images\/how-to-save-money-by-turning-off-your-azure-paas-resources\/aci-page1.webp\" alt=\"ACI Page 1\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">ACI Page 1<\/figcaption>\n  \n<\/figure>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/how-to-save-money-by-turning-off-your-azure-paas-resources\/aci-page2.webp\" data-fancybox=\"\" data-title=\"ACI Page 2\">\n    <img src=\"\/images\/how-to-save-money-by-turning-off-your-azure-paas-resources\/aci-page2.webp\" alt=\"ACI Page 2\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">ACI Page 2<\/figcaption>\n  \n<\/figure>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/how-to-save-money-by-turning-off-your-azure-paas-resources\/aci-page2.webp\" data-fancybox=\"\" data-title=\"ACI Page 3\">\n    <img src=\"\/images\/how-to-save-money-by-turning-off-your-azure-paas-resources\/aci-page2.webp\" alt=\"ACI Page 3\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">ACI Page 3<\/figcaption>\n  \n<\/figure>\n\n<p>After the ACI instances have been provisioned, assign a system managed identity, and provide it with enough access to create\/destroy your PaaS resources. Restart your ACI containers and verify that the RBAC is working as expected.<\/p>\n\n<p>We now have two containers running on ACI which can \u201cstart\u201d and \u201cstop\u201d our PaaS resources (in my case an SQL server). All we need to do now is to figure out how to trigger the ACI instances.<\/p>\n\n<h3 id=\"enter-azure-logic-apps\">Enter, Azure Logic Apps<\/h3>\n\n<p>Azure Logic Apps will be perfect for our trigger due to the large number of connectors and triggers that it supports. Go ahead and create two new Azure Logic Apps and configure the trigger of your choice (in my case I have chosen the recurrence trigger) and add a step to start the container instance as shown in the screenshot below:<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/how-to-save-money-by-turning-off-your-azure-paas-resources\/la-designer.webp\" data-fancybox=\"\" data-title=\"Azure logic apps designer showing the trigger and step to start the container\">\n    <img src=\"\/images\/how-to-save-money-by-turning-off-your-azure-paas-resources\/la-designer.webp\" alt=\"Azure logic apps designer showing the trigger and step to start the container\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Azure logic apps designer showing the trigger and step to start the container<\/figcaption>\n  \n<\/figure>\n\n<h1 id=\"conclusion\">Conclusion<\/h1>\n\n<p>In this post we had a look at how we can \u201cturn off\u201d Azure PaaS resources which do not have a Microsoft provided start\/stop button. To achieve this, we made use of Azure Container Register (ACI) to run the Docker containers which run PowerShell scripts to save\/restore the \u201cstate\u201d of the PaaS resource and then delete\/create resource. For triggering the ACI, we made use of Azure Logic Apps due to the large number of connectors and triggers that it supports.<\/p>\n","pubDate":"Sun, 24 May 2020 00:00:00 +0000","link":"https:\/\/lalitadithya.com\/blog\/how-to-save-money-by-turning-off-your-azure-paas-resources\/","guid":"https:\/\/lalitadithya.com\/blog\/how-to-save-money-by-turning-off-your-azure-paas-resources\/","category":["automation","azure-container-instances","azure-logic-apps"]},{"title":"Everything I\u2019ve Learnt About Distributed Locking So Far","description":"<p>In this post we are going to be looking at locking in a distributed system. But, first let us go back to the basics and remind ourselves what is a lock is and why we need locks.<\/p>\n\n<!--more-->\n\n<h1 id=\"what-is-a-lock-why-do-we-need-it\">What is a lock? Why do we need it?<\/h1>\n\n<p>In computer science, a lock is a synchronization mechanism that is used to control access to a resource when there are multiple threads of execution. Locks are used to guarantee mutual exclusion, i.e., only one thread of execution can be inside a critical section at any given point in time. At a very high level we make use of locks to achieve correctness in a multithreaded environment.<\/p>\n\n<p>In order to understand locks further let us look at an example: Consider a singly linked list that contains four items, namely \u201ci \u2013 1, \u201ci\u201d, \u201ci + 1\u201d and \u201ci + 2\u201d as shown in the diagram below:<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/everything-ive-learnt-about-distributed-locking-so-far\/locking-stage-1.webp\" data-fancybox=\"\" data-title=\"Linked list with 4 items\">\n    <img src=\"\/images\/everything-ive-learnt-about-distributed-locking-so-far\/locking-stage-1.webp\" alt=\"Linked list with 4 items\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Linked list with 4 items<\/figcaption>\n  \n<\/figure>\n\n<p>This linked list is being shared by two different threads of execution. At time t<sub>0<\/sub>, the first thread wants to remove node \u201ci\u201d, so it changes the \u201cnext\u201d pointer of node \u201ci \u2013 1\u201d to point to node \u201ci +1\u201d as shown in red. At the same time instant, t<sub>0<\/sub>, the second thread of execution wants to remove the node \u201ci + 1\u201d and hence it changes the \u201cnext\u201d pointer of node \u201ci\u201d to point to node \u201ci + 2\u201d as shown in blue.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/everything-ive-learnt-about-distributed-locking-so-far\/locking-stage-2.webp\" data-fancybox=\"\" data-title=\"Two threads attempting to remove two different elements\">\n    <img src=\"\/images\/everything-ive-learnt-about-distributed-locking-so-far\/locking-stage-2.webp\" alt=\"Two threads attempting to remove two different elements\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Two threads attempting to remove two different elements<\/figcaption>\n  \n<\/figure>\n\n<p>At time instant t<sub>1<\/sub> &gt; t<sub>0<\/sub>, the state of the linked list will be:<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/everything-ive-learnt-about-distributed-locking-so-far\/locking-stage-3.webp\" data-fancybox=\"\" data-title=\"State of the linked list after the removal\">\n    <img src=\"\/images\/everything-ive-learnt-about-distributed-locking-so-far\/locking-stage-3.webp\" alt=\"State of the linked list after the removal\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">State of the linked list after the removal<\/figcaption>\n  \n<\/figure>\n\n<p>Even though the remove operation was performed twice on the linked list, we ended up removing only one element. Our desired state of the linked list has not been achieved. This problem is commonly known as a race condition. To solve the race conditions, we need to make use to mutual exclusion (by extension locks) to ensure that simultaneous updates cannot occur.<\/p>\n\n<h1 id=\"distributed-locking\">Distributed Locking<\/h1>\n\n<p>In the previous section we looked at why locks are important to ensure correctness of results. We also looked at an example of a linked list in a multithreaded environment in a single system, now let us look at how can we make use of a locks in a loosely coupled distributed systems to ensure correctness of results. But, first let us remind ourselves of the CAP theorem.<\/p>\n\n<h2 id=\"cap-theorem\">CAP Theorem<\/h2>\n\n<p>When designing distributed services, there are three properties that are commonly desired: consistency, availability, and partition tolerance. It is impossible to achieve all three. [1]<\/p>\n<ol>\n  <li><strong><em>Consistency<\/em><\/strong>: Every read receives the most recent write or an error<\/li>\n  <li><strong><em>Availability<\/em><\/strong>: Every request receives a (non-error) response, without the guarantee that it contains the most recent write<\/li>\n  <li><strong><em>Partition tolerance<\/em><\/strong>: The system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes<\/li>\n<\/ol>\n\n<p>In a distributed system, a network partition is inevitable. Therefore, according to the CAP theorem, we must pick among:<\/p>\n<ol>\n  <li>Cancel the operation and thus decrease the availability but ensure consistency<\/li>\n  <li>Proceed with the operation and thus provide availability but risk inconsistency<\/li>\n<\/ol>\n\n<h3 id=\"are-networks-really-that-unreliable-yes\">Are networks really that unreliable? Yes.<\/h3>\n\n<p>A team from the University of Toronto and Microsoft Research studied the behaviour of network failures in several of Microsoft\u2019s datacentres. They found an average failure rate of 5.2 devices per day and 40.8 links per day with a median time to repair of approximately five minutes (and up to one week). While the researchers note that correlating link failures and communication partitions is challenging, they estimate a median packet loss of 59,000 packets per failure. Perhaps more concerning is their finding that network redundancy improves median traffic by only 43%; that is, network redundancy does not eliminate many common causes of network failure.[2]<\/p>\n\n<p>A joint study between researchers at University of California, San Diego and HP Labs examined the causes and severity of network failures in HP\u2019s managed networks by analysing support ticket data. \u201cConnectivity\u201d-related tickets accounted for 11.4% of support tickets (14% of which were of the highest priority level), with a median incident duration of 2 hours and 45 minutes for the highest priority tickets and a median duration of 4 hours 18 minutes for all priorities. [2]<\/p>\n\n<p>Google\u2019s paper describing the design and operation of Chubby, their distributed lock manager, outlines the root causes of 61 outages over 700 days of operation across several clusters. Of the nine outages that lasted greater than 30 seconds, four were caused by network maintenance and two were caused by \u201csuspected network connectivity problems.\u201d [2]<\/p>\n\n<h3 id=\"can-other-things-go-wrong-in-a-distributed-environment-absolutely\">Can other things go wrong in a distributed environment? Absolutely.<\/h3>\n\n<p>A network partition is only a small part of what can go wrong in a distributed environment. Some of the other things that can go wrong in a distributed environment include:<\/p>\n<ol>\n  <li>A node can crash, or be rebooted by the orchestrator<\/li>\n  <li>The service might become unresponsive for an extended period due to a super long GC pause<\/li>\n  <li>Time synchronization issues<\/li>\n<\/ol>\n\n<h2 id=\"goals\">Goals<\/h2>\n\n<p>We have established that it is not possible for us to build a prefect distributed system and we must pick between availability and consistency. Let us now look at what is the main purpose of a distributed lock.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/everything-ive-learnt-about-distributed-locking-so-far\/goal.webp\" data-fancybox=\"\" data-title=\"Purpose of a distributed lock\">\n    <img src=\"\/images\/everything-ive-learnt-about-distributed-locking-so-far\/goal.webp\" alt=\"Purpose of a distributed lock\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Purpose of a distributed lock<\/figcaption>\n  \n<\/figure>\n\n<p>Both Alice (shown in red) and Bob (shown in blue) want access to Service X and a bunch of other services (not shown), so they first request a lock from one of the many lock service nodes and then they access \u201cService X\u201d using the lock they acquired from the lock service. The lock service will ensure mutual exclusion for Alice and Bob for Service X and all the other services that would like access. Note: all the services must participate in the distributed locking mechanism.<\/p>\n\n<p>Let us now start looking at how we can get distributed lock with a single instance of the lock service and then we can look at how we can horizontally scale the lock service.<\/p>\n\n<h2 id=\"fenced-locks\">Fenced locks<\/h2>\n\n<p>A fenced lock is a lock that includes a fencing token. A fencing token is a monotonically increasing token that increments whenever a client acquires the lock.<\/p>\n\n<p>To understand why fencing tokens are necessary, let us look at what will happen when we do not have fencing tokens in a distributed system:<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/everything-ive-learnt-about-distributed-locking-so-far\/lock-without-fencing.webp\" data-fancybox=\"\" data-title=\"What can go wrong when we do not use a fenced lock?\">\n    <img src=\"\/images\/everything-ive-learnt-about-distributed-locking-so-far\/lock-without-fencing.webp\" alt=\"What can go wrong when we do not use a fenced lock?\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">What can go wrong when we do not use a fenced lock?<\/figcaption>\n  \n<\/figure>\n\n<p>In the above scenario, Bob has acquired the lock, but due to some reason (bug in code, super long GC), Bob stopped responding for an extended period. When Bob was in a \u201chung\u201d state, Bob\u2019s lock has expired due to a timeout (Note: It is always good to have timeout for locks otherwise, a crashed client will lock out every other client). Alice acquired the lock the lock for service \u2018X\u2019 and made a change to service \u2018X\u2019. When Bob is back from the \u201chung\u201d stage, he will now make an unsafe change to service \u2018X\u2019, because he still thinks he has the lock for the service. But, wait why can\u2019t Bob just check if he still has access to the lock just before accessing the service? Yes, he can but that will also not work as Bob can go into an unexpected \u201chung\u201d state at any time (It can be after he has made the check if has the lock).<\/p>\n\n<p>Now, let us look at how we can solve this by making use of fencing tokens.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/everything-ive-learnt-about-distributed-locking-so-far\/lock-with-fencing.webp\" data-fancybox=\"\" data-title=\"Using fencing tokens\">\n    <img src=\"\/images\/everything-ive-learnt-about-distributed-locking-so-far\/lock-with-fencing.webp\" alt=\"Using fencing tokens\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Using fencing tokens<\/figcaption>\n  \n<\/figure>\n\n<p>Now that we have fencing tokens, Bob acquires a lock and he gets a fencing token of 1. Just like in the previous scenario, he goes into a \u201chung\u201d state. While Bob is in this \u201chung\u201d state, his token times out. Alice comes online and acquires the lock. She is given a fencing token of 2. Alice passes her token (i.e., 2) to service \u2018X\u2019 along with her write request. The service stores Alice\u2019s token as the last known token and it honours Alice\u2019s request. Now, when Bob comes back from his \u201chung\u201d state, he attempts to make a write request to service \u2018X\u2019 by passing his fencing token. The service checks the token passed by Bob against its\u2019 last known token, finds out that the Bob holds an old token (remember, fencing tokens are monotonically increasing) and it will not honour Bob\u2019s request.<\/p>\n\n<p>Notice that it is no longer possible to make unsafe changes to service \u2018X\u2019 due to the monotonically increasing property of fencing tokens. And yes, this has made service \u2018X\u2019 linearized.<\/p>\n\n<p>Now that we have a fair idea of a distributed lock, let us now look at how to make our lock service horizontally scalable.<\/p>\n\n<h2 id=\"horizontally-scaling-the-lock-service\">Horizontally scaling the lock service<\/h2>\n\n<p>To horizontally scale the lock service itself we need to ensure that the service is always consistent while tolerating network partitions. This would mean that we are going to be sacrificing the availability of the lock service, i.e., in the face of a network partition, the lock service may decide to stop issuing locks till the partition is resolved. This is done so that we do not end up with a split-brain scenario.<\/p>\n\n<p>Split-brain is a computer term, based on an analogy with the medical Split-brain syndrome. It indicates data or availability inconsistencies originating from the maintenance of two separate data sets with overlap in scope, either because of servers in a network design, or a failure condition based on servers not communicating and synchronizing their data to each other. This last case is also commonly referred to as a network partition.[3]<\/p>\n\n<p>In the case of distributed locking, a split-brain scenario will become a nightmare because it will give two parties a \u201clock\u201d on the same resource. If two parties can hold the same lock, we have fundamentally broken the definition of a lock. To avoid this, we pick consistency over availability.<\/p>\n\n<p>To develop a consistent locking service that can tolerate network partitions each node should maintain the state of the lock along with the fencing tokens and it must achieve consensus using a consensus algorithm. You can either make use of the Raft consensus algorithm or the Paxos consensus algorithm. Raft consensus algorithm is easier to understand and is described below.<\/p>\n\n<h3 id=\"raft-consensus-algorithm\">Raft consensus algorithm<\/h3>\n\n<p>Raft is a consensus algorithm that is designed to be easy to understand. It is equivalent to Paxos in fault-tolerance and performance. The difference is that it is decomposed into relatively independent subproblems, and cleanly addresses all major pieces needed for practical systems. [4]<\/p>\n\n<p>I had planned on explaining the Raft consensus algorithm, but I found this beautiful visualization of the raft consensus algorithm <a href=\"http:\/\/thesecretlivesofdata.com\/raft\/\" rel=\"noreferrer noopener nofollow\">here<\/a>. I highly recommend that you check it out before proceeding. It very clearly explains how the algorithm behaves in a normal situation and during a network partition.<\/p>\n\n<h1 id=\"conclusion\">Conclusion<\/h1>\n\n<p>In this blog post, we had a look at one of the ways in which we can design a horizontally scalable distributed locking system that provides consistency (while sacrificing availability) while tolerating network partitions using fencing tokens and the Raft Consensus algorithm. This exact approach has been implemented by the Hazelcast team in their latest version of their in-memory data grid, please do check that our as well.<\/p>\n\n<h1 id=\"references\">References<\/h1>\n\n<table>\n  <tbody>\n    <tr>\n      <td>[1]<\/td>\n      <td>S. Gilbert and N. Lynch, \u201cBrewer\u2019s conjecture and the feasibility of consistent, available, partition-tolerant web services,\u201d <em>ACM SIGACT News, <\/em>vol. 33, no. 2, p. 51\u201359, 2002.<\/td>\n    <\/tr>\n    <tr>\n      <td>[2]<\/td>\n      <td>[Online]. <a href=\"https:\/\/aphyr.com\/posts\/288-the-network-is-reliable\" rel=\"noreferrer noopener nofollow\">Available<\/a>.<\/td>\n    <\/tr>\n    <tr>\n      <td>[3]<\/td>\n      <td>Wikipedia, \u201cSplit-brain (computing),\u201d Wikipedia, [Online]. <a href=\"https:\/\/en.wikipedia.org\/wiki\/Split-brain_(computing)\" rel=\"noreferrer noopener nofollow\">Available<\/a>.<\/td>\n    <\/tr>\n    <tr>\n      <td>[4]<\/td>\n      <td>\u201cRaft,\u201d [Online]. <a href=\"https:\/\/raft.github.io\/\" rel=\"noreferrer noopener nofollow\">Available<\/a>.<\/td>\n    <\/tr>\n  <\/tbody>\n<\/table>\n\n","pubDate":"Sun, 10 May 2020 00:00:00 +0000","link":"https:\/\/lalitadithya.com\/blog\/everything-ive-learnt-about-distributed-locking-so-far\/","guid":"https:\/\/lalitadithya.com\/blog\/everything-ive-learnt-about-distributed-locking-so-far\/","category":["cap-theorem","concurrency","consensus-algorithm","distributed-systems","fenced-locks"]},{"title":"Confidential Computing: Securing data while it is in use","description":"<p>According to the Confidential Computing Consortium, a Linux Foundation project, computing is moving to span multiple environments, from on premises to public cloud to edge. As companies move to these environments, they need protection controls for sensitive IP and workload data and are increasingly seeking greater assurances and more transparency of these controls. Current approaches address data at rest and in transit; confidential computing will address data in use.<\/p>\n\n<!--more-->\n\n<h1 id=\"confidential-computing\">Confidential computing<\/h1>\n\n<p>Confidential computing is a term defined by the Confidential Computing Consortium (CCC). The Confidential Computing Consortium is a Linux Foundation Project brings together hardware vendors, cloud providers, and software developers to accelerate the adoption of Trusted Execution Environment (TEE) technologies and standards.[1] The founding premiere members include Alibaba, Arm, Google Cloud, Huawei, Intel, Microsoft and Red Hat. General members include Baidu, ByteDance, decentriq, Fortanix, Kindite, Oasis Labs, Swisscom, Tencent and VMware.[2]<\/p>\n\n<blockquote>\n  <p>Confidential computing is defined as the protection of data in use by performing computation in a hardware-based Trusted Execution Environment.<\/p>\n\n  <p>\u2013<cite>Confidential Computing Consortium, \u201cFAQ - Confidential Computing Foundation,\u201d<\/cite><\/p>\n<\/blockquote>\n\n<p>Let us take a brief detour to understand Trusted Execution Environment (TEE).<\/p>\n\n<h2 id=\"trusted-execution-environment\">Trusted Execution Environment<\/h2>\n\n<p>TEEs are built on top of separation kernels.<\/p>\n\n<blockquote>\n  <p>A separation kernel is defined as \u201chardware and\/or firmware and\/or software mechanisms whose primary function is to establish, isolate and separate multiple partitions and control information flow between the subjects and exported resources allocated to those partitions\u201d<\/p>\n\n  <p>\u2013<cite>Information Assurance Directorate of the US National Security Agency (NSA) in the publication titled \u201cU.S. Government Protection Profile for\u00a0Separation Kernels in Environments Requiring High Robustness\u201d<\/cite><\/p>\n<\/blockquote>\n\n<p>At its core, the separation kernel should provide:<\/p>\n<ol>\n  <li>Protection of all resources (including CPU, memory and devices) from unauthorized access<\/li>\n  <li>Partitioning and isolation of exported resources<\/li>\n  <li>Mediation of information flows between partitions and between exported resources<\/li>\n<\/ol>\n\n<p>The separation kernel is what provides us with two isolated environments, i.e., the secure environment (TEE) and the real world (REE). It controls the information flow between the two environments and protects all resources in the secure environment.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/confidential-computing-securing-data-while-it-is-in-use\/tee-vs-ree.webp\" data-fancybox=\"\" data-title=\"Figure showing TEE, REE and the separation\">\n    <img src=\"\/images\/confidential-computing-securing-data-while-it-is-in-use\/tee-vs-ree.webp\" alt=\"Figure showing TEE, REE and the separation\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Figure showing TEE, REE and the separation<\/figcaption>\n  \n<\/figure>\n\n<p>Now that we know what a separation kernel is, let us look at TEEs. There are several definitions for TEEs that are available on the internet, the most complete among them is given below:<\/p>\n\n<blockquote>\n  <p>Trusted Execution Environment (TEE) is a tamper resistant processing environment that runs on a separation kernel. It guarantees the authenticity of the executed code, the integrity of the runtime states (e.g. CPU registers, memory and sensitive I\/O), and the confidentiality of its code, data and runtime states stored on a persistent memory. In addition, it shall be able to provide remote attestation that proves its trustworthiness for third parties. The content of TEE is not static; it can be securely updated. The TEE resists against all software attacks as well as the physical attacks performed on the main memory of the system. Attacks performed by exploiting backdoor security flaws are not possible.<\/p>\n\n  <p>\u2013<cite>M. Sabt, M. Achemlal and A. Bouabdallah in their paper titled \u201cTrusted Execution Environment: What It is, and What It is Not\u201d<\/cite><\/p>\n<\/blockquote>\n\n<p>At the core of any system that offers security is trust - Users must trust the system enough to use it, Engineers must trust the system enough to build applications around it.<\/p>\n\n<h3 id=\"trust\">Trust<\/h3>\n\n<p>In an ideal world, you would probably find a lab that is certified by other labs (or maybe even a Government agency) who can build and test a TEE. You would then take that TEE to a device manufacturer (that you trust) who can reproduce the sample TEE in production devices. But, the world we live in very far from ideal. Moreover, if one attempts the solution proposed above it will become very expensive, and hence it would become a limiting factor.<\/p>\n\n<p>In the real world, we trust the manufacturer of the TEE and we make use of lot and lots of signature checks to establish trust during run time.<\/p>\n\n<p>TEE establishes trust during the boot process using \u201chardware chain of trust\u201d. In \u201chardware chain of trust\u201d every image that is loaded and executed is cryptographically verified by the image loaded in the previous stage.<\/p>\n\n<p>The first image that is loaded and executed (also known as the primary boot loader) is loaded from immutable read-only memory. This immutable read-only memory is burned into the chip by the manufacturer of the chip and it cannot be altered physically or otherwise.<\/p>\n\n<p>The primary boot loader cryptographically verifies the signature of the next image that needs to be loaded and executed and this chain continues. All the signature checks are done using a set of private keys that are embedded directly onto the chip during the manufacturing process (hence the name \u201chardware chain of trust\u201d). These private keys are stored using read only memory and they cannot be altered physically or otherwise. If during some stage of the boot, a signature check fails, then the TEE will not offer the services for which the signature check failed.<\/p>\n\n<p>Once the boot is complete, the TEE can prove its validity by using a process called attestation. After the TEE has proved its validity, we trust the secure firmware that is loaded to execute trusted applications (TAs) in isolation, i.e., one TA can not affect other TAs that are running.<\/p>\n\n<p>TEEs rely on the underlying separation kernel to prevent tampering of the executing TAs. The separation kernel employs various hardware-based mechanisms to protect the trusted applications from tampering. It prevents all resources (including CPU, memory and devices) from unauthorized access while ensuring partitioning and isolation of the exported resources and mediating the information flows between secure word (TEE) and the real world (REE).<\/p>\n\n<p>Now that have a fair understanding of TEEs, let us jump back into confidential computing.<\/p>\n\n<h1 id=\"applications-of-confidential-computing-available-right-now\">Applications of Confidential Computing available right now<\/h1>\n\n<p>Some of the many applications of confidential computing that you can try out right now include:<\/p>\n<ol>\n  <li>Multiparty machine learning<\/li>\n  <li>Trusted networks<\/li>\n  <li>SQL Server Always Encrypted Technology<\/li>\n<\/ol>\n\n<h2 id=\"multiparty-machine-learning\">Multiparty machine learning<\/h2>\n\n<p>In many cases, multiple parties will benefit from sharing of data, but currently they are not able to share data while preserving privacy.<\/p>\n\n<p>Consider the example of hospitals that are using machine learning models to diagnose diseases early. Every hospital will have its own silo of data and it will train and run machine models on that silo. We know that the accuracy of any machine learning model depends entirely on the amount and accuracy of the data that is fed to it. With confidential computing, all hospitals can confidentially run a machine learning model on the combined dataset, thereby improving the model itself and benefiting all hospitals and by extension their patients.<\/p>\n\n<p>Similarly, if a bank is using machine learning models to detect fraud and money laundering; multiple banks can make use of confidential computing to build a combined machine learning model benefiting everyone.<\/p>\n\n<p>Olga Ohrimenko, Felix Schuster, Cedric Fournet, Aastha Mehta, Sebastian Nowozin, Kapil Vaswani and Manuel Costa published a paper titled \u201c<strong>Oblivious Multi-Party Machine Learning on Trusted Processors<\/strong>\u201d where they propose data oblivious machine learning algorithms for support vector machines, matrix factorization, neural networks, decision trees, and k-means clustering.<\/p>\n\n<h2 id=\"trusted-networks\">Trusted networks<\/h2>\n\n<p>Another application of confidential computing is the creation of trusted networks. The Confidential Consortium Framework (CCF), a joint project with Azure Engineering, is an open-source framework for building a new category of secure, highly available, and performant applications that focus on multi-party compute and data. While not limited just to blockchain applications, CCF can enable high-scale, confidential blockchain networks that meet key enterprise requirements \u2014 providing a means to accelerate production enterprise adoption of blockchain technology. [3]<\/p>\n\n<p>CCF enables enterprise-ready computation or blockchain networks that deliver[3]:<\/p>\n<ol>\n  <li>Throughput and latency approaching database speeds<\/li>\n  <li>Richer, more flexible confidentiality models<\/li>\n  <li>Network and service policy management through non-centralized governance<\/li>\n  <li>Improved efficiency versus traditional blockchain networks<\/li>\n<\/ol>\n\n<p>You can read more about CCF in the paper titled \u201c<strong>CCF: A Framework for Building Confidential Verifiable Replicated Services<\/strong>\u201d by Mark Russinovich, Edward Ashton, Christine Avanessians, Miguel Castro, Amaury Chamayou, Sylvan Clebsch, Manuel Costa, Cedric Fournet, Matthew Kerner, Sid Krishna, Julien Maffre, Thomas Moscibroda, Kartik Nayak, Olga Ohrimenko, Felix Schuster, Roy Schuster, Alex Shamis, Olga Vrousgou and Christoph M. Wintersteiger.<\/p>\n\n<p>Code for CCF can be found on GitHub <a href=\"https:\/\/github.com\/Microsoft\/CCF\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">here<\/a><\/p>\n\n<h2 id=\"sql-server-always-encrypted-technology\">SQL Server Always Encrypted Technology<\/h2>\n\n<p>People familiar with SQL Server would have heard of the \u201cAlways Encrypted\u201d technology that was introduced with SQL Server 2016. This technology allowed users to control the access of encrypted data, but, it had some limitations. Some of the limitations include: the only operations that SQL server could perform on a \u201cAlways Encrypted\u201d column was equality comparisons (only available with deterministic encryption). All the other operations must be performed on the client side and not on the database server. Say, if the user wanted to perform pattern matching, the user would have to move all the data outside the database and then perform these operations on the client side.<\/p>\n\n<p>With SQL server 2019 and later, Microsoft announced the \u201cAlways Encrypted with Secure Enclaves\u201d. This makes use of confidential computing to address the limitations described above. Read more about it <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/docs.microsoft.com\/en-us\/sql\/relational-databases\/security\/encryption\/always-encrypted-enclaves?view=sql-server-ver15\" target=\"_blank\">here<\/a>.<\/p>\n\n<h1 id=\"who-is-offering-confidential-computing\">Who is offering Confidential Computing?<\/h1>\n\n<p>Earlier this week, Mark Russinovich from Microsoft announced the general availability of the DCsv2-series VMs that has support for confidential computing. You can read the announcement <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/azure.microsoft.com\/en-us\/blog\/dcsv2series-vm-now-generally-available-from-azure-confidential-computing\/\">here<\/a>. IBM also has a similar offering that you can read about <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/www.ibm.com\/cloud\/blog\/data-use-protection-ibm-cloud-using-intel-sgx\">here<\/a>.<\/p>\n\n<h1 id=\"how-do-i-develop-trusted-apps-that-make-use-of-confidential-computing\">How do I develop trusted apps that make use of Confidential Computing?<\/h1>\n\n<p>Microsoft and Google are offering SDKs that help developers get started with developing application that make use of confidential computing.<\/p>\n\n<p>You can check out the SDKs <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/openenclave.io\/sdk\/\" target=\"_blank\">here<\/a> and <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/asylo.dev\/\" target=\"_blank\">here<\/a>.<\/p>\n\n<h1 id=\"conclusion\">Conclusion<\/h1>\n\n<p>With more and more confidential workloads moving to the public cloud and edge devices, it is becoming more and more important to secure data not only while it is in rest or transit, but also while it is in use. At the time of writing this article, confidential computing is gaining momentum. It shows a lot of promise for securing data while in use and has a tremendous potential to do good. \u00a0<\/p>\n\n<h1 id=\"references\">References<\/h1>\n\n<table>\n  <tbody>\n    <tr>\n      <td>[1]<\/td>\n      <td>Confidential Computing Consortium, \u201cFAQ - Confidential Computing Foundation,\u201d [Online]. <a href=\"https:\/\/confidentialcomputing.io\/faq\/\" rel=\"noreferrer noopener nofollow\">Available<\/a>.<\/td>\n    <\/tr>\n    <tr>\n      <td>[2]<\/td>\n      <td>Linux Foundation, \u201cConfidential Computing Consortium Establishes Formation with Founding Members and Open Governance Structure,\u201d [Online]. <a href=\"https:\/\/www.linuxfoundation.org\/uncategorized\/2019\/10\/confidential-computing-consortium-establishes-formation-with-founding-members-and-open-governance-structure\/\" rel=\"noreferrer noopener nofollow\">Available<\/a>.<\/td>\n    <\/tr>\n    <tr>\n      <td>[3]<\/td>\n      <td>Microsoft, \u201cConfidential Consortium Framework,\u201d [Online]. <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/project\/confidential-consortium-framework\/\" rel=\"noreferrer noopener nofollow\">Available<\/a>.<\/td>\n    <\/tr>\n  <\/tbody>\n<\/table>\n","pubDate":"Fri, 01 May 2020 00:00:00 +0000","link":"https:\/\/lalitadithya.com\/blog\/confidential-computing-securing-data-while-it-is-in-use\/","guid":"https:\/\/lalitadithya.com\/blog\/confidential-computing-securing-data-while-it-is-in-use\/","category":["cloud-computing","confidential-computing","tee","security","multiparty-machine-learning","trusted-execution-environment"]},{"title":"Head First: HTTP Routing in Kubernetes using Istio","description":"<p>In this blog post, we are going to look at how to use path-based HTTP routing in Kubernetes using a service mesh called Istio. I am going to be using Azure for this blog, but you should be able to customize the steps for any cloud provider easily.<\/p>\n\n<!--more-->\n\n<h1 id=\"what-is-a-service-mesh\">What is a service mesh?<\/h1>\n\n<p>A service mesh ensures that communication among containerized and often ephemeral application infrastructure services is fast, reliable, and secure. The mesh provides critical capabilities including service discovery, load balancing, encryption, observability, traceability, authentication and authorization, and support for the circuit breaker pattern. [1]<\/p>\n\n<h1 id=\"what-is-istio\">What is Istio?<\/h1>\n\n<p>Istio, backed by Google, IBM, and Lyft, is currently the best\u2011known service mesh architecture. Kubernetes, which was originally designed by Google, is currently the only container orchestration framework supported by Istio. [1]<\/p>\n\n<h1 id=\"let-us-dive-in-head-first\">Let us dive in head first!<\/h1>\n\n<p>The Acme Corp (<a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/github.com\/theacmecorp\">GitHub<\/a>) is a multi-national service aggregation company that has offices across the globe. They would like to convert their existing monolithic application to microservices.<\/p>\n\n<p>The architects have identified two microservices that they would like to start with, namely, the books aggregation microservice and the vinyl album aggregation microservice. The high-level architecture diagram for what we are going to be building is shown below:<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/head-first-http-routing-in-kubernetes-using-istio\/high-level-architecture.webp\" data-fancybox=\"\" data-title=\"High level architecture diagram\">\n    <img src=\"\/images\/head-first-http-routing-in-kubernetes-using-istio\/high-level-architecture.webp\" alt=\"High level architecture diagram\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">High level architecture diagram<\/figcaption>\n  \n<\/figure>\n\n<p>The developers have converted the monolithic application into two containerized application and they have pushed the images to docker hub (<a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/hub.docker.com\/r\/lalitadithya\/acmecorpvinylproductcalalogueapi\">Vinyl Microservice<\/a> and <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/hub.docker.com\/r\/lalitadithya\/acmecorpbookcatalogueapi\">Book Microservice<\/a>).<\/p>\n\n<p>The book microservice will respond to HTTP requests that have the URL prefix \u201c\/BookCatalogue\u201d and the vinyl microservice will respond to the \u201c\/VinylProductCatalogue\u201d URL prefix.<\/p>\n\n<p>Let us start by creating:<\/p>\n\n<ol>\n  <li>2 SQL databases. One for the vinyl microservice and other for the book microservice\n    <ul>\n      <li>Run the migration for vinyl database located <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/raw.githubusercontent.com\/lalitadithya\/Scripts\/master\/Blog\/K8%20Routing\/vinyl-database-migration.sql\">here<\/a><\/li>\n      <li>Run the migration for the book database located <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/raw.githubusercontent.com\/lalitadithya\/Scripts\/master\/Blog\/K8%20Routing\/book-database-migration.sql\">here<\/a><\/li>\n      <li>Be sure to note down the connection string for the two databases<\/li>\n    <\/ul>\n  <\/li>\n  <li>1 AKS cluster (You will need at least 2 vCPU and 7 GB of RAM)\n    <ul>\n      <li>Make sure that the AKS cluster can access the databases that you created above<\/li>\n    <\/ul>\n  <\/li>\n<\/ol>\n\n<p>Once all the resources have been successfully provisioned, let us start installing Istio<\/p>\n\n<h2 id=\"installing-istio\">Installing Istio<\/h2>\n\n<p>Fire up the Azure Cloud Shell and fetch credentials for Kubernetes cluster by running the following command<\/p>\n\n<div class=\"language-powershell highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"nf\">az<\/span><span class=\"w\"> <\/span><span class=\"nx\">aks<\/span><span class=\"w\"> <\/span><span class=\"nx\">get-credentials<\/span><span class=\"w\"> <\/span><span class=\"nt\">--resource-group<\/span><span class=\"w\"> <\/span><span class=\"err\">&lt;&lt;<\/span><span class=\"nx\">name<\/span><span class=\"w\"> <\/span><span class=\"nx\">of<\/span><span class=\"w\"> <\/span><span class=\"nx\">your<\/span><span class=\"w\"> <\/span><span class=\"nx\">resource<\/span><span class=\"w\"> <\/span><span class=\"nx\">group<\/span><span class=\"err\">&gt;&gt;<\/span><span class=\"w\">\n                       <\/span><span class=\"nt\">--name<\/span><span class=\"w\"> <\/span><span class=\"err\">&lt;&lt;<\/span><span class=\"nf\">name<\/span><span class=\"w\"> <\/span><span class=\"nx\">of<\/span><span class=\"w\"> <\/span><span class=\"nx\">your<\/span><span class=\"w\"> <\/span><span class=\"nx\">aks<\/span><span class=\"w\"> <\/span><span class=\"nx\">cluster<\/span><span class=\"err\">&gt;&gt;<\/span><span class=\"w\">\n<\/span><\/code><\/pre><\/div><\/div>\n\n<p>Now, create a namespace for Istio by running the following command<\/p>\n\n<div class=\"language-powershell highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"nf\">kubectl<\/span><span class=\"w\"> <\/span><span class=\"nx\">create<\/span><span class=\"w\"> <\/span><span class=\"nx\">namespace<\/span><span class=\"w\"> <\/span><span class=\"nx\">istio-system<\/span><span class=\"w\"> <\/span><span class=\"nt\">--save-config<\/span><span class=\"w\">\n<\/span><\/code><\/pre><\/div><\/div>\n\n<p>In order to install Istio, we will be using the YAML file shown below:<\/p>\n\n<div class=\"language-yaml highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"na\">apiVersion<\/span><span class=\"pi\">:<\/span> <span class=\"s\">install.istio.io\/v1alpha2<\/span>\n<span class=\"na\">kind<\/span><span class=\"pi\">:<\/span> <span class=\"s\">IstioControlPlane<\/span>\n<span class=\"na\">spec<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">profile<\/span><span class=\"pi\">:<\/span> <span class=\"s\">default<\/span>\n  <span class=\"na\">values<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">global<\/span><span class=\"pi\">:<\/span>\n      <span class=\"na\">defaultNodeSelector<\/span><span class=\"pi\">:<\/span>\n        <span class=\"s\">beta.kubernetes.io\/os<\/span><span class=\"pi\">:<\/span> <span class=\"s\">linux<\/span>\n      <span class=\"na\">controlPlaneSecurityEnabled<\/span><span class=\"pi\">:<\/span> <span class=\"no\">true<\/span>\n      <span class=\"na\">mtls<\/span><span class=\"pi\">:<\/span>\n        <span class=\"na\">enabled<\/span><span class=\"pi\">:<\/span> <span class=\"no\">false<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p><strong>Line 4<\/strong>: This will install Istio in the default profile. In the default profile we will get: an ingress gateway (used to route traffic), Istio pilot (used for traffic management) and Prometheus\n<strong>Line 8<\/strong>: This will ensure that Istio pods will only run on Linux nodes.\n<strong>Line 9<\/strong>: We enable mutual TLS for the control plane.\n<strong>Line 11<\/strong>: We disable mutual TLS for all services in the data plane<\/p>\n\n<p>Create this file in the could shell and call it \u201c<strong>istio.aks.yaml<\/strong>\u201d and then install Istio by running the following command:<\/p>\n\n<div class=\"language-powershell highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"nf\">istioctl<\/span><span class=\"w\"> <\/span><span class=\"nx\">manifest<\/span><span class=\"w\"> <\/span><span class=\"nx\">apply<\/span><span class=\"w\"> <\/span><span class=\"nt\">-f<\/span><span class=\"w\"> <\/span><span class=\"nx\">istio.aks.yaml<\/span><span class=\"w\"> <\/span><span class=\"err\">\u2013<\/span><span class=\"nx\">logtostderr<\/span><span class=\"w\">\n<\/span><\/code><\/pre><\/div><\/div>\n\n<p>After the install is complete, you should be seeing the following output<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/head-first-http-routing-in-kubernetes-using-istio\/install-istio.webp\" data-fancybox=\"\" data-title=\"Output after installing Istio\">\n    <img src=\"\/images\/head-first-http-routing-in-kubernetes-using-istio\/install-istio.webp\" alt=\"Output after installing Istio\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Output after installing Istio<\/figcaption>\n  \n<\/figure>\n\n<p>In order to use Istio, we need to enable Istio injection for the namespace that we plan to use. For this demo, we are going to be using the default namespace, so, let us enable Istio injection by running the following command:<\/p>\n\n<div class=\"language-powershell highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"nf\">kubectl<\/span><span class=\"w\"> <\/span><span class=\"nx\">label<\/span><span class=\"w\"> <\/span><span class=\"nx\">namespace<\/span><span class=\"w\"> <\/span><span class=\"nx\">default<\/span><span class=\"w\"> <\/span><span class=\"nx\">istio-injection<\/span><span class=\"o\">=<\/span><span class=\"nf\">enabled<\/span><span class=\"w\">\n<\/span><\/code><\/pre><\/div><\/div>\n\n<h2 id=\"deploy-the-vinyl-microservice\">Deploy the Vinyl Microservice<\/h2>\n\n<p>In this section we will be deploying the vinyl microservice using the YAML file below:<\/p>\n\n<div class=\"language-yaml highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"na\">apiVersion<\/span><span class=\"pi\">:<\/span> <span class=\"s\">apps\/v1<\/span>\n<span class=\"na\">kind<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Deployment<\/span>\n<span class=\"na\">metadata<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">vinyl-deployment<\/span>\n  <span class=\"na\">labels<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">app<\/span><span class=\"pi\">:<\/span> <span class=\"s\">vinyl-api<\/span>\n<span class=\"na\">spec<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">replicas<\/span><span class=\"pi\">:<\/span> <span class=\"m\">1<\/span>\n  <span class=\"na\">selector<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">matchLabels<\/span><span class=\"pi\">:<\/span>\n      <span class=\"na\">app<\/span><span class=\"pi\">:<\/span> <span class=\"s\">vinyl-api<\/span>\n  <span class=\"na\">template<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">metadata<\/span><span class=\"pi\">:<\/span>\n      <span class=\"na\">labels<\/span><span class=\"pi\">:<\/span>\n        <span class=\"na\">app<\/span><span class=\"pi\">:<\/span> <span class=\"s\">vinyl-api<\/span>\n    <span class=\"na\">spec<\/span><span class=\"pi\">:<\/span>\n      <span class=\"na\">containers<\/span><span class=\"pi\">:<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">vinyl-api<\/span>\n        <span class=\"na\">image<\/span><span class=\"pi\">:<\/span> <span class=\"s\">lalitadithya\/acmecorpvinylproductcalalogueapi:latest<\/span>\n        <span class=\"na\">ports<\/span><span class=\"pi\">:<\/span>\n        <span class=\"pi\">-<\/span> <span class=\"na\">containerPort<\/span><span class=\"pi\">:<\/span> <span class=\"m\">80<\/span>\n        <span class=\"na\">env<\/span><span class=\"pi\">:<\/span>\n        <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">ConnectionString<\/span>\n          <span class=\"na\">value<\/span><span class=\"pi\">:<\/span> <span class=\"s2\">\"<\/span><span class=\"s\">&lt;&lt;Insert<\/span><span class=\"nv\"> <\/span><span class=\"s\">your<\/span><span class=\"nv\"> <\/span><span class=\"s\">connection<\/span><span class=\"nv\"> <\/span><span class=\"s\">string<\/span><span class=\"nv\"> <\/span><span class=\"s\">here&gt;&gt;\"<\/span>\n<span class=\"nn\">---<\/span>\n<span class=\"na\">apiVersion<\/span><span class=\"pi\">:<\/span> <span class=\"s\">v1<\/span>\n<span class=\"na\">kind<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Service<\/span>\n<span class=\"na\">metadata<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">vinyl-api-service<\/span>\n  <span class=\"na\">labels<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">app<\/span><span class=\"pi\">:<\/span> <span class=\"s\">vinyl-api<\/span>\n<span class=\"na\">spec<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">ports<\/span><span class=\"pi\">:<\/span>\n  <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">vinyl-api-inbound<\/span>\n    <span class=\"na\">port<\/span><span class=\"pi\">:<\/span> <span class=\"m\">8000<\/span>\n    <span class=\"na\">targetPort<\/span><span class=\"pi\">:<\/span> <span class=\"m\">80<\/span>\n  <span class=\"na\">selector<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">app<\/span><span class=\"pi\">:<\/span> <span class=\"s\">vinyl-api<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>This YAML file tells Kubernetes to create a service called \u201cvinyl-api-service\u201d that will be listening on port 8000. The image for this service will be pulled from docker hub. Be sure to update the connection string in the YAML file.<\/p>\n\n<p>Create the above YAML file in the cloud shell and call it \u201c<strong>deploy-vinyl.yaml<\/strong>\u201d and deploy the microservice by running the following command:<\/p>\n\n<div class=\"language-powershell highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"nf\">kubectl<\/span><span class=\"w\"> <\/span><span class=\"nx\">apply<\/span><span class=\"w\"> <\/span><span class=\"nt\">-f<\/span><span class=\"w\"> <\/span><span class=\"o\">.<\/span><span class=\"nx\">\/deploy-vinyl.yaml<\/span><span class=\"w\">\n<\/span><\/code><\/pre><\/div><\/div>\n\n<h2 id=\"deploy-the-book-microservice\">Deploy the book microservice<\/h2>\n\n<p>In this section we will be deploying the book microservice using the YAML file below:<\/p>\n\n<div class=\"language-yaml highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"na\">apiVersion<\/span><span class=\"pi\">:<\/span> <span class=\"s\">apps\/v1<\/span>\n<span class=\"na\">kind<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Deployment<\/span>\n<span class=\"na\">metadata<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">book-deployment<\/span>\n  <span class=\"na\">labels<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">app<\/span><span class=\"pi\">:<\/span> <span class=\"s\">book-api<\/span>\n<span class=\"na\">spec<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">replicas<\/span><span class=\"pi\">:<\/span> <span class=\"m\">1<\/span>\n  <span class=\"na\">selector<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">matchLabels<\/span><span class=\"pi\">:<\/span>\n      <span class=\"na\">app<\/span><span class=\"pi\">:<\/span> <span class=\"s\">book-api<\/span>\n  <span class=\"na\">template<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">metadata<\/span><span class=\"pi\">:<\/span>\n      <span class=\"na\">labels<\/span><span class=\"pi\">:<\/span>\n        <span class=\"na\">app<\/span><span class=\"pi\">:<\/span> <span class=\"s\">book-api<\/span>\n    <span class=\"na\">spec<\/span><span class=\"pi\">:<\/span>\n      <span class=\"na\">containers<\/span><span class=\"pi\">:<\/span>\n      <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">book-api<\/span>\n        <span class=\"na\">image<\/span><span class=\"pi\">:<\/span> <span class=\"s\">lalitadithya\/acmecorpbookcatalogueapi:latest<\/span>\n        <span class=\"na\">ports<\/span><span class=\"pi\">:<\/span>\n        <span class=\"pi\">-<\/span> <span class=\"na\">containerPort<\/span><span class=\"pi\">:<\/span> <span class=\"m\">80<\/span>\n        <span class=\"na\">env<\/span><span class=\"pi\">:<\/span>\n        <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">ConnectionString<\/span>\n          <span class=\"na\">value<\/span><span class=\"pi\">:<\/span> <span class=\"s2\">\"<\/span><span class=\"s\">&lt;&lt;Insert<\/span><span class=\"nv\"> <\/span><span class=\"s\">your<\/span><span class=\"nv\"> <\/span><span class=\"s\">connection<\/span><span class=\"nv\"> <\/span><span class=\"s\">string<\/span><span class=\"nv\"> <\/span><span class=\"s\">here&gt;&gt;\"<\/span>\n<span class=\"nn\">---<\/span>\n<span class=\"na\">apiVersion<\/span><span class=\"pi\">:<\/span> <span class=\"s\">v1<\/span>\n<span class=\"na\">kind<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Service<\/span>\n<span class=\"na\">metadata<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">book-api-service<\/span>\n  <span class=\"na\">labels<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">app<\/span><span class=\"pi\">:<\/span> <span class=\"s\">book-api<\/span>\n<span class=\"na\">spec<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">ports<\/span><span class=\"pi\">:<\/span>\n  <span class=\"pi\">-<\/span> <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">book-api-inbound<\/span>\n    <span class=\"na\">port<\/span><span class=\"pi\">:<\/span> <span class=\"m\">9000<\/span>\n    <span class=\"na\">targetPort<\/span><span class=\"pi\">:<\/span> <span class=\"m\">80<\/span>\n  <span class=\"na\">selector<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">app<\/span><span class=\"pi\">:<\/span> <span class=\"s\">book-api<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>This YAML file tells Kubernetes to create a service called \u201cvinyl-book-service\u201d that will be listening on port 9000. The image for this service will be pulled from docker hub. Be sure to update the connection string in the YAML file.<\/p>\n\n<p>Create the above YAML file in the cloud shell and call it \u201c<strong>deploy-book.yaml<\/strong>\u201d and deploy the microservice by running the following command:<\/p>\n\n<div class=\"language-powershell highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"nf\">kubectl<\/span><span class=\"w\"> <\/span><span class=\"nx\">apply<\/span><span class=\"w\"> <\/span><span class=\"nt\">-f<\/span><span class=\"w\"> <\/span><span class=\"o\">.<\/span><span class=\"nx\">\/deploy-book.yaml<\/span><span class=\"w\">\n<\/span><\/code><\/pre><\/div><\/div>\n\n<h2 id=\"apply-the-routing-rules\">Apply the routing rules<\/h2>\n\n<p>For applying the routing rules, we are going to the use the following YAML:<\/p>\n\n<div class=\"language-yaml highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"na\">apiVersion<\/span><span class=\"pi\">:<\/span> <span class=\"s\">networking.istio.io\/v1alpha3<\/span>\n<span class=\"na\">kind<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Gateway<\/span>\n<span class=\"na\">metadata<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">public-gateway<\/span>\n<span class=\"na\">spec<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">selector<\/span><span class=\"pi\">:<\/span>\n    <span class=\"na\">istio<\/span><span class=\"pi\">:<\/span> <span class=\"s\">ingressgateway<\/span>\n  <span class=\"na\">servers<\/span><span class=\"pi\">:<\/span>\n  <span class=\"pi\">-<\/span> <span class=\"na\">port<\/span><span class=\"pi\">:<\/span>\n      <span class=\"na\">number<\/span><span class=\"pi\">:<\/span> <span class=\"m\">80<\/span>\n      <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">http<\/span>\n      <span class=\"na\">protocol<\/span><span class=\"pi\">:<\/span> <span class=\"s\">HTTP<\/span>\n    <span class=\"na\">hosts<\/span><span class=\"pi\">:<\/span>\n    <span class=\"pi\">-<\/span> <span class=\"s2\">\"<\/span><span class=\"s\">*\"<\/span>\n<span class=\"nn\">---<\/span>\n<span class=\"na\">apiVersion<\/span><span class=\"pi\">:<\/span> <span class=\"s\">networking.istio.io\/v1alpha3<\/span>\n<span class=\"na\">kind<\/span><span class=\"pi\">:<\/span> <span class=\"s\">VirtualService<\/span>\n<span class=\"na\">metadata<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">book-api-vservice<\/span>\n<span class=\"na\">spec<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">hosts<\/span><span class=\"pi\">:<\/span>\n  <span class=\"pi\">-<\/span> <span class=\"s2\">\"<\/span><span class=\"s\">*\"<\/span>\n  <span class=\"na\">gateways<\/span><span class=\"pi\">:<\/span>\n  <span class=\"pi\">-<\/span> <span class=\"s\">public-gateway<\/span>\n  <span class=\"na\">http<\/span><span class=\"pi\">:<\/span>\n  <span class=\"pi\">-<\/span> <span class=\"na\">match<\/span><span class=\"pi\">:<\/span>\n    <span class=\"pi\">-<\/span> <span class=\"na\">uri<\/span><span class=\"pi\">:<\/span>\n        <span class=\"na\">prefix<\/span><span class=\"pi\">:<\/span> <span class=\"s\">\/BookCatalogue<\/span>\n    <span class=\"na\">route<\/span><span class=\"pi\">:<\/span>\n    <span class=\"pi\">-<\/span> <span class=\"na\">destination<\/span><span class=\"pi\">:<\/span>\n        <span class=\"na\">port<\/span><span class=\"pi\">:<\/span>\n          <span class=\"na\">number<\/span><span class=\"pi\">:<\/span> <span class=\"m\">9000<\/span>\n        <span class=\"na\">host<\/span><span class=\"pi\">:<\/span> <span class=\"s\">book-api-service<\/span>\n<span class=\"nn\">---<\/span>\n<span class=\"na\">apiVersion<\/span><span class=\"pi\">:<\/span> <span class=\"s\">networking.istio.io\/v1alpha3<\/span>\n<span class=\"na\">kind<\/span><span class=\"pi\">:<\/span> <span class=\"s\">VirtualService<\/span>\n<span class=\"na\">metadata<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">vinyl-api-vservice<\/span>\n<span class=\"na\">spec<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">hosts<\/span><span class=\"pi\">:<\/span>\n  <span class=\"pi\">-<\/span> <span class=\"s2\">\"<\/span><span class=\"s\">*\"<\/span>\n  <span class=\"na\">gateways<\/span><span class=\"pi\">:<\/span>\n  <span class=\"pi\">-<\/span> <span class=\"s\">public-gateway<\/span>\n  <span class=\"na\">http<\/span><span class=\"pi\">:<\/span>\n  <span class=\"pi\">-<\/span> <span class=\"na\">match<\/span><span class=\"pi\">:<\/span>\n    <span class=\"pi\">-<\/span> <span class=\"na\">uri<\/span><span class=\"pi\">:<\/span>\n        <span class=\"na\">prefix<\/span><span class=\"pi\">:<\/span> <span class=\"s\">\/VinylProductCatalogue<\/span>\n    <span class=\"na\">route<\/span><span class=\"pi\">:<\/span>\n    <span class=\"pi\">-<\/span> <span class=\"na\">destination<\/span><span class=\"pi\">:<\/span>\n        <span class=\"na\">port<\/span><span class=\"pi\">:<\/span>\n          <span class=\"na\">number<\/span><span class=\"pi\">:<\/span> <span class=\"m\">8000<\/span>\n        <span class=\"na\">host<\/span><span class=\"pi\">:<\/span> <span class=\"s\">vinyl-api-service<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p><strong>Line 1 to 14<\/strong>: These lines create a default Istio gateway called \u201cpublic-gateway\u201d that will be listening for HTTP connections on port 80 from any host. \n<strong>Line 16 to 33<\/strong>: Here we create a virtual service for the \u201cbooks\u201d API service. A Virtual Service defines a set of traffic routing rules to apply when a host is addressed. Each routing rule defines matching criteria for traffic of a specific protocol. If the traffic is matched, then it is sent to a named destination service (or subset\/version of it) defined in the registry. In this case, we are telling the Istio gateway to route any HTTP traffic that start with \u201c\/BookCatalogue\u201d to the \u201cbook-api-service\u201d service on port 9000. \n<strong>Line 35 to 52<\/strong>: Here we create a virtual service for the \u201cvinyl\u201d API service. A Virtual Service defines a set of traffic routing rules to apply when a host is addressed. Each routing rule defines matching criteria for traffic of a specific protocol. If the traffic is matched, then it is sent to a named destination service (or subset\/version of it) defined in the registry. In this case, we are telling the Istio gateway to route any HTTP traffic that start with \u201c\/VinylProductCatalogue\u201d to the \u201cvinyl-api-service\u201d service on port 8000.<\/p>\n\n<p>Deploy the gateway along with the routing rules by running the following command:<\/p>\n\n<div class=\"language-powershell highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"nf\">kubectl<\/span><span class=\"w\"> <\/span><span class=\"nx\">apply<\/span><span class=\"w\"> <\/span><span class=\"nt\">-f<\/span><span class=\"w\"> <\/span><span class=\"o\">.<\/span><span class=\"nx\">\/istio-gateway.yaml<\/span><span class=\"w\">\n<\/span><\/code><\/pre><\/div><\/div>\n\n<h2 id=\"the-results\">The results<\/h2>\n\n<p>Find out the public IP of your AKS cluster by running the following command:<\/p>\n\n<div class=\"language-powershell highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"nf\">kubectl<\/span><span class=\"w\"> <\/span><span class=\"nx\">get<\/span><span class=\"w\"> <\/span><span class=\"nx\">svc<\/span><span class=\"w\"> <\/span><span class=\"nx\">istio-ingressgateway<\/span><span class=\"w\"> <\/span><span class=\"nt\">-n<\/span><span class=\"w\"> <\/span><span class=\"nx\">istio-system<\/span><span class=\"w\">\n<\/span><\/code><\/pre><\/div><\/div>\n\n<p>Note down the IP address in the column \u201cEXTERNAL-IP\u201d and use Postman to make HTTP GET requests to the following end points:<\/p>\n\n<ol>\n  <li>http:\/\/IP Address\/VinylProductCatalogue\/api\/VinylAlbums<\/li>\n  <li>http:\/\/IP Address\/BookCatalogue\/api\/Books<\/li>\n<\/ol>\n\n<p>The first endpoint should return a list of Vinyl Albums served by the Vinyl Microservice and the second endpoint will return a list of Books served by the Books Microservice.<\/p>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/head-first-http-routing-in-kubernetes-using-istio\/vinyl-postman.webp\" data-fancybox=\"\" data-title=\"Using Postman to call the APIs\">\n    <img src=\"\/images\/head-first-http-routing-in-kubernetes-using-istio\/vinyl-postman.webp\" alt=\"Using Postman to call the APIs\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Using Postman to call the APIs<\/figcaption>\n  \n<\/figure>\n\n<figure class=\"image-card caption\" style=\"text-align: center;\">\n  <a href=\"\/images\/head-first-http-routing-in-kubernetes-using-istio\/books-postman.webp\" data-fancybox=\"\" data-title=\"Using Postman to call the APIs\">\n    <img src=\"\/images\/head-first-http-routing-in-kubernetes-using-istio\/books-postman.webp\" alt=\"Using Postman to call the APIs\" \/>\n  <\/a>\n  \n  <figcaption class=\"caption-text\">Using Postman to call the APIs<\/figcaption>\n  \n<\/figure>\n\n<h2 id=\"what-did-we-just-do\">What did we just do?<\/h2>\n\n<p>We installed Istio on an AKS cluster, deployed two microservices and then an Istio gateway. We configured the gateway to use path-based routing and route traffic to the microservices based on the HTTP URL. We verified that all is good, by hitting the two microservices using Postman.<\/p>\n\n<h1 id=\"references\">References<\/h1>\n\n<table>\n  <tbody>\n    <tr>\n      <td>[1]<\/td>\n      <td>\u201cNgnix Blog,\u201d Ngnix, [Online]. <a href=\"https:\/\/www.nginx.com\/blog\/what-is-a-service-mesh\/\" rel=\"noreferrer noopener nofollow\">Available<\/a>. [Accessed 2020]<\/td>\n    <\/tr>\n  <\/tbody>\n<\/table>\n","pubDate":"Sat, 25 Apr 2020 00:00:00 +0000","link":"https:\/\/lalitadithya.com\/blog\/head-first-http-routing-in-kubernetes-using-istio\/","guid":"https:\/\/lalitadithya.com\/blog\/head-first-http-routing-in-kubernetes-using-istio\/","category":["azure","istio","kubernetes","k8s","microservices","service-mesh","routing"]},{"title":"Setting up IIS for hosting an ASP .NET Core website without inetmgr in 3 easy steps!","description":"<p>In this post we are going to look at how to set up IIS to host a ASP .NET Core website without inetmgr, i.e., no GUI.<\/p>\n\n<!--more-->\n\n<p>Our site will be called \u201cwww_mywebsite_com\u201d which will be present in \u201cC:\\inetpub\\wwwroot\\mywebsite\u201d and it will be listening on port 8080. In order to do this, we are going to be using a bat file and appcmd.<\/p>\n\n<h1 id=\"what-is-appcmd\">What is appcmd?<\/h1>\n\n<p>appcmd is a command line utility that was introduced by Microsoft when they released IIS 7 back in 2007. It can be used to modify or view any of IIS\u2019s settings from the command line.<\/p>\n\n<p>appcmd works best in command prompt, but you can use PowerShell\u2019s \u201cStart-Process\u201d cmdlet to invoke appcmd. The syntax to invoke any appcmd functionality is:<\/p>\n\n<pre><code class=\"language-cmd\">APPCMD &lt;COMMAND&gt; &lt;OBJECT&gt; &lt;ARGUMENTS&gt;\n<\/code><\/pre>\n\n<p>Now that we know what appcmd is, let us now use it to set up IIS for an ASP .NET Core website.<\/p>\n\n<h1 id=\"step-1-adding-an-application-pool\">Step 1: Adding an application pool<\/h1>\n\n<p>In order to add an application pool, we are going to make use of the \u201cadd\u201d command on the \u201capppool\u201d object as shown below:<\/p>\n\n<pre><code class=\"language-cmd\">C:\\windows\\system32\\inetsrv\\appcmd.exe add apppool \/name:\"MyWebsiteAppPool\" \/managedRuntimeVersion:\"\"\n<\/code><\/pre>\n\n<p>In the above command we pass the name of the application pool using the \u201c\/name\u201d switch and we set the CLR version to \u201cNo Managed Code\u201d by passing an empty string to the \u201c\/managedRuntimeVersion\u201d switch<\/p>\n\n<h1 id=\"step-2-adding-a-iis-website\">Step 2: Adding a IIS website<\/h1>\n\n<p>To add the website, we are going to make use of the \u201cadd\u201d command on the \u201csite\u201d object as shown.<\/p>\n\n<pre><code class=\"language-cmd\">C:\\windows\\system32\\inetsrv\\appcmd.exe add site \/name:www_mywebsite_com \/id:2 \/physicalPath:C:\\inetpub\\wwwroot\\mywebsite \/bindings:http\/*:8080:\n<\/code><\/pre>\n\n<p>We pass the name of our website (www_mywebsite_com) using the \u201c\/name\u201d switch. We have to pass in a unique ID for our website. So, we pick ID 2, because site ID 1 will be taken by the \u201cDefault Website\u201d in IIS. We pass the path in which our website will be present (C:\\inetpub\\wwwroot\\mywebsite) using the \u201c\/physicalPath\u201d switch. We pass our bindings, i.e., listen for http connections on port 8080, using the \u201c\/bindings\u201d flag<\/p>\n\n<h1 id=\"step-4-link-application-pool-to-the-website\">Step 4: Link application pool to the website<\/h1>\n\n<p>In the last and final step we will be linking the application pool we created in step 1 to the website we created in step 2. In order to do this, we will use the \u201cset\u201d command on the \u201csite\u201d object as shown:<\/p>\n\n<pre><code class=\"language-cmd\">C:\\windows\\system32\\inetsrv\\appcmd.exe set site \/site.name:\"www_mywebsite_com\" \/[path='\/'].applicationPool:\"MyWebsiteAppPool\"\n<\/code><\/pre>\n\n<p>In the above command, we select the website we want to modify using the \u201csite.name\u201d switch. We then use the \u201c\/[path=\u2019\/\u2019]\u201d switch to select the root application, i.e., \u2018\/\u2019 and then combine that with the \u201capplicationPool\u201d selector to pass the name of the application pool we created earlier.<\/p>\n\n<p>Now all that is left would be to copy the ASP .NET Core website artifacts to \u201cC:\\inetpub\\wwwroot\\mywebsite\u201d and you can view the website at localhost:8080.<\/p>\n\n<h1 id=\"summary\">Summary<\/h1>\n\n<p>We created a new IIS website linked to an application pool without using inetmgr. Here are the commands we used:<\/p>\n\n<pre><code class=\"language-cmd\">C:\\windows\\system32\\inetsrv\\appcmd.exe add apppool \/name:\"MyWebsiteAppPool\" \/managedRuntimeVersion:\"\"\nC:\\windows\\system32\\inetsrv\\appcmd.exe add site \/name:www_mywebsite_com \/id:2 \/physicalPath:C:\\inetpub\\wwwroot\\mywebsite \/bindings:http\/*:8080:\nC:\\windows\\system32\\inetsrv\\appcmd.exe set site \/site.name:\"www_mywebsite_com\" \/[path='\/'].applicationPool:\"MyWebsiteAppPool\"\n<\/code><\/pre>\n\n<p>There is so much more that can be done with appcmd! To learn more, have a look at the official documentation <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/docs.microsoft.com\/en-us\/iis\/get-started\/getting-started-with-iis\/getting-started-with-appcmdexe\">here<\/a>.<\/p>\n\n<p>You can find this complete bat script <a rel=\"nofollow noreferrer noopener\" href=\"https:\/\/github.com\/lalitadithya\/Scripts\/blob\/master\/Deployment\/set-up-iis.bat\">here<\/a>.<\/p>\n","pubDate":"Mon, 13 Apr 2020 00:00:00 +0000","link":"https:\/\/lalitadithya.com\/blog\/setting-up-iis-for-hosting-an-asp-net-core-website-without-inetmgr-in-3-easy-steps\/","guid":"https:\/\/lalitadithya.com\/blog\/setting-up-iis-for-hosting-an-asp-net-core-website-without-inetmgr-in-3-easy-steps\/","category":["iis","aspnet-core","deployment","devops","appcmd"]}]}}