Azure CI and CD pipelines using Linked Template Specs

Overcome limit of ARM Templates.

Posted by radekrezac on September 25, 2024

This document goes over how to write code to package up and promote a TEST environment Azure Data Factory to PROD environment Data Factory using Linked ARM Templates via a new Linked Template Specs process. Linked Templates are needed when your Data Factory size is over 4MB. This process goes over how to deploy large Data Factories.

Template limits

Limit size of ARM Template is 4 MB, and for each resource definition it is 1 MB. The limits apply to the final state of the template after it has been expanded with iterative resource definitions, and values for variables and parameters. The parameter file is also limited to 4 MB. You may get an error with a template or parameter file of less than 4 MB if the total size of the request is too large.

You're also limited to:

  • 256 parameters
  • 512 variables
  • 800 resources (including copy count)
  • 64 output values
  • 10 unique locations per subscription/tenant/management group scope
  • 24 576 characters in a template expression

Overcome limit of 256 parameters

Parameters of ARM template are generated by ARM template configuration defined in a ADF Managed Page. Procedure for limiting the number of parameters:

  1. Create a new branch
  2. Go to ADF Managed page
  3. Select ARM template tab
  4. Edit parameters

  1. Go through the parameters that are used in the PROD environment
  2. Remove unused parameter types and save definition

  1. Create pull request

Using ARM linked templates by Microsoft

To use linked templates instead of the full Resource Manager template, update your CI/CD task to point to ArmTemplate_master.json instead of ArmTemplateForFactory.json (the full Resource Manager template). Resource Manager also requires that you upload the linked templates into a storage account so Azure can access them during deployment. Since this is a Linked Template, the ARM deployment task requires the storage account URL and SAS token. The SAS token is needed even if the Service Principle has access to the blog since Linked Templates deploy inside Azure without context of the user. To achieve this, the Linked Template produced by the CI/CD steps require the following parameters containerURI and containerSasToken. It's recommended that you pass the SAS token in as a secret either as a secure variable or from a service like Azure Key Vault.

Steps

  1. Create a Storage Account (public access enabled)
  2. Copy ADF Linked ARM Template files to the Storage Account
  3. Create a SAS (Shared Access Signature) token on the Storage Account and set the expiration timestamp
  4. Add the SAS token as a secret variable on the Azure Pipeline or store it as a Key Vault Secret and then connect to it to the Pipeline
  5. Add the SAS token (or Key Vault secret) and Storage Account info to the Azure Pipeline deployment task so ARM (Azure Resource Manager) can use it to deploy the ADF Linked Templates
  6. Deploy the Linked ARM Templates to the target ADF (TEST, PROD, etc.)
  7. Either delete the Storage Account or wait for the SAS token to expire

Challenges

  • Storage Account has to have public access enabled (can't lock it down via firewall)
  • SAS doesn't need to authenticate users. Whoever has a SAS token can use it to access the Storage Account
  • SAS token is logged in the deployment operations even if passed as a secure string (see Additional ARM Limits)
  • No API or way to check how many SAS tokens are on a Storage Account
  • No way to delete a SAS once it has been created (have to let it expire)
  • Have to give elevated permissions on a Storage Account in order to generate a SAS token
  • AzureFileCopy@6 Azure Pipeline task only works on Windows agents. Have to use a different approach (ex: az storage blob) for other agents (Linux, etc.)

New Approach Using Linked Template Specs

A template spec is a resource type for storing an Azure Resource Manager template (ARM template) in Azure for later deployment. This resource type enables you to share ARM templates with other users in your organization. Just like any other Azure resource, you can use Azure role-based access control (Azure RBAC) to share the template spec.

To deploy a template spec as a linked template, we use template specs to share ARM templates with other users in our organization. After we create a template spec, we can deploy it by using Azure PowerShell or Azure CLI.

Steps (All Automated)

  1. Deploy ADF Linked ARM Templates as Template Specs (to deploy Template Spec to a Resource Group, Pipeline Service Connection needs at least the Template Spec Contributor RBAC role or greater)
  2. Create new ADF Master Linked ARM Template that uses the Template Specs as Linked Template Specs
  3. Deploy to the target ADF (TEST, PROD, etc.) using the Linked Template Specs
  4. Optionally, delete the Template Specs after successful deployment

Benefits

  • Process is automated via the new Pipeline code
  • Template Specs are RBAC secured by default
  • Can version Template Specs (1.0.0.0, 1.0.0.1, etc.)
  • No need to copy files to a public Storage Account or create and retrieve an SAS token
  • Can use any agent (Windows, Linux, etc.)

Azure CI (build) pipeline using Linked templates

  • Pipeline contains three stages:

    • BUILD - Here are the tasks that validate and generate ARM templates from ADF sources. Then they publish the artifact for later use.
    • LTS_TEST/LTS_PROD - In this stages we use the artifact from the previous BUILD stage, create Template specs for both subscription (environment) for each Linked Template and create a new template master files referencing the Template specs. Finally, we publish the artifacts with the new template master file.

  • The result of the CI pipeline is the creation of an artifacts for the release pipeline

  • Generated template specs in the defined resource group

Azure CD (release) pipeline using Linked templates

  • Pipeline contains stages for divisional ADF environments:

    • TEST
    • PROD

  • Stages the same tasks for each environment using variable group for each environment

  • in addition to the classic pre and post deployment task (stop/start triggers, remove deleted objects...), there are deploy task:

    • Deploy To ADF Using Linked Template Spec - In this step, we deploy to given resource group where the ADF resources is defined in a template specs generated by the LTS stage of the CI pipeline.

Deployment script

For deployment from Linked Template specs to ADFs are used Azure CLI script. Here is, among other things, a deployment group create comand using the created Linked Template specs. To override the template parameters, json files stored in the same directory as the deployment script are used:

  • adf-test-template-parameters.json for TEST environment

  • adf-prod-template-parameters.json for PROD environment

Data Management
ADF