Pipeline Security and Governance
For the fourth post in this series on the basics of Terraform for Microsoft Azure using Azure DevOps and Visual Studio Code, I’m going to be covering the slightly more complex issues of governance and security, along with a bit of automation using triggers. This post is quite detailed, and if you’re going to follow along with me, it’s important that you’ve followed all the steps in the previous three posts (prerequisites, repositories & pipelines, build pipeline), and all code, repos and pipelines are where we left them in the last episode.
As a reminder, this series has been written as a memory aid for myself, that may be able to help out others along the way. It’s not a detailed guide to Terraform for somebody new to it, it probably doesn’t meet a bunch of best practices, and as this piece was written in the middle of 2024, technology might have moved on, but it’s what works for me, right now. Thanks again go to James Meegan for his original documentation on which this series is based.
Introduction
The apply (or build) and validation pipelines that we have deployed thus far are very simple; they’re just a series of tasks, running one after the other. They’re also completely separate entities that are being manually run from within the Azure DevOps portal. The goal of this post is to help you understand the concept of stages (where the apply pipeline will have a stage to run init and plan, then a second stage to run init, plan and apply), so that we can add governance such as approvals for each stage, environments (allowing us to target our stages), branches (to keep development and deployment code separate and secure), and automation (allowing our pipelines to run without manual intervention).
The post may be more wordy than usual, as I’ll attempt to explain the processes behind each step and the reasons for doing them, so that the logic is understood for later implementation.
Environments
As mentioned above, environments are somewhere to target our stages. We’ll get into the detail of stages shortly, but best practice dictates that although we can create environments in our YAML pipeline code, we should do it in advance so that we can ensure appropriate security and approvals are configured. When we do get to stages, we will not be adding any to our validation pipeline, but for our apply pipeline we will want two of them: a planning stage and an apply stage. This means we need to create two environments for these stages to use, on each of which we can configure separate security and governance if we require it. For our apply stage we are going to add an approval check, so that the actual execution of that stage cannot continue until manually approved.
Approvals
Before choosing approvers, give consideration to who you need them to be. Try not to add named individuals directly, but rather add a group that contains the individuals as members. It’s good practice to add your built-in “Build Administrators” group to your approval step, and add the necessary approvers into that group as required. As I’ve said before, this is not an in-depth course on Azure DevOps or Terraform, but to add users to the built-in groups such as “Build Administrators”, go to the root of your project and select “Project Settings” from the bottom left:
Once in there, look for “Permissions” on the left and click on that. The group itself will then be visible on the right, along with all the other built-in groups. Select the “Build Administrators” group:
Select the “Members” tab at the top of the page, then add yourself in there (assuming you’re following along with the series, and for the purposes of this example). I’ve added myself to my own “Build Administrators” group, so that’s what I’m going to choose as my approver for the apply stage – it’s normally not exactly best practice to “mark your own homework” though, so maybe don’t do that for customer or production systems!
Environment Creation
We create environments in the Azure DevOps portal. Open this, and in the Pipelines section on the left, select “Environments”, before clicking “Create environment”:
Give your new environment a name based on your own naming convention. I’m using the name of the repository and the stage. Add a sensible description, leave the resource as “None” and click “Create”:
As previously mentioned, we won’t be adding any governance over the validation pipeline, because it won’t be doing anything destructive. The same applies to the validation stage of the apply pipeline; this stage will be doing nothing destructive so at this point no further security or governance is required. Although the stage is effectively repeating the work of the validation pipeline, because we are going to be using branches, we need to make sure that the plan stage happens in both our development branch, and in our main branch which will apply our changes to Azure.
We now need to create the environment that will perform the addition, changing or destruction of resources, and that will need our governance steps. Go back into “Environments” and repeat the process to create an environment for your apply stage:
Now that the apply environment has been created, we need to think about the governance we want around it. For this example we’re going to add an approval checkpoint, so the stage can’t run until somebody has reviewed the plan and approved the apply stage to go ahead. While still in your new apply environment, select “Approvals and Checks”:
In the presented list, select “Approvals”:
So… After clicking “Approvals”, select your approvers (As mentioned above, I’m using the built-in group “Build Administrators”). If names don’t automatically resolve in the top “Approvers” box, you need to be granted the permission to resolve names in Entra ID with reader permissions. Add a quick note to any potential approvers, then click “Create”:
We now have two environments for our stages, one for validation and one for apply, which has an approval step associated. It’s worth noting here that the first time a pipeline uses an environment, it will request permission which you need to grant. Once granted however, it will not ask again. As we have two environments, permission will be requested twice as the pipeline runs for the first time.
Stages
First a bit of a recap and explanation around stages. Because our validation pipeline does not actually alter or deploy any infrastructure, we won’t be adding governance or stages to it. Right now, in our apply pipeline, where we do want that governance, we have a series of tasks. We will be adding stages to split up these tasks; they will be arranged into separate “jobs”. A job is defined by Microsoft as “a collection of steps that run sequentially”. Microsoft show in the linked article some example code to demonstrate the syntax and format of a stage:
- stage: deploy
jobs:
- deployment: DeployWeb
displayName: deploy Web App
pool:
vmImage: 'Ubuntu-latest'
# creates an environment if it doesn't exist
environment:
name: 'smarthotel-dev'
resourceName: myVM
resourceType: virtualMachine
strategy:
runOnce:
deploy:
steps:
- script: echo Hello worldTo break that down into a simplified hierarchy as required by YAML, we have a stages section inside which are nested jobs, inside which are nested tasks:
stages:
- stage: <stage name>
jobs:
- job: <job name>
steps:
- task: <task 1 name>
- task: <task 2 name>We will now expand the code in our apply pipeline using stages to carry out more complex functions and add the governance of the approval step we set up earlier in our apply environment.
We need our apply pipeline to have one stage for each of the environments that we’ve created. The first stage will carry out the validation (“init” and “plan”), with the second stage carrying out the resource build / amend / destroy (“init”, “plan” and “apply”). It’s worth recognising here that Azure DevOps treats each stage as an individual piece of code, where details from one stage don’t feed into the next. This is why the “init” and “plan” commands are run twice in the same pipeline. Again, as the apply stage can be a little bit dangerous (note the word “destroy” above), we really want the pipeline to pause after that initial plan, and just ask us “Are you really sure you want to do this???” with that approval step. We don’t want any deployments, amendments or destruction happening automatically with us seeing something in an automatically running job and trying our best to cancel it while it’s in progress. To achieve all this, we need to amend our code and its indents to add the hierarchy seen above. Because we’re using stages, we’re also going to have to add a “checkout” step. In a basic pipeline such as our validation one, checking out the code from the repository is done automatically, but this doesn’t happen with stages so the step needs to be added to the YAML file.
Amend the Code
In Visual Studio Code, open up your apply pipeline. We know that to match the syntax detailed above we’ll have to put our existing tasks inside stages and jobs, meaning indenting the lines appropriately too. When we add the stages and jobs sections, we’ll add comments and sensible display names as we’ve been doing before, so that we understand what’s happening as we look at the code or as we monitor jobs being run. The code layout as you’ve seen is relatively logical, but the indentation and hyphen location is critical, and if they’re wrong the pipeline will fail. So first, above our existing tasks (or “steps”), we’ll add our “stages” section:
# Stages tied to environments
stages:
# Plan Stage
- stage: plan
# Apply Stage
- stage: apply
Now that we have our stages in place, we need to build them out by adding jobs (with the appropriate indentation). We need to mark that each of our jobs is a deployment job (even though the plan stage won’t run the “apply” step so won’t deploy anything), then give a display name and mark the environment that each stage should target. Each of our deployment jobs needs a name. This can be anything you wish but to reflect the governance in place I’m going to call them “Check” and “NoCheck”. The environment name should match that of the environment you are targeting. Because it’s vital, I’ll repeat, keep an eye on that formatting and indentation:
# Stages tied to environments
stages:
# Plan Stage
- stage: plan
jobs:
- deployment: NoCheck
displayName: No Check
environment: 'Test-Resources-Repo-Validation'
# Apply Stage
- stage: apply
jobs:
- deployment: Check
displayName: Check
environment: 'Test-Resources-Repo-Apply'
I mentioned earlier that the environments could be created in the YAML code but it was best practice to create them in advance and get your governance in place. This is done by examining the “environment” field as shown in the code above, and if an environment with that name doesn’t already exist, it will create it. This means that if you make a typo here, you could deploy to a completely new environment that doesn’t have your approvals step for example.
We’re slowly building up our stages now, and we clearly have two of them, targeted to our two environments. To meet the requirements of Microsoft’s syntax, we now need to add an appropriately indented “strategy” block to each stage. This is effectively done by copying in our previously existing code, but it’s here that we add that new “checkout” step. We are checking out to “self”, so that the pipeline checks out the branch that triggered its run. To make this simpler to follow, I’ll show each of the stages being built out in turn. First we copy our “init” and “plan” steps into our “plan” stage, and add that checkout step:
# Plan Stage
- stage: plan
jobs:
- deployment: NoCheck
displayName: No Check
environment: 'Test-Resources-Repo-Validation'
strategy:
runOnce:
deploy:
# Steps to perform as part of the pipeline operations
steps:
# Check out the current repository branch
- checkout: self
# Terraform Initialisation
- task: TerraformTaskV4@4
displayName: Run Terraform Init
inputs:
provider: 'azurerm'
command: 'init'
backendServiceArm: 'terraform-series-sc'
backendAzureRmResourceGroupName: 'uks-tfstatefiles-rg-01'
backendAzureRmStorageAccountName: 'ukstfstatefilessa01'
backendAzureRmContainerName: 'tfstatefiles'
backendAzureRmKey: 'terraformstatefiles/terraform_series.tfstate'
# Terraform Plan
- task: TerraformTaskV4@4
displayName: Run Terraform Plan
inputs:
provider: 'azurerm'
command: 'plan'
commandOptions: -out=plan
environmentServiceNameAzureRM: 'terraform-series-sc'
Note how the code has been indented to match the syntax provided earlier. Our tasks are clearly sub-components of “steps”, which comes under the new “deploy”, etc… Again, as mentioned above, our apply stage is treated as completely separate code, so we need to add our checkout step, then cut and paste in (we need to remove all the previous code that’s not part of our stages now) the “init”, “plan” and “apply” tasks, with the appropriate indentation:
# Apply Stage
- stage: apply
jobs:
- deployment: Check
displayName: Check
environment: 'Test-Resources-Repo-Apply'
strategy:
runOnce:
deploy:
# Steps to perform as part of the pipeline operations
steps:
# Check out the current repository branch
- checkout: self
# Terraform Initialisation
- task: TerraformTaskV4@4
displayName: Run Terraform Init
inputs:
provider: 'azurerm'
command: 'init'
backendServiceArm: 'terraform-series-sc'
backendAzureRmResourceGroupName: 'uks-tfstatefiles-rg-01'
backendAzureRmStorageAccountName: 'ukstfstatefilessa01'
backendAzureRmContainerName: 'tfstatefiles'
backendAzureRmKey: 'terraformstatefiles/terraform_series.tfstate'
# Terraform Plan
- task: TerraformTaskV4@4
displayName: Run Terraform Plan
inputs:
provider: 'azurerm'
command: 'plan'
commandOptions: -out=plan
environmentServiceNameAzureRM: 'terraform-series-sc'
# Terraform Apply
- task: TerraformTaskV4@4
displayName: Run Terraform Apply
inputs:
provider: 'azurerm'
command: 'apply'
commandOptions: plan
environmentServiceNameAzureRM: 'terraform-series-sc'
That’s effectively it. You’ve broken down your pipeline into two separate stages, that are targeted at environments which apply security and governance where it’s needed. Because we have that governance, we can start adding a bit of automation!
Branches and Automation
Before we save our pipeline file (don’t worry if you already have!), we want to automate its running. I’ve mentioned before that we will have a “Main” branch which will do the resource deployments, and an effectively sandboxed “Develop” branch where we can make changes and view the plan before committing that code to our main branch. We want our apply pipeline to run automatically as soon as code is deployed into our main branch, so to do that we amend our “Trigger” line at the top of the apply.yml pipeline file from “none” to “main”:
# When to run the code - "none" defines that the pipeline must be run manually
# - "main" defines an automatic run when the "main" branch is updated
trigger:
- main
You can now save the file, remembering to run “terraform fmt -recursive” afterwards to check your file’s formatting (although it doesn’t really pick up mistakes in YAML)!
Secure the “Main” Branch
If you cast your mind back, you’ll remember that the main branch is created and set as default when the repository is created. When other branches are created we can choose to change the default, but for now it will remain as “main”. The “Main” branch is our single source of truth. It contains the code that has been validated and approved and is what is deploying our resources into Azure. Our development code, where we run our validation pipeline and make changes and tests, will be put into a different branch (that we will create later!), and that will only be deployed to the main branch when we’re happy that the code does exactly what we want. We also know that when code is put into the main branch, the apply pipeline will trigger and start deploying what’s in there, so we really don’t want test or development code being saved there either intentionally or accidentally. We do this by securing the main branch to prevent anyone (including yourself) from uploading code directly to it; we need to make sure that code is only “pulled” into that branch from another branch using a fully governed process. We do this using branch policies, which are accessed from the “Project settings” screen:
Click on “Repositories” on the left, then select the “Policies” tab, and click the (+) button next to “Branch Policies” (you will need to scroll down to find some of the options):
Click “Create” to protect the default branch (which in our case is “Main”):
You can choose any option you wish, they will all automatically require pull requests to be used to update the branch. For this example, I’ve turned on the policy that requires a manual approval for pull requests into default. I’ve set the number of reviewers to one, and allowed requestors to approve their own changes:
Just one reviewer that can check their own homework is not the best form of governance, but it gives me protection from accidents and that’s pretty much all I need for this exercise. Make sure any production branches have better protection! The default or main branch is now protected across all repositories in this Azure DevOps project. You can set this protection on a repository by repository basis, but I find that doing it globally is the more secure option. If you try to synchronise your updated apply pipeline you’ll get a warning:
Create a Development Branch
So if we can’t push up our updated code from VS Code, how do we get it into our repository in Azure DevOps? We do it by creating a branch without that protection and uploading there! It’s possible to do this in Azure DevOps, but I find it works better for me doing it in VS Code. In that program, select “View / Command Palette…” (or press <Ctrl>+<Shift>+P):
In the resulting command box, find and select “Git: Create Branch…”:
Enter your chosen name. I’m going to stick to my own naming convention and call it “Develop”:
In the bottom left of your VS Code window you’ll now see the branch icon to the left of your branch name. Click the “Publish” icon (
) to the right to push this new branch up to Azure DevOps:
To switch between branches at any time, just click the branch name and select the one you want to be in.
Automate the Validation Pipeline
Now that we are working in a safe, sandboxed development branch, unable to accidentally send code off to our main branch to deploy or destroy resources, we can automate the running of the validation pipeline. Just like with the apply pipeline, we want to change the trigger to the name of the branch, then when the branch is updated, the pipeline will rune. It’s important to note that the branch name under “Trigger” is case-sensitive, so if you’ve got a capital letter in your branch name (as I do), then that must be replicated in the pipeline code. Open your validation.yml pipeline in VS Code and change the trigger text and any comments:
# When to run the code - "none" defines that the pipeline must be run manually
# - "Develop" means the pipeline will run on updates to the Develop branch (case sensitive)
trigger:
- Develop
When this is synchronised to Azure DevOps, we will no longer have to manually run our pipelines when updating our code – they’ll both run automatically.
Push and Pull the Code
We’re now in a position to push our code up to our development branch in Azure DevOps (synchronise), have our validation pipeline automatically run, then if we’re happy with that, pull our code into the main branch and have the apply pipeline run a validation, ask for approval, then perform the apply to create, modify or destroy resources.
Save your validation.yml file, run terraform fmt -recursive, then (after making sure in the bottom left that you’re still in your development branch) add a comment, commit and synchronise your changes:
If you now go to Azure DevOps and look at your pipelines, you should see that your validation pipeline is running (or has already run):
Click into that pipeline and select the most recent run at the top. If you now click on “Job”, you should see that the pipeline has run, with the “Run Terraform Plan” task completing and showing any changes that will happen:
I’ve added no new code to deploy or change infrastructure, so there are no changes planned for my resources. I’m happy that the code is good though, so I’m ready to pull that code which was pushed to my development branch, into my main branch. To do this, select “Repos” on the left and make sure you’re in the right repository. You can select the branch in the top left of the main window. We want to make sure we’re in “main” because we’re “Pulling” the code into the branch; our security will not let us push it there. When you’re in your main branch, select “Create a pull request”:
Add a title to your pull request along with an optional description. So that you can see and understand the history of your code pulls, and others can follow how the process went, it’s a good idea to make these sensible and easily readable. There are other options that can be selected here, which are definitely worth researching, but for our purposes, just click “Create”:
At this point, our governance step kicks in and we need to have the pull request approved. At this point you should contact your approver and ask them to review your request and approve it, but for this example, where I’ve allowed myself to approve my own requests, I’m just going to click “Approve”:
When the pull request has been approved, in the same screen, click the “Complete” button. Untick the option to delete your development branch, then click “Complete merge”:
If you now go into your pipelines, you should see your apply pipeline running:
Note that the name of my apply pipeline has defaulted to that of the repository again. As with the validation pipeline, you can easily rename this, and it’s probably sensible to do so:
Clicking into the pipeline will show us the most recent runs. Click the most recent at the top. Remember that in one of the steps above I mentioned that each stage would need once-only permissions granting? This is the message we have now. Click on “View”, then “Permit” twice:
Your plan stage will run (you can click into the stage and watch the process if you wish, just click the breadcrumb trail at the top of the page to get back), then pause again for its one-off permission request for the apply stage. Repeat the permission granting process. We then hit our apply stage approval check that was set in that stage’s environment. This happens each time the pipeline runs, and all nominated approvers should receive an email. Click on “Review”:
Add a comment if you wish, then click “Approve”:
If you now click into your apply stage, you will see the tasks being individually run. If you’re following my code, you’ll see init, followed by plan, followed by apply:
So that’s it! Why not try adding code for another resource group and following the process through to the end?
I know this has been a long and detailed post, but hopefully you now have an understanding of stages, how they target environments and what they are, what branches are and how they’re used, along with governance and security. That’s a lot to get through and take in. Don’t worry if you’re getting errors, troubleshooting is all part of the fun! Look through the outputs from your plan and apply stages, see what they say.
I’d love to hear from you as to how this series is going. Have I missed something, does something not work, is the pace of the articles about right, have you deployed a bunch of resources using multiple stages and branches? Leave me a comment to let me know.
Until next time…
– The Zoo Keeper
2 comments