Terraform for Azure: Basics (10)

For / Each Loops

Hello (again?), and welcome to this, the tenth and penultimate post in our series on the basics of using Terraform and Azure DevOps to deploy infrastructure as code into Microsoft Azure. I’m making no promises about an “intermediate” series, and I don’t see me getting to advanced ever, but I do hope to be able to add more posts to the site as my own skills improve and I learn new tips and tricks.

As always, when I’m writing these posts I make an assumption that you’re following along with me, that your code is in the same place as mine and that generally means you’ve worked through the previous episodes:

1. Prerequisites
2. Repositories and Pipelines
3. Build Pipeline and Resource Deployment
4. Pipeline Security and Governance
5. State File Storage Security
6. Pipeline Refinement
7. Modules
8. Directories and Stages
9. YAML Pipeline Templates

If something in this post doesn’t make sense, or is confusing, go back over the previous posts and see if the answer’s there. Alternatively, I’m human, I get things wrong, I forget things, and I always appreciate suggestions on how to improve my posts, so leave me a comment!

I’ve copied in my usual preamble here: I’m writing this series as a memory aid for myself with the hope that others can use it that might be on the same learning path as me. It’s not a detailed guide into Terraform, DevOps or Azure, it’s just an aid to understanding all the moving parts which will help with an understanding of and troubleshooting infrastructure as code environments. It probably doesn’t meet all the best practices that a DevOps engineer would use, but it works for me. I’m writing in the middle of 2024, so by the time you read this things might have moved on a little, but I suspect the core concepts will remain the same. As always I’d like to credit my friend and colleague James Meegan for his original documentation which has been used as a foundation for the series.

This post is all about simplifying our code by using for / each loops. Rather than filling .tf files with module blocks calling for data from other files, I want us to be able to leave those .tf files static, and when we want a new resource, just add the data to the .tfvars file where it’ll be picked up and applied. Again it’s about abstracting the code from the data that little bit further, whilst also reducing the number of code lines we have to write or repeat.

For / Each Loops

Underlying Concepts

As hinted at above, rather than using the default method of deploying resources, which is one at a time using separate resource blocks or module blocks, we want to simplify that and deploy however many of each type of resource is specified in our .tfvars file. If we want to deploy five resource groups, should we be putting their details into our .tfvars file, then adding five module blocks in our root .tf file, one for each resource group? Or would it be easier if we just put their details in the .tfvars file and had a single module block in the root .tf file that doesn’t change and just calls the Terraform module five times? I know what I prefer, and the for / each loop helps us to do that.

We can use the loops to deploy any Azure resources, so once we understand the logic of using them with one kind of resource, we can apply that same logic to any other resource type. Let’s start off with resource groups, as they’re relatively simple and can help us get to grips with the concepts. Right now, in our production .tfvars file, we have the following entry for creating a resource group:

#  First UK South Resource Group
first_uks_rg_name     = "uks-example-rg-01"
first_uks_rg_location = "UK South"

This is the minimum requirements for creating a resource group, and the .tf file in the root is passed this information when running its module block specific to that resource group:

#  First UK South Resource Group
module "first_uks_rg" {
  source      = "./modules/resourceGroups"
  rg_name     = var.first_uks_rg_name
  rg_location = var.first_uks_rg_location
}

When we’re passing the information to a for / each loop, it’s done slightly differently. Instead of being single lines of information passed through, we need to send it as an array, or list. Even though there’s only one resource right now, it still needs to be in an array, which as the following layout:

array_name = {
  resource_reference = {
    resource_value = "value"
  }
}

Let’s start by creating an array for resource groups in our production .tfvars file. Open the file in VS code and go to your resource groups area. Let’s call the array “resource_groups”, and put the details of our current resource group in there:

resource_groups = {
  #  First UK South Resource Group
  first_uks_rg = {
    rg_name     = "uks-example-rg-01"
    rg_location = "UK South"
  }
}

Make sure to delete or comment out your original resource group data. To demonstrate how we can deploy multiple resources, let’s add the data for a second resource group:

resource_groups = {
  #  First UK South Resource Group
  first_uks_rg = {
    rg_name     = "uks-example-rg-01"
    rg_location = "UK South"
  }
  #  Second UK South Resource Group
  second_uks_rg = {
    rg_name     = "uks-example-rg-02"
    rg_location = "UK South"
  }
}

The idea is that if we need another resource of the same type, all we need to do is add another item with its information to the array, and the for / each loop in our single module block will pick it up and deploy it – no extra resource or module blocks, no extra variables to declare, just the details of the resource added to the array in the .tfvars file.

Variable Declaration

Speaking of declaring variables, we do have to declare our array as a variable so that Terraform understands what it is and what type it is. As with our standard module blocks, we do that in our 01-variables.tf file in our root directory. Open the file, and where you have your current resource group variables declared, we will replace them with our array variable. It is in the same format as our original variables, but we’re adding another couple of lines to declare that the type of entry can be anything (type = any), to allow multiple values in the array, and that a default entry would be null (default = []). This means that the array isn’t mandatory, and we can deploy other resources without values given for that specific type of resource. With our comment and description, our declaration block should look like the following:

# Resource Groups Array Variable
variable "resource_groups" {
  description = "Array of resource groups to create"
  type        = any
  default     = []
}

Creating the Loop

The last thing we need to do is to add our module block that contains the for / each loop, which will take the array as an input and call the resource group creation module for each entry in that array. The module block has the following format:

module "module_reference" {
  source   = "module_location"
  for_each = var.declared_array_variable
  value    = each.value.defined_resource_value
}

Earlier in the series we created a separate 02-resourceGroups.tf file in our root directory, so we need to open this in VS Code. We replace the existing module block with a new one that contains our for / each loop, and the required values of rg_name and rg_location for our module:

# Resource Groups For / Each loop
module "resource_groups" {
  source      = "./modules/resourceGroups"
  for_each    = var.resource_groups
  rg_name     = each.value.rg_name
  rg_location = each.value.rg_location
}

You can see that I’ve updated the comment on the first line to reflect the updated purpose of the .tf file; I like to keep anybody looking at the files after me well informed. I’ve called the module “resource_groups” for simplicity, but it can be any name you choose and doesn’t need to be declared. I’m pointing the source to the resource groups module we created earlier in the series and telling the for_each loop to use the defined array variable of resource_groups. After defining the loop, we pass through the required values to the module, and we’re saying “For each item (each.value) in the array, pass through the rg_name value as rg_name to the module and the rg_location value as rg_location to the module”.

Dependencies

That’s all the code we need to deploy our resource groups using a for / each loop, but we still have resources (our VNets!) being deployed that have a dependency on our original resource group module to be deployed themselves. Because the previous module block was deploying only one resource group, we could refer to the output of that to get our resource group’s name. We need to update these resources to use the name of the specific resource group that we want, that has been created as part of the loop. Sounds like fun?

OK, first of all we need to open our VNets .tf file in our root directory and look for the “depends_on” lines. We need to change that from the current module block reference which points at the old module block for the first resource group, to the new module block in our resource groups .tf file that contains the for / each loop:

depends_on          = [module.resource_groups]

Next we need to amend the resource group name value. We want to point at the module block, then select the array item reference (from our .tfvars file) that we want to use, then as before ask for the module output of resource_group_name:

  resource_group_name = module.resource_groups.first_uks_rg.resource_group_name

Remember to do this for both your VNets. Why not save all your files now, check your formatting, commit and sync, then see what your validation pipeline job records in the plan. Ideally it should run successfully and you’ll see that your original resource group is set to be deleted and two new resource groups will be created:

Note again that there’s no reference to our VNets changing, meaning that if we were to attempt to apply this right now, the apply pipeline would fail because it can’t destroy a resource group that isn’t empty.

Creating a VNet For / Each Loop

So let’s take the knowledge we gained from our resource groups for / each loop and expand that into our VNets. Our first job is to declare the array variable in our variables .tf file in our root directory:

#  VNets Array Variables
variable "vnets" {
  description = "Array of VNets to create"
  type        = any
  default     = []
}

Now that file’s looking much clearer and easier to follow. Next up is completing the array information in our .tfvars file. We will copy the syntax from the resource groups array, and enter our information from the previous VNet variables, but we’ll also add a new value for each VNet. Our module block will become a for / each loop with no fixed values, so we need to put the resource group we want to use in this .tfvars file. We add it and refer to the resource groups array entry that we want to use. I’m choosing a different resource group for each of my VNets:

vnets = {
  #  UK South Hub VNet
  uks_hub_vnet = {
    vnet_name           = "uks-hub-vnet-01"
    vnet_address_space  = ["10.0.50.0/24"]
    vnet_location       = "UK South"
    vnet_resource_group = "first_uks_rg"
  }
  # UK South First Spoke VNet
  uks_spoke_vnet_1 = {
    vnet_name           = "uks-spoke-vnet-01"
    vnet_address_space  = ["10.0.0.0/24"]
    vnet_location       = "UK South"
    vnet_resource_group = "second_uks_rg"
  }
}

Like with resource groups, if we want another VNet, we just add another entry to the array. Last up is our main vnets .tf file and changing the module block to be a for / each loop. As with our resource groups module, we’ll give it a sensible name, point to the module source, define that it’s a for / each loop, add each.value for each of the values required by the module itself, and add the dependency on the resource groups module running so that we can get the resource group name. The only difference will be the format of the resource_group_name value. If we use the value that we’ve given in .tfvars for the name, we’ll just end up with the array reference, rather than the name value we actually want to use. The answer is to ask the resource group module block what name was given to the resource group it created for that array reference:

# VNets For / Each loop
module "vnets" {
  source              = "./modules/vnets"
  for_each            = var.vnets
  vnet_name           = each.value.vnet_name
  vnet_address_space  = each.value.vnet_address_space
  vnet_location       = each.value.vnet_location
  resource_group_name = module.resource_groups[each.value.vnet_resource_group].resource_group_name
  depends_on          = [module.resource_groups]
}

To try and clarify that a little bit, the value for “resource_group_name” passed down to the vnets module breaks down as:

• module.resource_groups – we will use the output of the resource_groups module block

• [each.value.vnet_resource_group]. – for every vnet instance in our vnets array, put the “vnet_resource_group” value here

• resource_group_name – give me the name that is output from the resource group creation module for the above value

OK, you should now be in a position to save all your files, check formatting, commit and sync, then check your validation pipeline output again. This time you’ll notice that it wants to destroy your resource group and create two new ones, and destroy both your VNets and again create two new ones, one in each resource group. If you are happy with your plan, pull to main, make sure your apply pipeline runs OK, then see what the deployment looks like!

We’ve now got even less project / client-specific data in our code. Our variables .tf only has the array names, our deployment .tf files just contain a single for / each loop, and any changes we want to make to the number of our resource groups or VNets is a simple amendment to our .tfvars file.

Formatted Naming

Before we move on, there’s something else we can do to prevent repeating ourselves in various array values, potentially adding typos, or deviating from our own standards. Every resource we create has a name, and ideally, we want that name to meet a set of standards or a naming convention. That means the name will normally be built up from things such as the region, the resource type, the environment etc. Let’s take our resource groups. Our first one is called “uks-example-rg-01”, which consists of:

• Region
• Descriptive Name
• Resource Type
• Unique numerical value

So the region’s going to be repeatable, the resource type’s going to be repeatable, and the numerical value seems to be in a consistent pattern. We also have a bunch of hyphens between each segment, so they’re repeated. What can we do to help ourselves here? We can use the “format” function of Terraform to build up our names when creating the resources, from values that are in our .tfvars file. Terraform has a few built-in functions, of which “format” is one. First of all, let’s put the values we want into our .tfvars file. I’m going to use the values above, but to help with our next post, I want to include the environment (production) in the name too. To break the name of the first resource group down into its individual components, we need to give each component a name and a value, so:

  first_uks_rg = {
    rg_name     = "uks-example-rg-01"
    rg_location = "UK South"
  }

becomes:

  first_uks_rg = {
    rg_name          = "example"
    rg_location      = "UK South"
    rg_region_prefix = "uks"
    rg_environment   = "prd"
    rg_id_suffix     = "01"
  }

Note that as our loop requires us to have all array components the same, I’ve also updated the code for our second resource group. You might also notice that I’ve not included the hyphens or the resource type (“rg”) here; these will come in our format command. We can now save the .tfvars file and open our resource groups .tf file in our root folder. Our “rg_name” value is currently getting the “rg_name” value directly from each array entry. We want to build that up to include all our values, in a command that has this structure:

resource_name = format("layout of name", value1, value2...)

So we first need to build out the layout of the name. I want it to be:

region-environment-descriptiveName-rg-uniqueValue

So we will be pulling values from our array for region, environment, descriptiveName and the uniqueValue, which we’ll represent in our name layout as strings, so as “%s”. This gives our actual name layout to be:

%s-%s-%s-rg-%s

In English, that would be:

<string>-<string>-<string>-rg-<string>

So each of our values being imported as strings will be separated by hyphens, and the last one having -rg- put in front of it. Next in our format command, we put a comma, then we will list the actual value names to bring in as strings, in the order they will be used. That means our format function becomes:

rg_name     = format("%s-%s-%s-rg-%s", each.value.rg_region_prefix, each.value.rg_environment, each.value.rg_name, each.value.rg_id_suffix)

There are four strings that will be pulled in, in order, each replacing one of the “%s” values in the layout of the name. Save all your files, check your formatting, commit and sync, then view the output of the plan job in your validation pipeline. Check the names it will give to your resource groups:

Have a play with different layouts for your resource names, see how you can build them in different ways. If for example you’re only going to deploy to a single region, why have that as a value in your array? Why not just hard format that in your format command in your for / each loop? Figure out what works for you, and your project or client. Different projects might need different naming conventions or layouts, and if you understand how the format command is built, you can quickly change the structure.

Summary

For / Each loops can be a daunting concept, but they are really powerful and allow us to really tighten up our code. They also give us the opportunity to play around with different naming formats and conventions, making the editing of code blocks that you’ve copied and pasted in your .tfvars files easier. While there’s obviously a whole lot more to learn about Terraform and Pipelines, we’re coming to the end of this short series. In our next post we’ll look at adding a different environment to our pipelines and expanding our VNet creation module to include some other parameters, making it more powerful and frankly, justifying its existence over a resource block!

Until next time…

– The Zoo Keeper