Terraform for Azure: Basics (7)

Terraform logo

Modules

Welcome (back?) to this, our seventh post in our series exploring the basics of using HashiCorp Terraform to deploy resources into Microsoft Azure using Azure DevOps. When writing these posts I make an assumption that you’re following along with me, your code is at the same level as mine and that generally means you’ve worked through the previous episodes:

If you’ve done that, you’ll have pipelines for validation and apply that are using a variable library, and you’ll have a repository with a main branch that’s protected from pushing code and a develop branch where code is pulled from to main.

The usual caveat here, I’m writing this series as a memory aid for myself with the hope that others can use it that might be on the same learning path as me. It’s not a detailed guide into Terraform, DevOps or Azure, it probably doesn’t meet all the best practices that a DevOps engineer would use, but it’s what I’ve found works for me when learning how something works. I’m writing in the middle of 2024, so by the time you read this things might have moved on a little, but I suspect the core concepts will remain the same. As always I’d like to credit my friend and colleague James Meegan for his original documentation which has been used as a foundation for the series.

This time we’re going to be looking at Terraform modules to try and simplify things and reduce the amount of lines in our main deployment code. It’s another long one I’m afraid, but hopefully it will really help some things gel in your mind. We will discuss what modules are and how to use them, then we’ll make all our code re-usable by moving the data into a separate variables file.

Modules

So what is a module? Let’s start by discussing how we might deploy a virtual machine in Azure using Terraform. Assuming our resource group and networking are already in place, we’d have a block of code for our network interface card and another block of code for the virtual machine itself. We might also have code blocks for VM extensions, or public IP addresses. That’s one VM. How many code blocks will we need for five VMs? Ten? You can see that our lines of code are going to be extensive, with a large number of code blocks and untidy files that are untidy and difficult to follow or manage.

Modules help us by pulling all these related blocks of code for a single resource into one place. For the VM example we might create a new Terraform file that holds a block of code for the VM, another for the NIC, and others for the extensions and public IPs. We can then call this new file from our main Terraform file, passing it values for each code block. If we’re deploying ten VMs, we just call that module ten times with unique values, rather than having ten examples of each of the code blocks in our main file. So what we’re doing is abstracting the code for each resource type into a separate file, making it easier to read, re-usable and consistent each time it’s called.

I want to note at this point that although it’s possible for you to configure your modules to call other modules, HashiCorp advise strongly against this so that the parent module is easier to re-use later on.

Once you’ve created a module for a specific purpose, you can share it with others that might have a use for it. If you put it in a public repository they can call it directly from their code, or they might take a copy and place it directly in their own repository. That means there are lots of modules already out there for you to do the same thing with! Have a look here for a list of modules that HashiCorp have published on their site. Although this series will have us writing and using our own modules, don’t try to reinvent the wheel; if you can find a module that does what you want it to do in a way that works for you, then use it. My goal is to give an understanding of the basics, and you can use those skills to either build your own modules, troubleshoot existing modules, or even improve on those. If you want more information on using modules, have a look at HashiCorp’s learning centre.

Storing Modules

Terraform uses the root folder of your repository (be that local or in Azure DevOps) as its working directory. This is where we currently have our providers.tf and main.tf, along with our pipelines. Terraform reads all .tf files in the working directory as a single file. We could have the providers.tf and main.tf data in a single file, but we separate them out to help compartmentalise each function and make it more readable to humans, but Terraform just lumps them all together as one.

Modules also use .tf files, but we don’t want these being automatically read as part of the main block, causing our pipeline to fail. We need to store the module files in a separate folder that is not the root. I find it good practice to create a “modules” folder in the root, with a separate folder for each actual module located under that. The name doesn’t matter, but for logical reading “modules” makes sense to me. Create your modules folder, either in Windows Explorer or VS Code:

Creating a Resource

OK, we’ve got an entry in our main.tf file for a resource group, that deploys the resource group to Azure as a single block of code. Let’s just add a resource into that resource group so we can see how we’d deploy without a module. A VNet is usually a good place to start. If we look at the HashiCorp website, we can get the code required to create a VNet. I’ve done that, popped it into main.tf and modified it a little to have my own address space and to reference the resource group I’ve already created. I’ve also added a couple of comments to my main.tf file:

# VNets
#  UK South Hub VNet
resource "azurerm_virtual_network" "uks_hub_vnet" {
  name                = "uks-hub-vnet-01"
  address_space       = ["10.0.50.0/24"]
  location            = "UK South"
  resource_group_name = azurerm_resource_group.first_uks_rg.name
}

So if we break that down, ignoring the comments we have our initial “resource” block that we created earlier to deploy the first resource group. We now have a new resource block that deploys an Azure Virtual Network. This resource block contains the following information:

  • name – the name of the VNet
  • address_space – the list of IP address ranges that form the address space of the VNet
  • location – the region in which the VNet is hosted
  • resource_group_name – the resource group that the VNet will be placed in. Note that I’ve referenced a resource block for “azurerm_resource_group”. I’ve given the name of that resource block (first_uks_rg), and I’m asking for the “name” component of that resource block to be my variable for the VNet resource block.

I could have just typed in “uks-example-rg-01” as the name of my resource group, but as we move through this series you’ll understand why it’s useful to reference the blocks that actually deploy the resources.

Save that file, check your formatting with terraform fmt -recursive, then commit and sync. Watch your validation pipeline job and see what the plan says it’s going to do:

Creating a Module

So that’s great, we know that resource blocks contain all the information we need to deploy a resource. So how do we create and use a module? Instead of using resource blocks, modules are called from module blocks, and they have this syntax:

module "module_name" {
  source "./modules/module_location"
  information_required_by_module
}

The resource block that we would normally use goes into the .tf file in the module’s own folder. So let’s create a “vnets” subfolder under “modules”:

In this folder, we’re going to create a .tf file. It can have any name, you can even call it main.tf if you want, but I like to make them logically readable, so I’m calling it vnets.tf:

Just to demonstrate the concept, let’s copy and slightly adjust the resource block that we created for the resource block in main.tf and paste it into our vnets.tf file. Note that you’d never normally put hard coded values into a modules file, this is just to understand the concept of the module itself. I’m making it a different VNet so am changing the name a little, I’m putting in the resource group’s actual name and I’m using a different address space. I’m also making the resource block’s name more generic:

resource "azurerm_virtual_network" "vnet" {
  name                = "uks-spoke-vnet-01"
  address_space       = ["10.0.0.0/24"]
  location            = "UK South"
  resource_group_name = "uks-example-rg-01"
}

If we saved and committed this to our repository now, nothing would happen – Terraform doesn’t know this module exists, or that we want to use it. We need a module block that our working directory can see (so in a .tf file in the root), with a source for that module. For now we’ll put that module block into our main.tf file. We’re just putting in the bare minimum for now to call the module, and its hard-coded values:

# UK South Spoke 1 VNet
module "uks_spoke_vnet_1" {
  source = "./modules/vnets"
}

Note that I’m just putting the “vnets” sub-folder in as a source – all files in that directory will be read, just as they would if they were in the root. This is done by the Terraform init process; when that command is run one of its tasks is to locate and pull down any modules that have been referenced in the root folder. When that process runs it will pick up the referenced module folder and sub-folders, and treat any .tf files contained therein as one. Save your files, check your formatting and synchronise to your repository. If you watch your validation pipeline’s job, you’ll see the VNets it’s planning on deploying:

So that’s how it works, which is straightforward enough, but we have those hard-coded values in our module. Like we’ve done previously in our pipelines, we can replace these with variables, which we’ll have to declare either in the same module .tf file, or preferably in a variables.tf file in the same folder, that will be treated as the same file by the Terraform init process but be more easily understood by the human eye. First of all let’s replace our values with variables in the vnets.tf file, commenting out the values so that we can use them later before deleting the comments:

resource "azurerm_virtual_network" "vnet" {
  name                = var.vnet_name               # "uks-spoke-vnet-01"
  address_space       = var.vnet_address_space      # ["10.0.0.0/24"]
  location            = var.vnet_location           # "UK South"
  resource_group_name = var.resource_group_name     # "uks-example-rg-01"
}

In the same folder as vnets.tf, we’ll create a variables.tf and declare each of those variables. This is the process of telling Terraform that we will be using variables, and adding information such as a description, or an optional default value. For now, we’re just adding a description to help when building our resource block in the module .tf file:

# VNet Variables
variable "vnet_name" {
    description = "Required field.  VNet name."
}

variable "vnet_address_space" {
    description = "Required field.  VNet address space."
}

variable "vnet_location" {
    description = "Required field.  VNet Region."
}

variable "resource_group_name" {
    description = "Required field.  VNet's Resource Group."
}

When we remove those commented values from our vnets.tf file, our module will be now completely abstracted and can be re-used many times either in this project or any other we care to put it in – we will just pass it the values needed each time we need to create a VNet. So how do we do that part? We simply modify the module block we created in our main.tf to send the values we want to those required variables:

# UK South Spoke 1 VNet
module "uks_spoke_vnet_1" {
  source = "./modules/vnets"
  vnet_name = "uks-spoke-vnet-01"
  vnet_address_space = ["10.0.0.0/24"]
  vnet_location = "UK South"
  resource_group_name = azurerm_resource_group.first_uks_rg.name
}

Note how I’ve used each of the variable names we declared, and have given those variables the values that we commented out in our module (except for the resource group name where I’ve again referred to the earlier module block). As they’re in place now, we can remove those comments from vnets.tf. Save all three of your files. Take a look at the screenshot above with the module block. After saving my files I’ve run “terraform fmt -recursive”. See the difference that is made to ensure all indentation and spacing is proper for Terraform:

It’s also corrected some of the indentation in my variables.tf file in my module. Have a look at your own and see what you notice! You can now commit and synchronise your code, then look at the output of the plan task of the pipeline job:

Let’s tidy things up a little more, and have the first VNet created using the module instead of as a resource block. While doing that, because we’re referring to the name of the resource group being created earlier, it’s a good idea to note that its resource block is a dependency, by adding a “depends on” line:

# VNets
#  UK South Hub VNet
module "uks_hub_vnet" {
  source              = "./modules/vnets"
  vnet_name           = "uks-hub-vnet-01"
  vnet_address_space  = ["10.0.50.0/24"]
  vnet_location       = "UK South"
  resource_group_name = azurerm_resource_group.first_uks_rg.name
  depends_on          = [azurerm_resource_group.first_uks_rg]
}

# UK South Spoke 1 VNet
module "uks_spoke_vnet_1" {
  source              = "./modules/vnets"
  vnet_name           = "uks-spoke-vnet-01"
  vnet_address_space  = ["10.0.0.0/24"]
  vnet_location       = "UK South"
  resource_group_name = azurerm_resource_group.first_uks_rg.name
  depends_on          = [azurerm_resource_group.first_uks_rg]
}

Abstracting the Variables

So now our module to create VNets is totally abstracted from the data that’s used in that creation, and we’re proving it can be re-used by calling that same module twice. But what about our main.tf? That still has values in it, so we can’t really use that on a different project. We can do the same as we did with our module code, and replace that data with variables that are held in a separate file. These variables will be put in a .tfvars file. We need to declare the variables again too, which we’ll do in a variables.tf file in the root. So first let’s declare all our variables in that variables.tf file – we can include the resource group variables as well as the VNet ones so that our main.tf will become completely clear of deployment specific code. Things can start getting complex initially in this file, so we’ll go in stages, and comment everything clearly, so that we get the logic of what’s going on. First of all, let’s declare our resource group variables:

###########################################
# Resource Group Variables
###########################################

#  First UK South Resource Group
variable "first_uks_rg_name" {
  description = "Required field.  Resource Group Name."
}
variable "first_uks_rg_location" {
  description = "Required field.  Resource Group Region."
}

You can see I’ve put a lovely big comment block at the top to help me identify what variables are in this section. I’ve then added another comment above the required variables because we will need to declare variables for every resource that is created (this is going to be easier in a later post, but right now we’re trying to get to grips with the logic of what’s happening). We’ll see this a bit more clearly with our VNet variables. We have two VNets, so we’ll have a nice big comment block to denote the new section, then the declaration of variables for each new VNet:

###########################################
# VNet Variables
###########################################

#  UK South Hub VNet Variables
variable "uks_hub_vnet_name" {
  description = "Required field.  VNet Name."
}
variable "uks_hub_vnet_address_space" {
  description = "Required field.  VNet Address Space."
}
variable "uks_hub_vnet_location" {
  description = "Required field.  VNet region."
}

# UK South Spoke 1 VNet Variables
variable "uks_spoke_vnet_1_name" {
  description = "Required field.  VNet Name."
}
variable "uks_spoke_vnet_1_address_space" {
  description = "Required field.  VNet Address Space."
}
variable "uks_spoke_vnet_1_location" {
  description = "Required field.  VNet region."
}

Now our variables are all declared, we need to store the actual key / value pairs in our terraform.tfvars file, also in the root. As before, let’s pop the values in for the resource group first:

###########################################
# Resource Groups
###########################################

#  First UK South Resource Group
first_uks_rg_name     = "uks-example-rg-01"
first_uks_rg_location = "UK South"

The values there are taken directly from the main.tf file, and the variable names are those that were declared in our variables.tf file. Now we do exactly the same for our VNet details, dropping them in the same terraform.tfvars file in our root:

###########################################
# VNets
###########################################

#  UK South Hub VNet
uks_hub_vnet_name          = "uks-hub-vnet-01"
uks_hub_vnet_address_space = ["10.0.50.0/24"]
uks_hub_vnet_location      = "UK South"

# UK South Spoke 1 VNet
uks_spoke_vnet_1_name          = "uks-spoke-vnet-01"
uks_spoke_vnet_1_address_space = ["10.0.0.0/24"]
uks_spoke_vnet_1_location      = "UK South"

All that’s left to do now is to update our main.tf to use the variables, instead of the specific data, and we’ve got some fully abstracted code that we can use for different projects or different clients, we just put in their specific names, IP addresses and locations in a new .tfvars file. Again, we’ll do our resource group first, amending our root main.tf file to point at the variables, in a “var.variableName” format:

# Resource Groups
#  First UK South Resource Group
resource "azurerm_resource_group" "first_uks_rg" {
  name     = var.first_uks_rg_name
  location = var.first_uks_rg_location
}

That’s relatively clean! Now let’s do the same for our VNets:

# VNets
#  UK South Hub VNet
module "uks_hub_vnet" {
  source              = "./modules/vnets"
  vnet_name           = var.uks_hub_vnet_name
  vnet_address_space  = var.uks_hub_vnet_address_space
  vnet_location       = var.uks_hub_vnet_location
  resource_group_name = azurerm_resource_group.first_uks_rg.name
  depends_on          = [azurerm_resource_group.first_uks_rg]
}

# UK South Spoke 1 VNet
module "uks_spoke_vnet_1" {
  source              = "./modules/vnets"
  vnet_name           = var.uks_spoke_vnet_1_name
  vnet_address_space  = var.uks_spoke_vnet_1_address_space
  vnet_location       = var.uks_spoke_vnet_1_location
  resource_group_name = azurerm_resource_group.first_uks_rg.name
  depends_on          = [azurerm_resource_group.first_uks_rg]
}

Note that we’ve not needed to declare the resource group name here. We’re referencing a resource block rather than using a variable, which is a different concept. But that’s it, you have all your code abstracted, your VNet is being created using a module and all your variables (which you could consider your data) are in a separate file. This seems like a good time to save your work, check your formatting, commit and sync, then pull to main and see your resources deployed:

Module Outputs

A resource group is a simple deployment, and as it wouldn’t normally be created with other objects (like say a VM and a NIC), we wouldn’t normally put it into a module. I’m going to do it here though, as both a practice of the concept, and to demonstrate the value of module outputs. HashiCorp discusses module outputs here, but for our example all we need to know is that once the module block in our main.tf file has created the resource group, we need to be able to refer to its name when we create our VNets, as we currently do with the resource block. We can’t just refer to the module block in the same way, because all it knows is that it’s passed some variables off to some sub-code in a different working directory. It doesn’t get any information back after that hand-off. If we want our code that’s running in root to use any information created in modules, such as a resource ID or as in this case, the name, we must instruct the module to pass the information back to the root code (or whatever called it) as an output.

Let’s create our resource groups module first. As before, we’ll create a new subfolder in our modules folder, and create a file for the resource blocks and a file for the variable declarations:

In our variables.tf file we need to declare all the variables we’ll use to create a resource group. Remember your variable names can be anything you want but they must be unique within the module:

# Resource Group Variables
variable "rg_name" {
  description = "Required field.  Resource Group name."
}

variable "rg_location" {
  description = "Required field.  Resource Group Region."
}

So that’s the variables declared, let’s get our resource block in the new resourceGroups.tf pointing at the variables we’ve just declared:

#  Module code to deploy a resource group

resource "azurerm_resource_group" "resource_group" {
  name     = var.rg_name
  location = var.rg_location
}

Now we need to add the code that outputs the created resource group’s name, allowing our main code in our root folder to use that:

output "resource_group_name" {
  value       = azurerm_resource_group.resource_group.name
  description = "The name of the created resource group"
}

We can output other information, such as the ID, the location or any tags, and they use the same format (check the HashiCorp website for supported outputs for each command), but for now we only want the name.

That’s our module created, so we can save those files and check the formatting. Next, we need to change our resource block in main.tf to a module block and point it to the module (using the “source” command) with the appropriate variables:

# Resource Groups
#  First UK South Resource Group
module "first_uks_rg" {
  source      = "./modules/resourceGroups"
  rg_name     = var.first_uks_rg_name
  rg_location = var.first_uks_rg_location
}

All that remains is for us to change the references in our VNet module blocks to use the module output of the resource group name rather than referencing the old resource block, and change the dependency from the resource block to the module block:

  resource_group_name = module.first_uks_rg.resource_group_name
  depends_on          = [module.first_uks_rg]

Once this is done, save all your files, check the formatting and commit and sync to your repository. In Azure DevOps, look at your pipeline’s run and check the “plan” step:

Note that the resource group that currently exists will be destroyed, and effectively recreated, just in a different way. The same thing will happen if you change a variable name too, which is one reason why it’s important to get your naming conventions right before you start. Although the plan doesn’t mention it, there are also your two VNets in that resource group (and any other resources you might have created outside this series), meaning (in certain circumstances) *they* will be deleted and recreated too!!! In our case, because they’re not mentioned, they won’t be deleted, and that unfortunately means that our resource group won’t be deleted, because it contains resources!

If you’re interested in how to allow your resources to be deleted and recreated if you’re changing or deleting a resource group, try pulling the code to main. When your apply stage of your apply pipeline fails, it gives you some pretty helpful instructions:

I’d advise against doing that, for the simple reason that humans make mistakes, and it’s nice to be fully in control if you’re doing something destructive. I think it’s better to delete your resources in the code along with the resource group change, then put them back after the change has been applied.

Deleting Resources

While we’re on the subject, we’ve not really discussed how to delete or destroy resources that we’ve created. In reality, it’s relatively simple. You just either delete their resource / module blocks in your root .tf file, or you can comment them out. Let’s try that and see how it changes our plan. In your main.tf, add a hash to the beginning of each line (comment out) for your spoke VNet. You can do this quickly in VS Code by selecting the whole block then pressing “<ctrl>+/”:

Save, check your formatting, commit and sync, then watch the pipeline job for the plan output:

So, to allow our resource group to be deleted and recreated, let’s comment out both our hub and spoke VNets, commit and sync, then pull to main. That should recreate the resource group and allow us to continue. Sometimes you’ll find that the deletion of the resource group takes a while and your apply stage times out, saying that the resource already exists. If this happens, just check your Azure portal, wait for the resource group to actually disappear (keep hitting that refresh button, I’m sure that speeds things up!), then run the failed job again. If you look at your apply pipeline’s jobs, you should see the option to re-run it. It’s really nice how helpful the information is in the pipeline task windows when there is a failure, and it’s so satisfying to see everything turn green when you’ve fixed it!

At this point, now that my resource group has been recreated, I’m going to uncomment the VNet section, sync my files up and pull my code into main again. The resource group and all its resources will be back in all their glory using methods that are getting us closer to best practice.

Summary

So, this has been a long post, but hopefully we’ve learned quite a bit. We now understand what modules are, how they can be used, and even where we can pick up ones that have already been created. We’ve learned about resource blocks and module blocks. We’ve learned how to declare variables and abstract those from the deployment files, meaning we have code and modules that can be re-used and shared. We now know how to delete resources, and what happens to existing resources if we change something small in the code.

How do you think you’re progressing? Is there anything I’m not going into enough depth on? How can I make these posts any better? I’d love to hear your views, please leave me a comment.

Until next time…

– The Zoo Keeper

By TheZooKeeper

An Azure Cloud Architect with a background in messaging and infrastructure (Wintel). Bearded dog parent who likes chocolate, doughnuts and Frank's RedHot sauce, but has not yet attempted to try all three in combination!