Don’t push your terraform state files

Automatically save them where everyone can find them, automatically !

Recently one of our colleagues left. He was generally VERY good with pushing his changes to VCS.

Until … The day he left.

This is when we realised that he did not push his latest changes and we had no state files and eventually resorted to rebuilding everything. Luckily this happened in a Dev environment.

What did we learn from this experience ?

DON’T USE GIT TO STORE YOUR STATE FILES !!!

What other option is there you ask ?

Terraform Backends: https://www.terraform.io/docs/backends/

What is this magic you speak of ?

It’s simple. Instead of storing state files in version control, terraform will store them in an S3 bucket, or an etcd cluster, or even just using plain old http.

It will recheck this file every time you do a terraform plan / apply etc. and make changes on that state, not the local state.

It will also lock state files, to prevent changes by multiple people at the same time etc.

Essentially: Everyone is using the same state files as though they exist on the same machine !

Let’s dive in

First we need somewhere to spin up stuff.

In this tutorial, we’ll be using DigitalOcean, simply because it’s simple and it’s cheap and has a great terraform provider.

If you already have a DigitalOcean account, skip over to creating a new personal access token

Create a DigitalOcean account if you don’t have one.

You can use this link to sign up: https://m.do.co/c/0545df410d16

The above is a referral link and will give you $50 free credit.

You are going to use like $0.5 for this tutorial and the rest is all yours !

Grab an API Token to use with Terraform

In your DigitalOcean cloud panel, in the left menu go to MANAGE > API.

There you will see a section called Personal Access Tokens.

Click Generate New Token, and enter a name. This token will be used by terraform, so name it something like terraform-testing. You can come back and delete it later.

Click Generate Token and a new token should be displayed in the table.

Copy this token now and store it somewhere safe. Once you move away from this page, you will not be able to see this token ever again. You will have to regenerate a new token if you lose this one.

Configure some infrastructure

Create a directory for you new infrastructure called tf-backends

mkdir -p tf-backends
cd tf-backends

Create a new config file called example.tf to create a small VM

# ./example.tf

# DO Token for provisioning infrastructure
variable "do_token" {}

# Configure the DigitalOcean Provider
provider "digitalocean" {
  token = "${var.do_token}"
}

# Configure a small VM
resource "digitalocean_droplet" "tiny-beast" {
  image  = "ubuntu-18-04-x64"
  name   = "tiny-beast"
  region = "ams3"
  size   = "s-1vcpu-1gb"
  ssh_keys = [] # SSH keys that need to be installed on the server.
  tags = ["tf-backends"]
}

Create a vars file for terraform, terraform.tfvars, where it can find your DigitalOcean token

do_token = "YOUR_TOKEN_HERE" 

Onto terraform:

First initialise terraform so it can install the necessary providers:

$ terraform init
Initializing the backend...
[...]
Terraform has been successfully initialized!
[...]

You’re all setup to create infrastructure.

Run terraform plan to see what terraform is going to do:

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
[...]
Terraform will perform the following actions:

# digitalocean_droplet.tiny-beast will be created
[...]

Now go ahead and apply that. (don’t stress about pricing. The server above will cost you $5 a MONTH, never mind a few minutes.)

Run terraform apply , and at the prompt enter yes to create the infrastructure

$ terraform apply
An execution plan has been generated and is shown below.
[...]
Terraform will perform the following actions:

# digitalocean_droplet.tiny-beast will be created
Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

Enter a value: yes

digitalocean_droplet.tiny-beast: Creating...
[...]
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

All Done ! Onto the real topic of this blog post.

The state problem.

If you have a look in your directory now, you will see that terraform has created a state file, so it can remember what it created.

Next time you apply this, terraform will check your config changes against the state file, and decide what to change accordingly.

So what is the problem ?

If you run terraform plan now, you will se that it won’t change anything, because state = config.

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
[...]
No changes. Infrastructure is up-to-date.

But what happens when you delete the state file ?

$ rm -rf terraform.tfstate
$ terraform plan
Refreshing Terraform state in-memory prior to plan...
[...]
An execution plan has been generated and is shown below.
[...]
Terraform will perform the following actions:

# digitalocean_droplet.tiny-beast will be created

But it exists already though ?

This is exactly the problem. If another team member applies some changes and you don’t have the new state file, you will be overriding one another’s infrastructure changes.

Secondly, every time you make changes, you have to let the rest of the team know to pull the latest state files before they make changes. And what happens if both of you apply at the same time.

Now what if there were a solution?

Well, you’ve come to the right place.

Welcome to Terraform backends.

What does it do ? Essentially it stores state files in a central place and not locally. Meaning that your machine & your colleagues machine are both using the exact same state file. When you make changes, it will lock the state file, so your colleague cannot make changes until you are done & vice-versa.

Now let’s get your state files in a central place.

We are going to use DigitalOcean spaces to store our files

Create a DigitalOcean space for your terraform state

In the DigitalOcean Nav Bar, got to Create > Spaces

Choose an appropriate region.

Make sure Allow file listing is RESTRICTED

Choose a unique name for you space.

Save it !

Create a Spaces Access Key

Our space is now secure and you can’y just get files from it.

So we need an access key to check out the files.

Got to MANAGE > API > Spaces access keys

Click Generate New Key and call it terraform-testing

Store the secret as it will also disappear when you move away from this page

Setup Terraform to use your new Space to store state

Create a file in your tf-backends directory called backend.tf

terraform {
  # DigitalOcean uses the S3 spec. 
  backend "s3" {
    bucket = "[YOUR_DO_SPACE_NAME]"
    key    = "terraform-testing.tfstate"
    access_key = "[DO_SPACES_KEY]"
    secret_key = "[DO_SPACES_SECRET]"
    endpoint = "https://ams3.digitaloceanspaces.com"
    # DO uses the S3 format
    # eu-west-1 is used to pass TF validation
    # Region is ACTUALLY ams3 on DO
    region = "eu-west-1"
    # Deactivate a few checks as TF will attempt these against AWS
    skip_credentials_validation = true
    skip_get_ec2_platforms = true
    skip_requesting_account_id = true
    skip_metadata_api_check = true
  }
}

What are we doing here ?

We are essentially telling terraform to use a S3 backend and passing it the values it needs to store a state file there.

Run terraform init again to pull down the backend provider

$ terraform init
Initializing the backend...
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
[...]
Terraform has been successfully initialized!
[...]

We are now setup with a terraform backend !

The moment of truth

Run terraform apply again to generate the state file

$ terraform apply
An execution plan has been generated and is shown below.
[...]
Terraform will perform the following actions:

# digitalocean_droplet.tiny-beast will be created
[...]
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Now if you look at your directory, you’ll see no state file.

Why is there no state file ?!?!

Have you checked the space in DO ?

Your state files are no longer stored locally. They are no stored in a share space for all developers to reach in real time.

Even better, you can now do things like locking state etc.

Remember that S3 is not the only way to store state remotely, you can checkout https://www.terraform.io/docs/backends/types/index.html for even more !

Cleanup

Remember that .tfstate file we deleted ?

The server that was managing still exists. In your DO control panel, destroy that droplet.

Then run terraform destroy to have terraform delete all the resources it created.

Then delete the testing DO space, api token and spaces access key and recreate for your actual environment with better names.

Resources

For a detailed post on how to destroy resources using Terraform, checkout this post from SpaceLift by Sumeet Ninawe https://spacelift.io/blog/how-to-destroy-terraform-resources

Jack Roper also wrote a detailed post on Terraform init on the SpaceLift blog https://spacelift.io/blog/terraform-init