Configuring AWS S3 and DynamoDB for Terraform state management

RECAP: Understanding Terraform and state management

By default, Terraform stores its state file in the local directory where it was run. While this is fine for personal projects, it becomes problematic when multiple developers collaborate on the same Terraform project leading to concurrency and versioning issues.

Storing the Terraform state in the cloud is highly recommended, leveraging services like S3 and DynamoDB. S3 securely stores the project’s state file, while DynamoDB ensures exclusive access, preventing concurrent modifications by multiple individuals through state locking — meaning the infrastructure can not be applied until the previous operation has completed and unlocked the state.

This article will cover configuring AWS S3 and DynamoDB for production-like state management, covering setup through the AWS console, AWS CLI, and Terraform.

Step 1: Create an S3 Bucket for state storage and a DynamoDB Table for state locking

(Option 1) Manual setup using the AWS Console

Create S3 bucket:

Create an S3 bucket using AWS console

1. Navigate to AWS Management Console > S3.
2. Click “Create bucket”.
3. Name your bucket following naming conventions (e.g., demo-terraform-state).
4. Configure settings:

• Disable ACLs (access control lists)
• Block all public access
• Enable bucket versioning
• Enable default encryption using SSE-S3

Create DynamoDB table:

Create a DynamoDB table using AWS console

1. Navigate to AWS Management Console > DynamoDB > Tables.
2. Click “Create table”.
3. Name your table (e.g., demo-terraform-state-lock).
4. Set a primary key (e.g., LockID).

(Option 2) Setup using AWS CLI

Assuming you have the AWS CLI installed and credentials configured, follow these commands:

Create S3 bucket:

1. Create bucket

aws s3api create-bucket \
    --bucket demo-terraform-state-eu-west-1 \
    --create-bucket-configuration LocationConstraint=eu-west-1

2. Configure settings:

• Disable ACLs (access control lists) and block all public access

aws s3api put-public-access-block \
    --bucket demo-terraform-state-eu-west-1 \
    --public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

• Enable bucket versioning

aws s3api put-bucket-versioning \
    --bucket demo-terraform-state-eu-west-1 \
    --versioning-configuration Status=Enabled

• Enable encryption using SSE-S3 (this should already be the default setting when creating the bucket)

aws s3api put-bucket-encryption \
    --bucket demo-terraform-state-eu-west-1 \
    --server-side-encryption-configuration '{"Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]}'

Create DynamoDB table:

aws dynamodb create-table \
    --region eu-west-1 \
    --table-name demo-terraform-state-lock \
    --attribute-definitions AttributeName=LockID,AttributeType=S \
    --key-schema AttributeName=LockID,KeyType=HASH \
    --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

Official AWS CLI documentation for the above steps can be found here.

(Option 3) Setup using Terraform

You can also create your S3 bucket and DynamoDB using Terraform. It is not recommended to have these configurations be part of your project’s Terraform configurations. Instead, create a separate project or repository dedicated to deploying your state resources, distinct from your main project repository.

Step 2: Configure Terraform backend

Now, configure your Terraform backend in your project:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.13.0"
    }
  }

  # Configure Terraform backend
  backend "s3" {
    bucket          = "demo-terraform-state-eu-west-1"
    key             = "serverless-backend/terraform.tfstate" # Where "serverless-backend" is the project name
    region          = "eu-west-1"
    encrypt         = true
    dynamodb_table  = "demo-terraform-state-lock"
  }
}

provider "aws" {
  region = "eu-west-1"
}

Step 3: Initialize the backend

Run terraform init to set up your backend. Then, execute terraform plan and terraform apply to deploy your project resources. Your state files will be stored in S3.

Additional considerations for Terraform backend

If you have multiple Terraform projects in the same AWS account, such as separate backend and frontend projects, you can optimise your setup by configuring both projects’ Terraform backends to use the same S3 state bucket and DynamoDB table. Each project should use a different bucket key within this bucket to preserve individual state files.

Always consider the potential cost implications of using AWS services for state management. The free tier offers most resources needed for personal serverless projects. If you have concerns, visit the official AWS website.