How to run Terraform in your Gitlab CI/CD pipeline?

This is something that we get asked very often. Integrating Terraform into GitLab CI clearly offers a streamlined approach to manage and automate infrastructure deployment, and infra engineers are always curious and keen to POC this to their peers.

The reason is clear - cloud credentials never leave the privileged environment of your CI, and you don’t overpay for private runners in custom “rip and replace” terraform CI’s. This guide, therefore, covers the necessary steps to run Terraform within GitLab CI, including an optional method to utilize GitLab's HTTP backend for Terraform state.

Prerequisites

• A GitLab account with CI/CD pipeline access.
• A Terraform project.

Steps involved

• Setting Up the GitLab CI/CD Pipeline
• Planning Infrastructure Changes
• Applying Infrastructure Changes
• Managing Terraform State
• Secret management - Securing Sensitive Data

Setting Up the GitLab CI/CD Pipeline

Create a .gitlab-ci.yml file in your Terraform project's root directory. This file defines the CI/CD pipeline in GitLab.

stages:
  - validate
  - plan
  - apply

terraform:
  image:
    name: hashicorp/terraform:latest
  stage: validate
  script:
    - terraform init
    - terraform validate

This setup introduces stages for validation, planning, and applying your Terraform configuration, using the official Terraform Docker image.

Planning Infrastructure Changes

Add a plan job to the .gitlab-ci.yml file to output the execution plan.

terraform_plan:
  stage: plan
  script:
    - terraform plan -out=tfplan
  artifacts:
    paths:
      - tfplan
  only:
    - branches

This job generates an execution plan and saves it as an artifact, configured to run only on branch pipelines.

Applying Infrastructure Changes

Implement an apply stage to deploy the planned changes, configured for manual execution for better control.

terraform_apply:
  stage: apply
  script:
    - terraform apply tfplan
  when: manual
  only:
    - master

This configuration ensures changes are reviewed and approved before deployment.

Managing Terraform State

Standard Remote Backend

Configure a remote backend like S3 for secure state management:

terraform {
  backend "s3" {
    bucket = "my-terraform-state-bucket"
    key    = "path/to/my/key"
    region = "us-east-1"
  }
}

Optional: GitLab's HTTP Backend

Alternatively, use GitLab's HTTP backend for centralized state management:

terraform {
  backend "http" {
    address        = "<https://gitlab.com/api/v4/projects/><PROJECT_ID>/terraform/state/<STATE_NAME>"
    lock_address   = "<https://gitlab.com/api/v4/projects/><PROJECT_ID>/terraform/state/<STATE_NAME>/lock"
    unlock_address = "<https://gitlab.com/api/v4/projects/><PROJECT_ID>/terraform/state/<STATE_NAME>/lock"
    username       = "gitlab-ci-token"
    password       = var.GITLAB_TOKEN
  }
}

Set GITLAB_TOKEN as an environment variable in GitLab for authentication.

Secret Management - Securing Sensitive Data

Handle sensitive data like cloud credentials using GitLab CI/CD environment variables:

provider "aws" {
  access_key = var.AWS_ACCESS_KEY_ID
  secret_key = var.AWS_SECRET_ACCESS_KEY
  region     = var.AWS_REGION
}

By following these steps, you can integrate Terraform with GitLab CI for efficient infrastructure management. This setup provides a scalable and secure approach to infrastructure as code, ensuring consistent and reliable deployment processes. Whether using a standard remote backend or GitLab's HTTP backend, the key is thorough testing and validation at each pipeline stage.

Digger - an orchestrator that enables RBAC, Drift Detection and Concurrency while running Terraform in your CI.

Digger is an Open Source Infrastructure as Code orchestration tool, that allows teams using Terraform/OpenTofu at scale to be able to enable role based access controls via Open Policy Agent, Drift detection (together with slack alerts) and concurrency, all while IaC is still running in your CI.

Incorporating Digger into the workflow detailed above enhances the automation and security aspects. Digger consists of two main components:

1. CLI Agent: This component runs within your CI environment and interacts directly with the Terraform (Upto 1.5.5)/OpenTofu CLI. It's responsible for executing Terraform commands and managing their output.
2. Orchestrator Backend: This component responds to events from Gitlab, triggering CI jobs as needed. When a merge request is opened, the orchestrator instructs Digger to start a CI job that runs terraform plan, posting the plan's output as a comment. Users can then comment “digger apply” to initiate terraform apply. Additionally, Digger offers flexibility in configuration, such as running apply post-MR merge, enforcing OPA policies, or performing drift detection on a schedule.

Crucially, the orchestrator backend of Digger does not have access to your cloud account, Terraform states, plan outputs, tfvars, or any other sensitive data.

It merely triggers CI jobs, ensuring that your sensitive data remains secure within the high-trust environment of your CI.

Combining Terraform/OpenTofu with GitLab CI and Digger offers a robust, secure, and efficient way to manage infrastructure deployments. This ensures that infrastructure management is not only automated but also aligns with the best practices of security and data privacy, keeping your VP eng/CTO happy! 😃

Note: Multi CI support is in active development and will be live shortly. Follow this blog which details out our design decisions. Feel free to message us on slack if you have any specific questions.