Developers looking for efficiency and automation to simplify project setup and deployment is (supposed to be) a “need” we already know about. Recognizing this need, we introduced a a functionality in Digger, tailored to enhance project initialization and setup for Terraform projects.

This functionality, known as generate_projects, emerged from a real-world user query that highlighted the necessity for dynamism and flexibility in managing complex project structures.

The Genesis of generate_projects

The journey began when a user approached our team with a seemingly unconventional request. They sought a mechanism within Digger that could automatically generate Terraform projects based on their existing directory structure.

Initially, the suggestion appeared unconventional to us maintainers. Conventional wisdom advocates for explicitly populating projects in configuration files, like digger.yml, to minimize errors and ensure safety.

This user’s situation clearly showed the range of challenges developers deal with (you don’t know what you don’t know, as they say). They managed a complex folder hierarchy and aimed to alleviate their team’s burden of manually editing the digger.yml for each new project addition.

The Solution

Driven by the user’s needs, we decided to accommodate this request, as we realised it would definitely end up enhancing Digger’s functionality. And, boy it has. It is one of the most used features of Digger and piques the interest of most people stumbling on it for the first time. You should join our slack to see what they think, btw.

Coming back to the solution — generate_projects is designed to traverse a specified directory, identifying and dynamically generating projects based on the provided include_patterns and exclude_patterns. This approach significantly reduces manual effort, especially in environments characterized by a multitude of small state files.

How It Works

To leverage this feature, users can define the generate_projects directive in their digger.yml as follows:

generate_projects:
  include: "projects/dev/*"
  exclude: "projects/dev/test3"

Advanced Configuration: Blocks Syntax

For more granular control, multiple include/exclude patterns can be specified using the blocks syntax. This allows for the definition of distinct rules for different sections of the project directory, each potentially associated with a specific workflow:

yamlCopy code
generate_projects:
  blocks:
    - include: "environments/core/**"
      workflow: default_work_flow
    - include: "environments/cicd/**"
      workflow: default_work_flow
    - include: "environments/dev/**"
      workflow: default_work_flow
    - include: "environments/qa/**"
      workflow: default_work_flow

Traversing Nested Directories

An optional traverse_to_nested_projects argument can be set to true, enabling Digger to generate a project for each sub-directory within the specified path. This is particularly useful for deeply nested project structures:

yamlCopy code
generate_projects:
  blocks:
    - include: "environments/core/**"
      workflow: default_work_flow
    - include: "environments/cicd/**"
      workflow: default_work_flow
    - include: "environments/dev/**"
      workflow: default_work_flow
    - include: "environments/qa/**"
      workflow: default_work_flow

In Practice: A Demonstration

To showcase the practical application and benefits of the generate_projects functionality, we've prepared a demo repository. This repository serves as a hands-on guide for users to understand how Digger dynamically generates Terraform projects, simplifying the initial setup and ongoing management of project infrastructures. Do kick the tires and let us know what you think!

Digger

Thank you for reading until the end!

• We’re building an Open Source Tool that helps you orchestrate Terraform and OpenTofu within CI/CD systems such as GitHub Actions while providing RBAC via OPA, Drift Detection and Concurrency with a self hostable orchestrator backend. We would love your feedback!
• Star us on GitHub | Check out Docs | Blog | Slack