Best way to structure your Terraform projects — Part 2!
Part 2 on answers to the most asked questions to a DevOps Specialist!
Please refer to Part 1 of this blog here.
As mentioned in Part 1, the opinions provided in the blog are subjective and I wanted to write this article as an open discussion to not only share some of the approaches that I followed but also to invite people to suggest and discuss any other options and to get more feedback on the pros and cons listed.
Should I use a pipeline for Terraform deployments?
Definitely yes.
Create a pipeline with roles having the least access privileges. Restrict access to the pipeline as well. No manual updates should be allowed apart from this. This might make development slow, but eventually, this will ensure consistency. Enable auditability for this pipeline so that the changes can be traced.
How should I design the pipeline for deployment in Terraform?
One common complaint we get when we try this approach is that the pipeline quickly becomes bulky and that it becomes slower. To solve this, separate the Terraform code deployments into logical blocks and deploy them separately with a different pipeline. This will make the changes smaller, quicker and will make them easily testable.
The output of the smaller pipelines can be stored in a central repository or AWS SSM param and the integration pipeline should be built in such a way that it can refer to the output of the previous pipeline . For example, we can use the ‘data’ resource in the code to refer to these dependencies while creating the resources in the integration pipeline.
This is an oversimplified diagram depicting the above theory. Feel free to separate the pipelines in a logical approach that makes sense for the team.
Create a pipeline separately for dev folks that they can use for their local testing. Encourage testing via code to make it a practice.
Pros:
No need to run a big pipeline with the whole environment changes to make small changes like renaming the tags.
Quicker feedback loops and the issues in the code can be identified earlier.
Testing becomes simple.
Cons:
Makes the whole setup complicated and can cause mismatches in the logic chosen to separate the pipelines.
- For example, a team mate might choose to run vpc, subnets, gateways etc in a single pipeline and a new joiner to the team could create a separate pipeline for just creating route tables, which might get complicated in the long run.
- Need clear documentation that explains the logical approach chosen
How to effectively test my Terraform setup?
I have written two blogs on testing using Terratest and Checkov. Please refer to them for information on tools used and how to use them.
Yet another resource I constantly refer to for testing Terraform is this excellent video by Yevgeniy Brikman. It talks about different types of ways in which we can test Terraform and the tools that can be used for them.
Other recommended tools for testing:
- OPA to use pre-defined or custom policies to check your Terraform code against
- Tflint or Checkov for static testing
- Use Variable validation that comes in-built with Terraform for contract testing
- Important Note: One thing that stops a lot of people from adding tests to their setup is that they might have to learn a new programming language like Go or Ruby. With Terraform 1.6(Released on October 4,2023) , Hashicorp has introduced testing using HCL itself, which makes the unit and integration testing very easy. I am currently experimenting with it and will post the process and setup in my upcoming blog and link it here.
5. Use Terratest for Integration and End to End testing
There are n number of tools to perform these tasks and I would suggest the reader to choose a tool that works for them based on their own deciding factors.
What are the recommended tools to be used alongside Terraform?
Apart from the above-mentioned testing tools, I’ve also used/come across these tools for various use cases
- Git-Secret or Talisman to not commit sensitive data to a repository
- Infracost for getting an approximate break down on the costs that your Terraform setup might incur
- Terraform-docs to overcome the pesky work of writing documentation for your code
- Terragrunt for keeping your code DRY
Hearing a lot of news on Terraform licensing but I do not understand it fully. Can you explain how this would affect me?
There are two scenarios for this and I’ll try to explain them in simple terms.
If you are a DevOps Engineer or Infra Engineer in your company and you use Terraform to build your resources, the good news is, nothing is changed. You can continue doing so!
If you are working on a product using Terraform’s source code and created a wrapper on top of it and have monetized this product or planning to monetize it, you need to get the license from Terraform. This might affect the ecosystem of the tools surrounding Terraform and there could be changes in how you use them going forward.
Will the advent of AI create an impact in DevOps? Is there a chance of losing jobs in the future?
The answers are both ‘Yes’ and ‘No’. I’m working on a blog explaining the way in which we can use AI to our advantage in DevOps. Will be linking it once its done. There are use cases popping up everyday about how we can use AI in DevOps and I would suggest folks to subscribe to newsletters to learn further.
How to keep myself updated on what is happening in Tech? I am overwhelmed by the news around.
It is difficult but the following points could help
- Follow newsletters and subscriptions that resonates with you.
- Some of the newsletters I’ve subscribed as a DevOps Specialist are ByteByteGo, weekly.tf, System Design Newsletter by NK,Medium digests on the relevant topics, Let’s make cloud
For newbies, it is understandable that it might get overwhelming despite having these resources. The only way to step over that feeling is to take one step at a time. If all you could do that day is read one tweet by a tech person, stick to it. The information might help someday when you are coming up with a solution
