Table of contents
- Security Scanners for Terraform
- Introduction to Trivy
- Installing Trivy
- Scanning an Example Terraform Module
- Inspecting the Results
- Resolving the Issues
- Scanning Terraform Plans
- Running Trivy in CI
- Bonus for GitHub Users
- Summary
In this blog post we will look at securing an AWS Terraform configuration using Trivy to check for known security issues. We will explore different ways of using Trivy, integrating it into your CI pipelines, practical issues you might face and solutions to those issues to get you started with improving the security of your IaC codebases.
Terraform is a powerful tool with a thriving community that makes it easy to find ready-made modules and providers for practically any cloud platform or service that exposes an API. Also internally many companies have a great deal of modules available. One of the strengths of Terraform is that modules provide an abstraction. You don’t have to worry about what is underneath the module’s variables (interface); you provide the necessary values and off you go, but this might lead into some trouble security-wise. Especially in public cloud platforms, it’s easy to expose a VM, load balancer or an object storage bucket publicly to the internet, and when using an abstraction this can happen without you truly acknowledging it. If you are familiar with Terraform, you might say, “Well I will just review the plan before applying”. But if the module is presenting you a plan of creating/modifying 200 resources, are you really confident you can eyeball that information and catch a misconfiguration that would expose your infrastructure to an attacker?
At the time of writing there are around 16,000 modules available from HashiCorp’s public registry. In this post we will pick a couple of AWS modules and check for insecure configurations. For this “check”, we will use an open source tool called Trivy.
Security Scanners for Terraform#
One of the big upsides of maintaining your infrastructure using an IaC approach is the fact that your infrastructure can be analysed by static analysis tools since your infrastructure is in plain text files. We can analyse the infrastructure before creating any resources to get quick feedback on the security posture and fix any issues before deployment. The only problem is that there are so many tools! After trying few alternatives, however, I have settled on a favourite that is both easy to use and effective at finding issues with built-in checks. In the past this favourite tool was tfsec , but quite recently the development efforts of the Tfsec project have been migrated into the Trivy project. Thus, it’s time to move over to Trivy although it’s not specialised to Terraform like Tfsec was.
Worth noting that there are some great open-source alternatives to Trivy, but overall we have found Trivy to be both easy to use locally and to integrate into build pipelines.
There’s of course nobody stopping you from using multiple tools, and when you automate the checks, that might not be a big deal to implement in the end. However, the goal of this blog post is not to focus on comparing different tools. What we want to focus on is that you should use a tool like this to perform security checks on your Terraform code, and it’s quite trivial to accomplish in the end.
Introduction to Trivy#
Trivy is a Swiss army knife type of tool for security scanning of various types of artifacts and code. It can scan different targets such as your local filesystem or a container image from a container registry. It can also check for many kinds of security issues such as known vulnerabilities, exposed secrets and most relevant to this blog post; misconfigurations.
At the time of writing Trivy supports scanning of various IaC configurations such as Terraform, CloudFormation and Azure Resource Manager. So even if your organisation uses different tools across teams, Trivy might just be the right tool. Trivy comes with built-in checks for various cloud platforms and in this blog post we will only use the built-in checks, but you can also define your own custom checks/policies.
Trivy can also scan for secrets which you should also use in the IaC context, but this is not really specific to the Terraform use-case. I suggest looking into the Trivy documentation to discover all of it’s power beyond what I already covered here as this will naturally evolve over time.
Now it’s time to get our hands dirty and look at an example of how Trivy can save you from doing things that might put your organisation in jeopardy.
Installing Trivy#
For installation I suggest checking out the installation guide in the documentation that covers all supported platforms. But for a quick start, here are a couple of commands that work for most folks:
1brew install trivy
For Debian/Ubuntu:
1apt install trivy
For Windows:
1choco install trivy
There are also pre-built packages available for various Linux distros, or grab the binary from GitHub releases: https://github.com/aquasecurity/trivy/releases
I highly suggest verifying the signature when installing, especially when you are using Trivy in your production build pipelines.
Scanning an Example Terraform Module#
Let’s create an example Terraform root module in order to get something to point Trivy at. Like I mentioned earlier, there are many open-source modules for Terraform that we can utilise in order to quickly build infrastructure. The AWS modules are especially popular, so I thought let’s write an example by utilising a couple of these modules with mostly their default configuration. Here’s what I came up with:
1#main.tf
2terraform {
3 required_providers {
4 aws = {
5 source = "hashicorp/aws"
6 version = "~> 5.0"
7 }
8 }
9}
10
11locals {
12 common_tags = {
13 Terraform = "true"
14 Environment = "dev"
15 }
16}
17
18module "vpc" {
19 source = "terraform-aws-modules/vpc/aws"
20 version = "5.0.0"
21
22 name = "my-vpc"
23 cidr = "10.0.0.0/16"
24
25 azs = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
26 private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
27 public_subnets = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]
28
29 enable_nat_gateway = false
30
31 tags = local.common_tags
32}
33
34data "aws_ami" "amazon_linux" {
35 most_recent = true
36
37 owners = ["amazon"]
38
39 filter {
40 name = "name"
41
42 values = [
43 "amzn2-ami-hvm-*-x86_64-gp2",
44 ]
45 }
46
47 filter {
48 name = "owner-alias"
49
50 values = [
51 "amazon",
52 ]
53 }
54}
55
56resource "aws_instance" "this" {
57 ami = data.aws_ami.amazon_linux.id
58 instance_type = "t3.nano"
59 subnet_id = element(module.vpc.private_subnets, 0)
60}
61
62module "alb" {
63 source = "terraform-aws-modules/alb/aws"
64 version = "8.7.0"
65
66 name = "my-alb"
67
68 load_balancer_type = "application"
69
70 vpc_id = module.vpc.vpc_id
71 subnets = module.vpc.public_subnets
72
73 target_groups = [
74 {
75 name_prefix = "pref-"
76 backend_protocol = "HTTP"
77 backend_port = 80
78 target_type = "instance"
79 targets = {
80 my_ec2 = {
81 target_id = aws_instance.this.id
82 port = 8080
83 }
84 }
85 }
86 ]
87
88 http_tcp_listeners = [
89 {
90 port = 80
91 protocol = "HTTP"
92 target_group_index = 0
93 }
94 ]
95
96 tags = local.common_tags
97}
This configuration is ~100 LoC and it will create a VPC, an EC2 instance and an ALB. Naturally, the ALB also targets the EC2 instance. After creating the file, let’s initialise Terraform to download the external modules:
1terraform init
Simplest way to run a Trivy misconfiguration scan is to point it at your current folder:
1trivy config .
Like mentioned earlier, we can also scan for secrets at the same time with Trivy:
1trivy fs --scanners misconfig,secret .
Due to the focus on Terraform, I’ll use the config subcommand for the rest of the blog post, but in a CI pipeline I would run the secrets scanning definitely for the whole repository as well, not only in IaC folders.
Before showing the full results, I noticed there are some example configurations picked up by Trivy from the remote modules, such as this:
1HIGH: IAM policy document uses sensitive action 'logs:CreateLogStream' on wildcarded resource '*'
2═══════════════════════════════════════════════════════════════════════════════════════════════════════
3You should use the principle of least privilege when defining your IAM policies.
4This means you should specify each exact permission required without using wildcards,
5as this could cause the granting of access to certain undesired actions, resources and principals.
6
7See https://avd.aquasec.com/misconfig/avd-aws-0057
8───────────────────────────────────────────────────────────────────────────────────────────────────────
9 modules/vpc/vpc-flow-logs.tf:112
10 via modules/vpc/vpc-flow-logs.tf:100-113 (data.aws_iam_policy_document.vpc_flow_log_cloudwatch[0])
11 via modules/vpc/vpc-flow-logs.tf:97-114 (data.aws_iam_policy_document.vpc_flow_log_cloudwatch[0])
12 via modules/vpc/examples/complete/main.tf:25-82 (module.vpc)
13───────────────────────────────────────────────────────────────────────────────────────────────────────
14 97 data "aws_iam_policy_document" "vpc_flow_log_cloudwatch" {
15 ..
16 112 [ resources = ["*"]
17 ...
18 114 }
If you look closely you notice that the source of the finding is a main.tf file in the examples folder of the VPC module: via modules/vpc/examples/complete/main.tf:25-82 (module.vpc)
This is not really our configuration and we should not include these files in the scan. This is also a difference between Tfsec and Trivy, when running Tfsec it does not pickup the examples folder.
However, we can easily resolve this by skipping all files under examples folders and then we should get proper report of findings:
1trivy config . --skip-dirs '**/examples'
Here are the results:
1.terraform/modules/alb/main.tf (terraform)
2==========================================
3Tests: 3 (SUCCESSES: 1, FAILURES: 2, EXCEPTIONS: 0)
4Failures: 2 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 2, CRITICAL: 0)
5
6HIGH: Application load balancer is not set to drop invalid headers.
7════════════════════════════════════════════════════════════════════════════════════════════════
8Passing unknown or invalid headers through to the target poses a potential risk of compromise.
9
10By setting drop_invalid_header_fields to true, anything that doe not conform to well known,
11defined headers will be removed by the load balancer.
12
13See https://avd.aquasec.com/misconfig/avd-aws-0052
14────────────────────────────────────────────────────────────────────────────────────────────────
15 .terraform/modules/alb/main.tf:23
16 via .terraform/modules/alb/main.tf:5-63 (aws_lb.this[0])
17────────────────────────────────────────────────────────────────────────────────────────────────
18 5 resource "aws_lb" "this" {
19 .
20 23 [ drop_invalid_header_fields = var.drop_invalid_header_fields
21 ..
22 63 }
23────────────────────────────────────────────────────────────────────────────────────────────────
24
25HIGH: Load balancer is exposed publicly.
26════════════════════════════════════════════════════════════════════════════════════════════════
27There are many scenarios in which you would want to expose a load balancer to the wider internet,
28but this check exists as a warning to prevent accidental exposure of internal assets.
29You should ensure that this resource should be exposed publicly.
30
31See https://avd.aquasec.com/misconfig/avd-aws-0053
32────────────────────────────────────────────────────────────────────────────────────────────────
33 .terraform/modules/alb/main.tf:12
34 via .terraform/modules/alb/main.tf:5-63 (aws_lb.this[0])
35────────────────────────────────────────────────────────────────────────────────────────────────
36 5 resource "aws_lb" "this" {
37 .
38 12 [ internal = var.internal
39 ..
40 63 }
41────────────────────────────────────────────────────────────────────────────────────────────────
42
43.terraform/modules/vpc/main.tf (terraform)
44==========================================
45Tests: 1 (SUCCESSES: 0, FAILURES: 1, EXCEPTIONS: 0)
46Failures: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 0, CRITICAL: 0)
47
48MEDIUM: VPC Flow Logs is not enabled for VPC
49════════════════════════════════════════════════════════════════════════════════════════════════
50VPC Flow Logs provide visibility into network traffic that traverses the VPC and can be used to
51detect anomalous traffic or insight during security workflows.
52
53See https://avd.aquasec.com/misconfig/avd-aws-0178
54────────────────────────────────────────────────────────────────────────────────────────────────
55 .terraform/modules/vpc/main.tf:29-52
56────────────────────────────────────────────────────────────────────────────────────────────────
57 29 ┌ resource "aws_vpc" "this" {
58 30 │ count = local.create_vpc ? 1 : 0
59 31 │
60 32 │ cidr_block = var.use_ipam_pool ? null : var.cidr
61 33 │ ipv4_ipam_pool_id = var.ipv4_ipam_pool_id
62 34 │ ipv4_netmask_length = var.ipv4_netmask_length
63 35 │
64 36 │ assign_generated_ipv6_cidr_block = var.enable_ipv6 && !var.use_ipam_pool ? true : null
65 37 └ ipv6_cidr_block = var.ipv6_cidr
66 ..
67────────────────────────────────────────────────────────────────────────────────────────────────
68
69main.tf (terraform)
70===================
71Tests: 3 (SUCCESSES: 1, FAILURES: 2, EXCEPTIONS: 0)
72Failures: 2 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 2, CRITICAL: 0)
73
74HIGH: Instance does not require IMDS access to require a token
75════════════════════════════════════════════════════════════════════════════════════════════════
76
77IMDS v2 (Instance Metadata Service) introduced session authentication tokens which improve
78security when talking to IMDS.
79By default <code>aws_instance</code> resource sets IMDS session auth tokens to be optional.
80To fully protect IMDS you need to enable session tokens by using <code>metadata_options</code>
81block and its <code>http_tokens</code> variable set to <code>required</code>.
82
83See https://avd.aquasec.com/misconfig/avd-aws-0028
84────────────────────────────────────────────────────────────────────────────────────────────────
85 main.tf:55-59
86────────────────────────────────────────────────────────────────────────────────────────────────
87 55 ┌ resource "aws_instance" "this" {
88 56 │ ami = data.aws_ami.amazon_linux.id
89 57 │ instance_type = "t3.nano"
90 58 │ subnet_id = element(module.vpc.private_subnets, 0)
91 59 └ }
92────────────────────────────────────────────────────────────────────────────────────────────────
93
94HIGH: Root block device is not encrypted.
95════════════════════════════════════════════════════════════════════════════════════════════════
96Block devices should be encrypted to ensure sensitive data is held securely at rest.
97
98See https://avd.aquasec.com/misconfig/avd-aws-0131
99────────────────────────────────────────────────────────────────────────────────────────────────
100 main.tf:55-59
101────────────────────────────────────────────────────────────────────────────────────────────────
102 55 ┌ resource "aws_instance" "this" {
103 56 │ ami = data.aws_ami.amazon_linux.id
104 57 │ instance_type = "t3.nano"
105 58 │ subnet_id = element(module.vpc.private_subnets, 0)
106 59 └ }
107────────────────────────────────────────────────────────────────────────────────────────────────
For brevity I shortened some of the long lines.
Inspecting the Results#
Unfortunately Trivy does not print a summary in the end like tfsec does which makes it nice to read the output from bottom to top. Trivy does offer different ways to modify the resulting report, but for the needs of this blog I quickly used grep to find a short summary of each finding:
1HIGH: Application load balancer is not set to drop invalid headers.
2HIGH: Load balancer is exposed publicly.
3MEDIUM: VPC Flow Logs is not enabled for VPC
4HIGH: Instance does not require IMDS access to require a token
5HIGH: Root block device is not encrypted.
Looking at the list, I think we want to change the configuration before deploying the resources. Next, let’s look at our options for resolving these findings.
Resolving the Issues#
We have two choices when it comes to resolving these issues so that we can have a nice clean report (until we change the configuration again). We can either resolve the issues by modifying our configuration or we can choose to accept the finding as something that is not relevant for our requirements and ignore the findings for future scans.
Since we use the public AWS modules, we cannot easily make changes besides the inputs without forking the source module, however we can change the EC2 instance which is defined directly in the main.tf . So let’s look into the issues related to the EC2 instance first.
There is a finding related to the AWS Instance Metadata Service (IMDS). The finding is related to making sure the instance uses the IMDSv2 instead of the legacy IMDSv1. Looking at the full report above, you can see that Trivy explicitly tells us what the problem is and how to resolve it:
1IMDS v2 (Instance Metadata Service) introduced session authentication tokens which improve
2security when talking to IMDS.
3By default <code>aws_instance</code> resource sets IMDS session auth tokens to be optional.
4To fully protect IMDS you need to enable session tokens by using <code>metadata_options</code>
5block and its <code>http_tokens</code> variable set to <code>required</code>.
You can read more of the security benefits and scenarios where this configuration matters in the IMDSv2 announcement blog post by AWS.
To resolve this, we are going to change the configuration in the following way, like the description suggested:
1resource "aws_instance" "this" {
2 ami = data.aws_ami.amazon_linux.id
3 instance_type = "t3.nano"
4 subnet_id = element(module.vpc.private_subnets, 0)
5+
6+ metadata_options {
7+ http_tokens = "required"
8+ }
9 }
If you run the scan again you will notice the finding is gone:
1trivy config . --skip-dirs '**/examples'
Let’s move onto the next issue that is related to the root disk being unencrypted for this EC2 instance. In reality I would configure encryption because it is a common compliancy requirement and AWS makes it very easy, but for the sake of the example let’s ignore this instead, saying that we are ok with running unencrypted root disk:
1+#trivy:ignore:avd-aws-0131
2 resource "aws_instance" "this" {
3 ami = data.aws_ami.amazon_linux.id
4 instance_type = "t3.nano"
5 subnet_id = element(module.vpc.private_subnets, 0)
6
7 metadata_options {
8 http_tokens = "required"
9 }
10 }
Using the inline method for ignoring findings is the most intuitive way in my opinion, this might be familiar to you if you have worked with just about any code linter in the past.
Now the only remaining issues are related to the AWS modules which we did not author. Unfortunately, right now Trivy can’t figure out that the remote modules are downloaded under different path (.terraform/modules) than what is declared when specifying the source for the modules:
1module "alb" {
2 source = "terraform-aws-modules/alb/aws"
3 version = "8.7.0"
4...
Again, not an issue when using local modules since the paths nicely match between the findings report and the declaration in the Terraform configuration. This issue is actively being discussed in the Trivy GitHub repository, so when you read this it might be fixed and I should update this blog.
Luckily Trivy has a cure for this even without us waiting for a fix. I quickly brewed a solution using an advanced filtering mechanism in Trivy that uses the Rego language:
package trivy
import data.lib.trivy
default ignore = false
ignore_avdid := {"AVD-AWS-0052", "AVD-AWS-0053"}
ignore_severities := {"LOW", "MEDIUM"}
ignore {
input.AVDID == ignore_avdid[_]
}
ignore {
input.Severity == ignore_severities[_]
}
This will resolve the issue with the VPC module because all MEDIUM findings are ignored, and the findings in the ALB module are ignored explicitly by the finding IDs.
Now we can run a scan and include this policy from a local file:
1trivy config . --skip-dirs '**/examples' --ignore-policy custom-policy.rego
Now all the findings should be resolved (of course if you try this yourself, there might be new built-in checks and you get a different list of findings).
While authoring the custom ignore policy, I couldn’t figure out how to connect the source of the finding (the ALB module) to the AVDID for a more fine-grained rule, but that does not seem like a big deal to me. You should not place all your IaC configuration into a single root module after all, in production I would separate the VPC creation from this Terraform root module and use an existing one that is part of a different root module.
Scanning Terraform Plans#
Another way to run the scan is to first create a plan and then convert the plan from the default binary format to JSON, and then point Trivy to scan this plan that contains a list of changes:
1terraform plan --out tf.plan
2terraform show -json tf.plan > tfplan.json
3trivy config tfplan.json
I noticed that there’s one additional finding when running Trivy against the plan:
1CRITICAL: Listener for application load balancer does not use HTTPS.
2═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
3Plain HTTP is unencrypted and human-readable. This means that if a malicious actor was to eavesdrop on your connection,
4they would be able to see all of your data flowing back and forth.
5
6You should use HTTPS, which is HTTP over an encrypted (TLS) connection, meaning eavesdroppers cannot read your traffic.
7
8See https://avd.aquasec.com/misconfig/avd-aws-0054
9──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
10 main.tf:36
11 via main.tf:34-48 (aws_lb_listener.frontend_http_tcp_ffdb4db32d85be4b5cd7539e4d3c6d16)
12──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
13 ..
14 36 [ protocol = "HTTP"
15 ..
16 48 }
17────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
I found it odd that this was not found in the previous scan and after some testing this seems to be caused by using remote modules (again!). I also noticed that running tfsec instead of Trivy will catch this issue. When using local modules Trivy can right away pick up this finding. Right now it seems necessary to scan both your plan and your configuration for the best accuracy, but I might revisit this in the future and see if the issue is resolved given the active discussion around remote modules support in Trivy.
Luckily, we can work around this also by scanning everything at once, so generate the plan and then scan the entire folder (then Trivy processes the configuration AND the plan):
1terraform plan --out tf.plan
2terraform show -json tf.plan > tfplan.json
3trivy config . --skip-dirs '**/examples'
Trivy is smart enough to not duplicate the findings - awesome! With Tfsec it was not possible to scan Terraform plans, so this is great if scanning the plan fits your workflow better.
As we saw above, scanning the Terraform plan is more accurate than scanning just the files, but the downside is that you need to generate a plan for each Terraform root module and it takes much more time to generate a plan prior to running Trivy. You also need to be able to connect and authenticate to the providers for Terraform to generate the plan, although that’s not typically an issue.
One more thing to note about plans is that your inline #trivy:ignore comments will be ignored since that information will not make it into the plan, so if you are using plans primarily for your scanning, then you might need to get comfortable defining the Rego ignore policies instead.
Running Trivy in CI#
Including Trivy scans in your IaC repositories’ CI pipelines is a must. If you don’t have CI pipelines for your IaC… Well you should! Trivy offers integrations with many CI/CD tools, IDEs and other systems, see the documentation for an up-to-date list.
When using Trivy in CI it’s wise to use a configuration file instead of the command line flags, this makes it easy to reproduce the scan using same configuration locally if you need to investigate some new findings. If you are using GitHub Actions, there’s an official Action that you can use to integrate Trivy into your CI pipeline, here’s a simple example which uses a configuration file:
1name: build
2on:
3 push:
4 branches:
5 - main
6 pull_request:
7jobs:
8 build:
9 name: Build
10 runs-on: ubuntu-20.04
11 steps:
12 - name: Checkout code
13 uses: actions/checkout@v3
14
15 - name: Run Trivy vulnerability scanner in fs mode
16 uses: aquasecurity/trivy-action@master
17 with:
18 scan-type: 'fs'
19 scan-ref: '.'
20 trivy-config: trivy.yaml
As seen in the above example, in CI you likely want to run the fs scan which includes by default all the scanners, meaning Trivy will also scan for secrets and vulnerabilities, not only for misconfigurations.
However, keep in mind this excellent blog post by my colleague Thierry. The Trivy action only really wraps the GitHub workflow YAML inputs to CLI flags. If you are using another CI/CD system, you can simply install and invoke the CLI as well, making transition between CI/CD tools extremely simple.
Bonus for GitHub Users#
If you have an open-source project in GitHub or you pay GitHub for the advanced security features, then you can also upload the Trivy scan results into the GitHub code scanning which you should be using if you are not already. Refer to the Trivy Action’s README to view a sample configuration of uploading the results. This will help you to track the findings in addition to gatekeeping with a pipeline that must pass always before merging code (or however your team works).
Summary#
In this blog post we rolled up our sleeves and looked into how to secure Terraform configuration by using static analysis. As an example and recommended tool we explored Trivy, but honestly the tool choice isn’t as important as the principle of integrating such checks into your workflow. I hope you can see the value of running a simple scan over your configuration. Thanks to extensive builtin checks in Trivy you can get actionable findings without spending time reviewing the configuration manually and magically knowing all the security intricacies of AWS infrastructure.
If you found something wrong with the content or something felt vague or awesome, leave us a comment! Additionally, if you’d like any help with Terraform and/or Trivy please get in touch!
