Investigate Why Terraform Drifted From Production

Overview

By the end of this tutorial, you'll learn how to use Stakpak to investigate Terraform drift in a live AWS environment, apply the fixes safely, validate that the infrastructure matches code again, and configure Stakpak Autopilotto help detect similar drift incidents automatically in the future.

Stakpak is open source, vendor neutral, and works with any model you choose.

Problem

You manage your AWS infrastructure with Terraform, and everything seems fine at first.

The code is in Git.
The state file is up to date.
The last terraform apply ran cleanly two weeks ago.
The application is serving traffic.

But a compliance scan just flagged your account for new "public S3" and "wide-open security group" findings, and you have no idea where they came from.

You check the workload:

curl -sS -o /dev/null -w "%{http_code}\n" http://northstar-prod-edge-alb-391835729.us-east-1.elb.amazonaws.com/

And the app returns 200. Traffic is fine. So you start the usual Terraform drift debugging loop:

terraform state list

terraform plan

aws ec2 describe-security-groups --filters "Name=vpc-id,Values=#### Vpc"

aws s3api get-bucket-policy --bucket #### Bucket

aws s3api get-public-access-block --bucket #### Bucket

aws iam list-role-policies --role-name #### Role

aws iam list-attached-role-policies --role-name #### Role

aws ec2 describe-instances --filters "Name=vpc-id,Values=#### Vpc"

aws cloudtrail lookup-events --max-results 50

Terraform plan shows Plan: 0 to add, 5 to change, 0 to destroy , but is that the whole story?

Is the plan output complete, or is it missing resources that were never in state
Are there resources running in AWS that Terraform doesn't know about at all
Did someone attach an IAM policy out of band that escalates blast radius?
Did someone modify an EC2 instance directly, and will it revert on the next ASG refresh?
Did someone open a public S3 path through a resource type your code doesn't manage?

Terraform gives you part of the picture. AWS gives you the rest. But you still have to connect them.

How Stakpak Helps?

Instead of manually cross referencing Terraform state, terraform plan output, AWS API responses, CloudTrail events, and resource tags across security groups, IAM roles, S3 buckets, EC2 instances, and load balancers, we can ask Stakpak to investigate the environment for us.

Stakpak inspects both sides of the drift, the Terraform code and state and the live AWS environment, connects the signals across the account, identifies every resource that has diverged (including the ones terraform plan cannot see), applies the fix to reconcile code and reality, and validates that the application stays healthy throughout.

Then, we'll configure Stakpak Autopilot to continuously monitor the AWS account and help detect similar Terraform drift incidents automatically in the future.

Application

The application is the public edge tier of the Northstar Commerce platform, running on AWS and provisioned with Terraform.

It runs as an Auto Scaling Group of EC2 instances behind an Application Load Balancer because it needs horizontal scaling, automatic replacement of unhealthy instances, and a stable public entrypoint.

Terraform is the source of truth for the stack, and AWS must reflect exactly what the Terraform code and state describe, otherwise the environment has drifted and Terraform can no longer safely manage it.

The main components are:

ALB: Public entrypoint that forwards HTTP traffic to healthy instances.
Auto Scaling Group: Maintains the desired number of EC2 instances and replaces unhealthy ones.
Launch Template: Defines the AMI, instance type, IAM profile, and security groups for every instance the ASG launches.
Security Groups: One for the ALB (HTTP from internet), one for the instances (HTTP from ALB only).
IAM role: Grants instances least-privilege access to SSM and the logs bucket.
S3 bucket: Stores edge access logs. Versioned, encrypted, and blocked from public access.
Terraform state: Records every managed resource and its expected configuration.

The normal management flow is: an engineer changes the Terraform code, CI runs terraform plan, the change is merged, and terraform apply reconciles AWS to match. Between applies, terraform plan should report No changes.

Now that we understand the app and architecture, we can start investigating the drift.

Step-by-Step Guide

Prerequisites

Install Stakpak
Cloud provider credentials configured

Troubleshooting

Open Stakpak and ask it to investigate the Terraform drift

Now lets let it do its magic

Stakpak started investigating Terraform drift in the northstar-edge module and traced the issue through the module structure, Terraform plan refresh, declared resource definitions, and the live AWS state of each drifted resource.

It found that 5 resources had drifted from their declared Terraform state, all from out of band changes (likely AWS console edits):