Every Manual AWS Change Is Configuration Drift You'll Pay for Later

Introduction

The security group rule that got added "just for a few hours" to let a vendor's IP through during an integration test is still there fourteen months later. Nobody knows who added it. Nobody knows if the vendor still needs it. And it's now in the security group for a database tier that handles healthcare records.

Infrastructure configuration drift is the gap between what your IaC says exists and what actually exists in AWS. It accumulates through manual console changes made under pressure, through aws cli commands run in production to "just fix this quickly," through AWS services that mutate configuration on your behalf, Auto Scaling adjusting group sizes, RDS promoting a replica, ECS updating a service's running task count. Every drift between declared state and actual state is a liability. Some of it is benign. Some of it is a security group rule opened to the world that your Terraform never knew about. You often can't tell which until something goes wrong.

This post covers how configuration drift forms, why it's more dangerous than teams typically recognize, and the specific detection and prevention patterns that stop it from compounding.

How Configuration Drift Forms and Why It's Hard to See

Configuration drift almost never starts as a deliberate choice to bypass your IaC process. It starts as an emergency.

A service is down. An engineer needs to open a port to diagnose a connection issue. The Terraform change would take fifteen minutes to write, review, and apply. The console change takes thirty seconds. The port gets opened, the diagnosis happens, the issue gets fixed. Then the engineer gets pulled into the next thing. The security group rule stays.

That's the benign version. The malignant version looks similar from the outside: an engineer opens a port for what seems like a contained reason, and the resulting drift creates a network path that nobody's threat model accounted for, that no IaC review will ever catch, and that will exist until someone thinks to audit it.

What makes drift particularly dangerous in AWS is its invisibility. Unlike application code, infrastructure configuration doesn't produce compilation errors when it diverges from a specification. A security group with an extra ingress rule runs identically to a security group without one, right up until the moment that rule becomes the attack vector for a breach. An S3 bucket with a manually applied public access exception runs identically to a properly configured bucket, until someone accesses data they shouldn't.

The accumulation pattern is what elevates drift from an inconvenience to a risk category. Individually, each drift instance looks like a minor variance. Collectively, they represent an infrastructure whose actual security posture is unknown. Your Terraform says one thing. AWS says another. You don't know how large the gap is unless you specifically look.

The most common failure mode I've encountered: a team has solid Terraform coverage for their compute and networking layer, but their IAM policies have been modified directly in the console over two years of operational iteration. Engineers add permissions when they hit access denied errors, remove permissions they think are stale, create new roles for specific integrations. The IAM configuration that exists in AWS has diverged so far from the Terraform state that running terraform plan on the IAM module would produce hundreds of changes, and nobody is confident which changes are "correct" and which represent two years of operational decisions that would break things if reverted.

That's drift at the point where it becomes a remediation project rather than a maintenance task.

AWS Deep Dive: Detecting and Preventing Drift in Practice

AWS Config and CloudFormation Drift Detection

AWS provides two native drift detection mechanisms. CloudFormation drift detection compares the current state of CloudFormation-managed resources against the last-known stack configuration. AWS Config continuously records resource configuration and can alert on configuration changes that deviate from a desired state.

CloudFormation drift detection is straightforward to initiate but often under-used because teams don't run it on a schedule:

# Initiate drift detection on a specific stack
aws cloudformation detect-stack-drift \
  --stack-name my-production-stack

# Wait for completion and retrieve results
aws cloudformation describe-stack-drift-detection-status \
  --stack-drift-detection-id <detection-id>

# List drifted resources in the stack
aws cloudformation describe-stack-resource-drifts \
  --stack-name my-production-stack \
  --stack-resource-drift-status-filters MODIFIED DELETED

The non-obvious CloudFormation behavior: drift detection does not tell you who made the drift or when. It tells you that the current resource configuration doesn't match the last-deployed stack template. For timeline and attribution, you need CloudTrail. Correlating a CloudFormation drift finding with a CloudTrail event, filtering by the resource ARN and the time window before the drift detection, is the forensic process for understanding how drift got there.

AWS Config's approach is continuous rather than on-demand. Config records a configuration snapshot every time a supported resource changes, and Config Rules can evaluate those snapshots against desired-state conditions. The two Config Rules most relevant to drift detection:

• CLOUD_FORMATION_STACK_DRIFT_DETECTION_CHECK, alerts when a CloudFormation stack has drifted (requires drift detection to have been run; Config checks the existing drift status, it doesn't initiate new detection)
• Custom Config Rules via Lambda, for drift conditions that aren't covered by managed rules, such as "no security group in the database subnet should have any ingress rule with a source CIDR outside the VPC"

The important limitation: AWS Config only covers resources it supports, and the supported resource list, while extensive, does not include everything. Some newer services and some service-specific resource types don't have Config recording support. Verify coverage for your specific resource types in the AWS Config supported resources documentation before relying on Config as your sole drift detection mechanism.

Terraform State Drift Detection and the refresh Behavior

For Terraform-managed infrastructure, drift detection is a function of the gap between Terraform state and actual AWS resource configuration. terraform plan performs a refresh by default: it queries the current state of each resource in state and compares it to the configuration. Resources that have been manually modified will show as changes in the plan.

The workflow that makes drift visible on a schedule rather than only before a deployment:

# Run a plan with detailed exit codes in CI on a schedule
# Exit code 0 = no changes, 1 = error, 2 = changes detected
terraform plan -detailed-exitcode -out=drift-detection.tfplan

if [ $? -eq 2 ]; then
  echo "Drift detected: actual infrastructure differs from Terraform configuration"
  # Alert via SNS, Slack, PagerDuty, or your alerting channel
fi

The non-obvious Terraform behavior that trips teams on drift remediation: when Terraform detects drift and you run terraform apply, it will attempt to bring the resource back to the declared configuration, which means overwriting the manual change. If the manual change was deliberate and necessary (a security group rule that was added for a real production reason and never codified in Terraform), the apply will remove it. Before running terraform apply after drift detection, you must triage the drift: is this a change that should be preserved in Terraform state, or is it unauthorized drift that should be reverted?

The two categories require different responses:

Authorized drift (change was intentional, should be preserved):

# Import the current resource state into Terraform
terraform import aws_security_group_rule.vendor_access sg-12345678_ingress_tcp_443_443_1.2.3.4/32

# Then update the Terraform configuration to match
resource "aws_security_group_rule" "vendor_access" {
  type              = "ingress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  cidr_blocks       = ["1.2.3.4/32"]
  security_group_id = aws_security_group.database.id
  description       = "Vendor X integration - ticket INC-4821 - review quarterly"
}

Unauthorized drift (change was not intentional, should be reverted):

# Revert by applying the Terraform configuration
terraform apply drift-detection.tfplan

The description field on security group rules deserves emphasis. AWS security group rules support a description field that most teams leave empty. A populated description field (with the ticket number, the reason, and a review cadence), is the difference between a security group rule that can be evaluated for necessity and one that has to be treated as unknown.

Preventing Drift at the Source: SCP and IAM Guardrails

Detection is the safety net. Prevention is the preference. The most effective prevention mechanism for manual drift in a multi-account AWS organization is an SCP that requires changes to specific resource types to be made through an approved automation pathway rather than directly.

This is not feasible for all resource types: an SCP that prevents all manual console changes would break on-call operations and routine management tasks. But it's feasible for high-risk resource categories:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RequireTagForManualSecurityGroupChanges",
      "Effect": "Deny",
      "Action": [
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:AuthorizeSecurityGroupEgress",
        "ec2:RevokeSecurityGroupIngress",
        "ec2:RevokeSecurityGroupEgress"
      ],
      "Resource": "*",
      "Condition": {
        "Null": {
          "aws:RequestedRegion": "false"
        },
        "StringNotLike": {
          "aws:PrincipalArn": [
            "arn:aws:iam::*:role/TerraformDeployRole",
            "arn:aws:iam::*:role/BreakGlassRole"
          ]
        }
      }
    }
  ]
}

This SCP denies security group rule modifications from any principal that isn't the Terraform deployment role or a named break-glass role. Engineers who need to make an emergency manual change must assume the break-glass role, which generates a CloudTrail event with the role assumption clearly logged, instead of modifying the security group directly from their own IAM user or SSO session.

The break-glass role pattern is the critical complement to any restrictive SCP. Preventing all manual changes with no emergency escape path creates operational risk: the next time an engineer needs to open a port to diagnose a production issue, they won't be able to. A break-glass role with permission to bypass the SCP, with assumption logged in CloudTrail and an alert fired on assumption, gives you both the guardrail and the escape valve.

Tradeoffs & Decision Framework

Not all drift prevention is worth the operational overhead. The right level of enforcement depends on the blast radius of the resource category and the operational maturity of the team.

High-enforcement, low-tolerance for drift:

• Security group rules in production environments
• IAM policies and roles
• S3 bucket policies and public access settings
• CloudTrail configuration
• KMS key policies

These are the resource categories where unauthorized drift directly translates to security exposure. The cost of a restrictive SCP or a break-glass process for emergency changes is justified by the risk of undetected drift.

Medium-enforcement, tolerate short-lived drift:

• ECS task definition revisions, Auto Scaling desired counts, and similar operational parameters that change frequently and are designed to be adjusted at runtime
• CloudWatch alarm thresholds that get tuned during incident response
• Route 53 records for emergency traffic steering

For these, manual changes are operationally normal. The prevention model is detection and reconciliation: run a scheduled Terraform plan to detect drift, and require engineers to codify any manual change that persists beyond the immediate incident window.

Low-enforcement, drift is expected:

• Sandbox and development accounts where experimentation is the purpose
• Auto Scaling state (desired count, current count) which AWS updates continuously
• Resources managed by AWS on your behalf (ECS task replacement, RDS failover)

For these, drift detection in your IaC pipeline creates noise without signal. Configure Terraform's lifecycle { ignore_changes = [...] } for attributes that AWS manages autonomously, and use separate alerting for sandbox accounts that focuses on cost and egregious security violations rather than configuration parity.

Lessons From the Field

1. The worst configuration drift I've ever encountered wasn't malicious: it was inherited. Took over an AWS environment at a SaaS company where the previous infrastructure team had been applying Terraform to about 40% of the environment and managing the rest manually through the console. The Terraform state had not been refreshed against the actual infrastructure in over a year. When I ran terraform plan, it showed 847 resource changes. It took six weeks to triage which changes were drift that should be reconciled, which were drift that should be codified, and which were Terraform's attempt to remove things that manual operations had replaced with something different.

2. Break-glass roles with CloudTrail alerts change behavior without blocking operations. At a fintech client, we implemented an SCP preventing direct security group modifications from engineer IAM sessions and required engineers to assume a break-glass role for any manual security group change. CloudTrail alerted the security team on every break-glass role assumption. In the first month, twelve manual changes were made through the break-glass role. In the second month, four. In the third, one. Engineers started using Terraform first because the break-glass path, while not blocked, was visible and had overhead. Visibility changed behavior without creating operational friction.

3. terraform plan on a schedule is the cheapest drift detection you can run. Set up a scheduled GitHub Actions workflow for an e-commerce client that ran terraform plan nightly and opened a GitHub issue with the plan output if changes were detected. No third-party tooling. No Terraform Cloud subscription. Just a cron job and a bash script. Over six months, it caught three meaningful drift events: an engineer who had added an S3 bucket public access exception for a demo, an Auto Scaling group whose max capacity had been manually increased and never reset, and a Lambda function whose reserved concurrency had been manually set to zero to stop it during an incident and never re-enabled.

4. Drift in IAM is the most dangerous kind because it doesn't produce monitoring signals. An over-permissioned IAM role produces no CloudWatch alarms. It doesn't increase your error rate or your latency. It sits quietly until a credential attached to that role is compromised, at which point the blast radius is determined by whatever permissions ended up in the role through two years of operational iteration. Schedule quarterly IAM Access Analyzer reviews and diff the results against your Terraform-declared IAM configuration. The gap between the two is your risk exposure.

5. The engineer who made the manual change is rarely the engineer who gets paged when it causes a problem. The security group rule opened "just for testing" doesn't cause a problem the day it's added. It causes a problem eight months later, for an on-call engineer who has never heard of the test it was added for, at 3am. The drift cost is paid by someone who didn't incur it. Building drift detection and prevention into your platform isn't just good hygiene, it's an act of respect for whoever inherits your on-call rotation.

Final Thoughts

The industry's answer to configuration drift has been converging on two things: policy as code for prevention and continuous state comparison for detection. AWS Config, CloudFormation drift detection, Terraform's plan-as-diff model, and tools like driftctl and Terraform Cloud's continuous plan features are all moving in the same direction, making the gap between declared state and actual state visible in real time rather than discoverable only during an incident.

What the tooling doesn't solve is the cultural piece: teams that treat IaC as the authoritative record of infrastructure and manual changes as exceptions requiring reconciliation, versus teams that treat IaC as the starting point and manual changes as normal operations. The tools can surface drift. They can't make engineers care about closing it.

The teams that manage drift well have one thing in common: they treat a non-empty terraform plan as a question that requires an answer, not background noise. Either the plan shows a change they intended to make, a change they need to codify from an emergency, or unauthorized drift they need to revert. Each category has a response. None of them is "ignore it and run apply anyway."

Infrastructure configuration drift isn't an exotic problem. It's the default outcome of operating cloud infrastructure without deliberate prevention. The discipline that prevents it (IaC as the authoritative record, scheduled drift detection, break-glass patterns for emergencies, quarterly reconciliation), is the discipline we embed in every infrastructure engagement at PulseSoft. If your infrastructure has accumulated more drift than you can see, let's talk.

Key Takeaways

• Configuration drift is the default outcome of operating AWS without deliberate prevention. Every console click and aws CLI command that isn't codified into IaC is drift that accumulates. It doesn't announce itself until something goes wrong.
• CloudFormation drift detection identifies what drifted but not who or when. Correlate drift findings with CloudTrail events filtered by the resource ARN and the time window before detection to get the full forensic picture. Both tools are necessary; neither is sufficient alone.
• Run terraform plan on a schedule as your cheapest drift detection mechanism. A nightly plan with --detailed-exitcode and an alert on exit code 2 costs a scheduled GitHub Actions run and catches drift before it compounds: no third-party tooling required.
• Drift remediation requires triage before apply. When Terraform detects drift, terraform apply will revert the manual change. If the change was intentional, import the resource state and codify the change first. If it was unauthorized, revert via apply. Running apply without triage risks removing deliberate operational decisions.
• Security group rules should have populated description fields. An empty description turns a security group rule into an archaeological problem during audit and incident response. A description with a ticket number, reason, and review date makes the rule evaluable without investigating its history.
• Break-glass roles with CloudTrail alerts change engineer behavior without blocking emergency operations. Visible, logged escape valves reduce the incidence of unauthorized manual changes over time because the overhead of the break-glass path prompts engineers to use the IaC path first.
• IAM drift is the most dangerous kind because it produces no monitoring signals. An over-permissioned role doesn't increase error rates or latency. It sits quietly until a compromised credential activates it. Schedule quarterly IAM Access Analyzer reviews and diff results against Terraform-declared IAM configuration.