Your CI/CD Pipeline Should Catch Infrastructure Mistakes Before a Human Reviewer Ever Sees Them

Introduction

A senior engineer approved a Terraform PR that opened port 22 to 0.0.0.0/0 on a production security group. Not because they missed it, because it was buried in line 847 of a 900-line plan output, surrounded by expected changes from a VPC module refactor, and the review happened on a Friday afternoon when the deploy needed to go out before the weekend. Nobody caught it until a security scanner flagged it six weeks later.

Infrastructure CI/CD pipeline validation exists to prevent exactly this. Not to replace human review, human review catches intent problems, architectural concerns, and context that automated tools can't evaluate. But to front-load the mechanical, rule-based checks that humans reliably miss under time pressure: security group rules opened to the world, encryption disabled on storage resources, resource names that violate your naming convention, IAM policies that grant * actions. Those checks run in seconds, block the PR automatically, and never have a bad Friday afternoon.

This post covers the specific validation layers that belong in an infrastructure pipeline, how to sequence them, and where teams consistently underinvest.

Why Human Review Alone Fails Infrastructure Code

Infrastructure code review has a different failure profile than application code review. In application code, a reviewer is evaluating logic, does this function do what the author intends, does it handle edge cases, does it introduce a bug? The output space is large but the reviewer has good cognitive tools for evaluating it. Code review for logic bugs works reasonably well.

In infrastructure code review, a reviewer is evaluating configuration: is this resource configured correctly, does this security group rule create an unintended network path, does this IAM policy violate least privilege? The problem is that configuration correctness often requires knowledge the reviewer doesn't have or can't hold in context: the current state of the security group before this change, the full set of IAM policies already attached to this role, the naming convention for every other resource in this module.

The second problem is plan output volume. A Terraform plan for a meaningful infrastructure change routinely produces hundreds of lines of output. A meaningful security issue, an ingress rule added to a security group, is six or seven lines in that output. A reviewer's ability to locate and evaluate every security-relevant configuration change in a 900-line plan, consistently, under time pressure, is not a reasonable operational assumption.

The third problem is the consistency gap. A human reviewer who has been thorough and careful for the first forty PRs they reviewed will be less thorough on the forty-first, especially if it arrives at 4pm. An automated validator that checks for 0.0.0.0/0 in every security group rule is equally thorough on PR one and PR four thousand.

The failure mode this produces: teams with strong engineers and good review culture still pass security misconfigurations into production, because the checks that should be mechanical are being executed by humans. The fix is not better humans. It's moving the mechanical checks out of human review entirely.

A concrete example: a healthcare client's Terraform PR review process relied on a senior engineer manually reading every plan output. Over eighteen months, three separate PRs introduced S3 buckets without server-side encryption enabled. Each was caught eventually, two in the next audit cycle, one during a compliance review six months later. The fix was not a longer checklist for reviewers. It was a Checkov rule that blocked any PR that created an S3 bucket without server_side_encryption_configuration defined. Zero unencrypted S3 buckets in production since.

AWS Deep Dive: The Validation Stack That Catches What Humans Miss

Layer 1: Static Analysis Before the Plan Runs

Static analysis runs against the Terraform HCL before any AWS API calls are made. It's the fastest and cheapest validation layer, and it should run first, catching syntactic, structural, and policy violations before you spend the time and credentials on a terraform plan.

terraform validate is the baseline: it checks that the configuration is syntactically valid and internally consistent. It's not a security tool, it won't catch a security group opened to the world, but it catches configuration errors that would fail at plan time, and it runs without AWS credentials. It belongs in the first step of every pipeline.

tflint catches Terraform-specific issues that terraform validate misses: deprecated resource attributes, invalid AWS resource configurations, naming convention violations if you've configured custom rules, and common misconfigurations for specific AWS providers. The AWS provider ruleset for tflint catches things like invalid instance types, deprecated parameter values, and missing required tags:

# .tflint.hcl
config {
module = false
}

plugin "aws" {
enabled = true
version = "0.27.0"
source  = "github.com/terraform-linters/tflint-ruleset-aws"
}

rule "aws_instance_invalid_type" {
enabled = true
}

rule "aws_resource_missing_tags" {
enabled = true
tags    = ["Environment", "Team", "Service", "ManagedBy"]
}

Checkov is the security policy scanner. It evaluates Terraform HCL against a library of security and compliance checks, hundreds of rules covering encryption, public access, logging, IAM policy structure, and service-specific misconfigurations. The rules that matter most for AWS infrastructure:

# Run Checkov against Terraform with specific checks enabled
checkov -d . \
  --framework terraform \
  --check CKV_AWS_23 \   # Security group does not allow ingress from 0.0.0.0/0 to port 22
  --check CKV_AWS_24 \   # Security group does not allow ingress from 0.0.0.0/0 to port 3389
  --check CKV_AWS_19 \   # S3 bucket has encryption enabled
  --check CKV_AWS_21 \   # S3 bucket versioning enabled
  --check CKV_AWS_57 \   # S3 bucket has public access blocked
  --check CKV_AWS_111 \  # IAM policy does not allow write access without constraint
  --check CKV_AWS_40     # IAM user does not have inline policy

The non-obvious Checkov behavior that matters at scale: Checkov's default behavior is to fail on the first check that fails. For a PR pipeline, you usually want to collect all failures and report them at once, so the author can address everything in a single revision cycle rather than playing whack-a-mole with sequential failures:

checkov -d . --framework terraform --soft-fail-on LOW --output cli --output junitxml --output-file-path results/

--soft-fail-on LOW treats low-severity findings as warnings rather than failures, preventing check fatigue while still blocking on medium and high severity. --output junitxml produces output that integrates directly with GitHub Actions and most CI systems for PR annotations.

Layer 2: Plan Generation and Policy Evaluation

After static analysis passes, generate the plan. The plan is the authoritative statement of what Terraform intends to do to actual AWS resources, and it's the input to the most important validation layer: policy evaluation against the plan output.

OPA (Open Policy Agent) with Conftest allows you to write policies that evaluate the Terraform plan JSON rather than the HCL source. This catches things static analysis can't, particularly, changes that are valid HCL but produce dangerous infrastructure configurations when combined with existing state:

# policy/security_groups.rego
package main

import future.keywords.in

deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_security_group_rule"
  resource.change.after.type == "ingress"
  "0.0.0.0/0" in resource.change.after.cidr_blocks
  msg := sprintf(
    "Security group rule '%s' allows ingress from 0.0.0.0/0. Unrestricted ingress is not permitted in production.",
    [resource.address]
  )
}

deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_s3_bucket"
  not resource.change.after.server_side_encryption_configuration
  msg := sprintf(
    "S3 bucket '%s' does not have server-side encryption configured.",
    [resource.address]
  )
}

# Generate plan JSON and evaluate with Conftest
terraform plan -out=plan.tfplan
terraform show -json plan.tfplan > plan.json
conftest test plan.json --policy policy/

The advantage of OPA over Checkov for plan evaluation: OPA policies can evaluate contextual conditions that require knowledge of the change type, not just the final configuration. You can write a policy that allows an SSH ingress rule from a specific CIDR for a bastion host resource type, but blocks the same rule on any other resource type: a distinction that source-only static analysis can't make.

Layer 3: The Pipeline Structure That Enforces Sequence

The validation layers need to run in the right order with the right failure behavior. A pipeline that runs Checkov after generating a plan wastes the time spent on plan generation if the HCL has obvious violations. A pipeline that doesn't post the plan output as a PR comment makes it hard for reviewers to evaluate what will actually change:

# .github/workflows/terraform-validate.yml
name: Terraform Validation

on:
  pull_request:
    paths:
      - '**.tf'
      - '**.tfvars'

jobs:
  validate:
    name: Static Analysis
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: '~1.7'

      - name: Terraform Format Check
        run: terraform fmt -check -recursive

      - name: Terraform Init
        run: terraform init -backend=false

      - name: Terraform Validate
        run: terraform validate

      - name: tflint
        uses: terraform-linters/setup-tflint@v4
        with:
          tflint_version: latest
      - run: tflint --config=.tflint.hcl

      - name: Checkov
        uses: bridgecrewio/checkov-action@v12
        with:
          directory: .
          framework: terraform
          soft_fail_on: LOW
          output_format: cli,sarif
          output_file_path: results/checkov.sarif

  plan:
    name: Terraform Plan
    needs: validate
    runs-on: ubuntu-latest
    permissions:
      id-token: write # Required for OIDC
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/GitHubActionsTerraformRole
          aws-region: us-east-1

      - name: Terraform Plan
        id: plan
        run: |
          terraform plan -out=plan.tfplan -detailed-exitcode
          terraform show -json plan.tfplan > plan.json

      - name: OPA Policy Check
        run: conftest test plan.json --policy policy/

      - name: Post Plan to PR
        uses: actions/github-script@v7
        with:
          script: |
            const plan = require('fs').readFileSync('plan.json', 'utf8');
            // Post truncated plan summary as PR comment

The non-obvious GitHub Actions behavior that affects infrastructure pipelines: OIDC token issuance for the configure-aws-credentials action requires the id-token: write permission at the job level. If the permission is set at the workflow level but not the job level, and you have a permissions block at the job level for other reasons (like pull-requests: write for posting comments), the job-level block overrides the workflow-level block completely: the id-token: write permission must be explicitly included in the job-level permissions block or the OIDC exchange will fail with a confusing auth error that doesn't clearly indicate the missing permission.

Tradeoffs & Decision Framework

The right level of automated validation investment depends on the size of the team, the rate of infrastructure changes, and the risk profile of what's being managed.

Small teams (1–3 infrastructure engineers), low change frequency: terraform fmt, terraform validate, and Checkov on pull requests covers the highest-value checks with minimal pipeline complexity. Skip OPA initially: the policy language has a learning curve and the benefit over Checkov's rule library is marginal until you have custom policy requirements that Checkov's managed rules don't express.

Growth teams (4–10 engineers), regular infrastructure changes: Add tflint with custom rules that enforce your naming conventions and tagging requirements. Add plan posting to PRs so reviewers are evaluating the plan output, not the HCL in isolation. Consider OPA when you have policy requirements that are contextual, rules that depend on the change type or the relationship between resources, not just the configuration values.

Mature teams or regulated environments: OPA with a managed policy library, Checkov mapped to your compliance framework (CIS Benchmark, SOC 2 controls, PCI DSS requirements), and plan-level approval gates that require explicit sign-off on any changes touching production security groups, IAM policies, or encryption configuration. Sentinel (for Terraform Cloud/Enterprise) is worth evaluating as an alternative to OPA for teams already on that platform.

The tradeoff nobody likes to hear: more validation gates mean slower PR cycles. A pipeline that runs four validation tools sequentially before generating a plan can add eight to twelve minutes to a PR that previously merged in two. Engineers under delivery pressure will route around it, submitting smaller PRs to avoid the validation overhead, or arguing for gates to be disabled "just this once." The mitigation is fast failure: run static analysis first, fail fast on the cheapest checks, and only spend time on plan generation and OPA evaluation if the HCL passes static analysis. A well-sequenced pipeline that fails in thirty seconds on a formatting violation is less friction than one that runs for ten minutes before failing on the same issue.

Lessons From the Field

1. The first time a CI check blocks a production security issue, the investment pays for itself. Built out a Terraform CI pipeline for a fintech client that included Checkov and OPA plan evaluation. Six weeks after launch, the pipeline blocked a PR that created an RDS parameter group with log_min_duration_statement set in a way that would have logged full query text including query parameters: a PII exposure in their audit logs. The engineer who submitted the PR hadn't realized the implication. The OPA policy caught it before review. The CISO used that example in the next board security update.

2. Plan output posted to PR comments changes how engineers write IaC. Before implementing plan-as-PR-comment for a SaaS client, code reviews of Terraform PRs were reviewing HCL in isolation. Authors described what they intended to change in the PR description. Reviewers trusted the description. After implementing plan posting, reviewers started catching discrepancies between what the author said the change would do and what the plan actually showed. In the first month, three PRs were revised after reviewers identified that the plan showed more resource changes than the author had intended, two module refactors that inadvertently triggered replacements, and one variable change that cascaded to twelve resources the author didn't realize the variable controlled.

3. terraform fmt -check in CI eliminates an entire category of formatting review comments. The single most consistent noise in Terraform PR reviews, in my experience, is formatting. Two spaces vs. four. Inconsistent alignment of = signs in resource blocks. Trailing whitespace. These are mechanical checks that don't require human judgment and consume review cycles that should go to substantive concerns. terraform fmt -check fails the pipeline on any unformatted file. After one sprint of formatting failures, engineers start running terraform fmt before pushing. The formatting comments in code review drop to zero.

4. Checkov false positives erode trust faster than false negatives. Deployed Checkov for a logistics client with all checks enabled. The check failure volume on the first run was 847 findings. Most were legitimate misconfigurations. About 200 were false positives, Checkov checks that flagged resources for missing configuration that was intentionally absent (a Lambda that didn't need a VPC configuration, an S3 bucket intentionally public for static website hosting). Engineers started ignoring Checkov output entirely within two weeks. Trim the check list to your actual policy requirements. Add # checkov:skip=CKV_AWS_115:Lambda does not need VPC for this use case skip annotations with documented reasons for intentional deviations. A pipeline that produces meaningful signal is more valuable than one that produces comprehensive noise.

5. The OIDC permission block interaction in GitHub Actions will cost you two hours the first time. Set up a Terraform CI pipeline for an enterprise client using OIDC authentication to AWS. The workflow-level permissions block had id-token: write. The plan job had a permissions block for pull-requests: write to post comments. The job-level block silently overrode the workflow-level block and dropped id-token: write. The OIDC exchange failed with Error: Credentials could not be loaded: not Error: Missing id-token permission. It took two hours of debugging to identify the cause. Always include id-token: write explicitly in job-level permissions blocks when using OIDC.

Final Thoughts

The infrastructure CI/CD pipeline validation stack is converging on a relatively stable set of tools (terraform fmt, terraform validate, tflint, Checkov, and OPA or Sentinel for plan-level policy evaluation), and the patterns for sequencing them are well-understood. What varies is the investment teams make in actually implementing them, and the discipline to maintain the policies as the infrastructure evolves.

The gap I see most consistently: teams that implement static analysis but stop short of plan-level policy evaluation. Checkov against HCL is valuable. OPA against a plan JSON is more valuable for a specific class of policy: the contextual policies that require knowing what's actually changing, not just what the configuration declares. Both layers together cover the full spectrum of automated infrastructure validation.

The direction the tooling is moving: policy libraries that map directly to compliance frameworks (SOC 2 controls, CIS Benchmark, HIPAA safeguards) are getting better and more current, which reduces the work of building and maintaining custom policy sets. AWS Security Hub's integration with Terraform via third-party providers is improving visibility into the compliance posture of IaC-defined resources before they're deployed. The gap between "what my Terraform declares" and "what my compliance posture actually is" is getting easier to close with automation.

The discipline that makes all of it work is treating a CI failure as a blocker, not a suggestion. A pipeline that can be overridden with a comment or a merge button click provides the appearance of validation without the substance. The investment in infrastructure CI/CD pipeline validation pays back in every incident that didn't happen because a check caught the configuration before it reached production. That's a return that doesn't show up in any metric, and is entirely real.

Building that discipline into your infrastructure delivery process is exactly the kind of platform work we do at PulseSoft. If your Terraform review process is still primarily a human exercise, let's talk.

Key Takeaways

• Automated infrastructure validation moves mechanical checks out of human review. Security group rules opened to the world, unencrypted storage, missing tags, and naming convention violations are rule-based checks that humans reliably miss under time pressure. CI catches them consistently on every PR.
• Run static analysis before plan generation. terraform fmt, terraform validate, tflint, and Checkov against HCL should fail fast before spending time and credentials on a terraform plan. A well-sequenced pipeline fails in under a minute on formatting and policy violations, not after a ten-minute plan run.
• Checkov's --soft-fail-on LOW prevents alert fatigue without disabling meaningful checks. Low-severity findings become warnings; medium and high remain blockers. Use # checkov:skip annotations with documented reasons for intentional deviations, undocumented skips are technical debt.
• OPA policy evaluation against plan JSON catches contextual violations that source analysis misses. A policy that allows SSH ingress on a bastion host resource but blocks it on all others requires knowledge of the change type and resource address, information available in the plan JSON, not in the HCL source.
• The GitHub Actions OIDC id-token: write permission must appear in the job-level permissions block explicitly. A job-level permissions block silently overrides the workflow-level block. Missing id-token: write at the job level fails OIDC authentication with a credential error that doesn't identify the missing permission.
• Posting the plan output to the PR comment changes how reviewers evaluate infrastructure changes. Reviewers evaluating HCL in isolation trust the author's PR description of intended changes. Reviewers evaluating the plan output catch discrepancies between what the author intended and what Terraform will actually do.
• A CI failure that can be overridden provides the appearance of validation without the substance. The return on investment in infrastructure pipeline validation comes entirely from the failures that block bad configuration before production. Gates that can be bypassed with a button click deliver zero of that return.