Immutable Infrastructure: Replace, Never Repair Servers

Immutable infrastructure is a deployment model where servers are never modified after they are deployed. When a change is needed — a configuration update, a security patch, a new application version — you build a new image from scratch and replace the running instances. The old instances are destroyed. This approach eliminates configuration drift, simplifies rollback, and ensures every deployment is deterministic and reproducible. Netflix pioneered this model in production, replacing thousands of instances daily with an average instance lifetime measured in hours.

Mutable vs Immutable

In mutable infrastructure, servers accumulate changes over months of operation. Two servers built from the same base image two months apart behave differently — one received package updates the other missed, one had a manual config change for a hotfix, another had a cron job that modified permissions. Immutable infrastructure eliminates drift entirely because every replacement starts from the same golden image and lasts only until the next deployment. The 2024 CrowdStrike outage demonstrated the risks of mutable infrastructure at global scale, where a single configuration push affected millions of endpoints because there was no image-based validation prior to deployment.

Golden Images

A golden image is a pre-baked machine image containing the operating system, runtime, application code, and all dependencies. It is built once by a CI pipeline and stored in a registry, ready for deployment. Golden images are the foundation of immutable infrastructure — they guarantee that every instance is identical to every other instance from the same image.

Building with Packer

Packer by HashiCorp is the standard tool for creating golden images across platforms:

source "amazon-ebs" "webapp" {
  ami_name      = "webapp-{{timestamp}}"
  instance_type = "t3.medium"
  region        = "us-east-1"
  source_ami_filter {
    filters = {
      name                = "ubuntu/images/hvm-ssd/ubuntu-24.04-amd64-server-*"
      root-device-type    = "ebs"
    }
    most_recent = true
  }
  ssh_username = "ubuntu"
---

build {
  sources = ["source.amazon-ebs.webapp"]

  provisioner "shell" {
    inline = [
      "sudo apt-get update",
      "sudo apt-get install -y nginx nodejs",
      "sudo systemctl enable nginx",
    ]
  }

  provisioner "file" {
    source      = "app/"
    destination = "/opt/app"
  }
---

packer build packer.pkr.hcl

Packer supports building images for AWS (AMI), Azure (VHD), Google Cloud (machine image), VMware (vSphere template), and Docker (container image) from the same template definition. This cross-platform capability makes it the standard choice for organizations running hybrid or multi-cloud infrastructure.

Image Versioning

Every image gets a unique, immutable version tag:

webapp-v1.2.3-20250615-abc123def

The format includes semantic version, build date, and Git commit SHA. This ensures traceability from running instance back to source code. Never reuse or overwrite an existing tag — if a deployment fails, the previous tag is still available for immediate rollback. Google Cloud’s recommendations for immutable deployments specify that image tags must be unique and never mutated.

Security Scanning for Images

Integrate image scanning into the build pipeline to catch vulnerabilities before deployment:

# Scan AMI with Trivy
trivy image --severity CRITICAL,HIGH public.ecr.aws/myapp:v1.2.3

# Policy check: fail build if critical CVEs found
trivy image --exit-code 1 --severity CRITICAL public.ecr.aws/myapp:v1.2.3

Scanning at build time rather than runtime is a core security practice. If a critical vulnerability is found, the image is never deployed, and the pipeline notifies the team to address the vulnerability in the next build.

Replacement Over Repair

The operational workflow for any change follows a strict pattern:

1. Update the image definition (Packer template, Dockerfile, or configuration script)
2. Build and tag a new image via CI/CD pipeline
3. Deploy the new image across the fleet using rolling update, blue-green, or canary strategy
4. Terminate old instances after the new ones are healthy

# AWS Auto Scaling instance refresh
aws autoscaling start-instance-refresh \
  --auto-scaling-group-name webapp-asg \
  --preferences '{"InstanceWarmup": 60, "MinHealthyPercentage": 90}'

Configuration and Secrets

Immutable infrastructure still needs configuration and secrets at runtime. The image should be environment-agnostic:

The key principle: the image is identical across environments. Staging and production use different ConfigMaps and secrets, but the same image. This guarantees that what you test in staging is exactly what runs in production, eliminating the most common source of deployment failures.

Operational Benefits

• Deterministic deployments — every instance from the same image behaves identically, eliminating “works on my machine” for infrastructure
• No snowflake servers — no “that server is special, we cannot replace it”
• Instant rollback — deploy the previous image tag
• Simplified debugging — SSH into a staging instance built from the same image; production instances are disposable
• Security — SSH access to production is rarely needed; access is through a bastion with audit logging
• Versioned artifacts — every image is a recoverable, auditable deployment artifact
• Immutable audit trail — you can reconstruct the exact state of any deployment in history

Immutable Infrastructure in CI/CD

Integrating immutable infrastructure with CI/CD pipelines requires a build-and-deploy workflow that treats image creation as the primary artifact. This is conceptually different from mutable workflows where configuration management tools apply incremental changes to running servers.

CI/CD Pipeline for Immutable Deployments

A typical immutable deployment pipeline has three stages:

1. Build — Packer or Docker builds a new golden image
2. Test — the image is deployed to a staging environment with automated validation
3. Deploy — the validated image replaces production instances

# GitHub Actions workflow for immutable deployments
jobs:
  build-image:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build AMI with Packer
        run: packer build .
      - name: Scan image for vulnerabilities
        run: trivy image --severity CRITICAL --exit-code 1 my-image:latest
      - name: Test image
        run: |
          echo "Running image validation tests..."

  deploy:
    needs: build-image
    runs-on: ubuntu-latest
    steps:
      - name: Update Auto Scaling group
        run: |
          aws autoscaling start-instance-refresh \
            --auto-scaling-group-name webapp-asg \
            --preferences '{"InstanceWarmup": 120}'

Rollback Strategy

Immutable infrastructure provides simple rollback — deploy the previous image tag:

# Rollback by redeploying the previous image version
terraform apply -var="image_version=v1.2.2"

Automate rollback triggers based on deployment monitoring. If error rates increase within 15 minutes of deployment, automatically roll back to the previous image. This “deploy with guardrails” approach minimizes mean time to recovery for bad deployments.

Blue-Green with Immutable Images

Blue-green deployment is the natural deployment strategy for immutable infrastructure:

1. Build a golden image (v1.2.3)
2. Deploy to the green environment
3. Run smoke tests against green
4. Switch traffic to green
5. Keep blue running as rollback target
6. Terminate blue after validation period

This pattern provides the fastest rollback (traffic switch, seconds) and ensures the new image is fully validated before serving production traffic.

FAQ

Is immutable infrastructure practical for stateful services?

Yes, but it requires more planning. Use persistent volumes attached to new instances during deployment. Cloud providers support detaching and reattaching volumes. For databases, use managed services (RDS, Cloud SQL) that are inherently mutable but managed by the provider, while the application tier is immutable. Services like Amazon EFS and Azure Files provide persistent shared storage that follows instances.

How do I handle emergency patches with immutable infrastructure?

The immutable approach does not prevent emergency fixes — it just routes them through the same build-and-replace pipeline. For a critical security patch, build a new image with the patch, test it (fast-track verification), and deploy. The process is scripted and automated, so it is actually faster than SSHing into 100 servers to run apt update. Most organizations that adopt immutable infrastructure find emergency patches reach production faster than their mutable workflow.

What is the cost impact of constantly replacing servers?

Instance refresh is part of normal operations in auto-scaling groups. Cloud providers do not charge extra for instance replacement. The cost of building images (CI runner time, temporary instances during Packer build) is minimal compared to operational savings from eliminated drift and reduced debugging time. In most cases, the total cost of ownership decreases because operations teams spend less time on server maintenance.

Can I use immutable infrastructure on-premises?

Yes, it works for on-premises deployments as well. Use VM templates (vSphere, Hyper-V) with Packer to build golden images, and deploy with rolling replacement using the on-premises equivalent of instance refresh. Tools like Ansible Tower or vRealize Automation can orchestrate the replacement workflow.

How do databases fit into immutable infrastructure?

Run databases as managed services or in stateful sets with persistent volumes attached during instance replacement. The immutable principle applies to the application tier. The database tier uses its own deployment pattern — typically blue-green with backward-compatible schema changes and replication-based migration.

Conclusion

Immutable infrastructure transforms server management from repair to replacement. Golden images built with Packer eliminate configuration drift. Blue-green and canary deployments provide instant rollback and zero-downtime updates. Environment-agnostic images ensure testing matches production exactly. The operational benefits — deterministic deployments, no snowflake servers, simplified debugging, and comprehensive audit trails — make immutable infrastructure a cornerstone of modern DevOps practice.

For deployment strategies, see blue-green deployments. For image building fundamentals, see Docker multi-stage builds.