CI/CD Pipelines: Automate Build, Test, and Deploy

Cloud Computing Cloud Computing 8 min read 1633 words Beginner ExcellentWiki Editorial Team

Continuous Integration and Continuous Delivery (CI/CD) automates the software delivery process from code commit to production deployment. Continuous Integration (CI) automatically builds and tests every code change. Continuous Delivery (CD) automatically deploys approved changes to production. According to the 2024 DORA Report (Google Cloud), elite-performing teams deploy 973x more frequently, have 6,570x faster lead times, and recover from incidents 6,570x faster than low-performing teams — and CI/CD is the primary enabler of these outcomes.

The Case for CI/CD

Manual deployment processes are slow, error-prone, and non-auditable. A single omitted step or forgotten configuration causes production incidents. CI/CD replaces manual runbooks with automated, version-controlled pipelines:

Faster releases — elite teams deploy on-demand, multiple times per day. CI/CD eliminates the release day bottleneck.
Consistent quality — every commit goes through the same automated test suite. No skipped testing because a deadline is approaching.
Auditability — every deployment has a commit hash, pipeline run ID, and artifact version. Rollback is a single click.
Reduced risk — small, frequent deployments are easier to validate and roll back than large, infrequent releases.

GitHub Actions

GitHub Actions is the most widely adopted CI/CD platform due to its native GitHub integration, extensive marketplace (over 20,000 actions), and generous free tier.

name: ExcellentWiki CI/CD
on:
  push:
    branches: [main]
    tags: ['v*']
  pull_request:
    branches: [main]

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

env:
  IMAGE_NAME: excellentwiki-api
  REGISTRY: ghcr.io

jobs:
  test:
    runs-on: ubuntu-24
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
          cache: 'pip'
      - run: pip install -r requirements-dev.txt
      - run: pytest --cov --junitxml=test-results.xml
      - uses: dorny/test-reporter@v1
        if: always()
        with:
          name: Test Results
          path: test-results.xml
          reporter: java-junit

  build-and-push:
    needs: test
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-24
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t $IMAGE_NAME:${{ github.sha }} .
      - run: docker tag $IMAGE_NAME:${{ github.sha }} $REGISTRY/${{ github.repository }}/$IMAGE_NAME:${{ github.sha }}
      - run: docker push $REGISTRY/${{ github.repository }}/$IMAGE_NAME:${{ github.sha }}

  deploy:
    needs: build-and-push
    runs-on: ubuntu-24
    environment: production
    steps:
      - uses: actions/checkout@v4
      - run: kubectl set image deployment/api app=$REGISTRY/${{ github.repository }}/$IMAGE_NAME:${{ github.sha }}

Key GitHub Actions features:

Matrix builds — test across multiple OS, language versions, and dependency configurations in parallel
Reusable workflows — define common pipelines once and reference from multiple repositories
Environments — deployment environments with protection rules (required reviewers, wait timer)
Composite actions — package shared step logic (deploy to Kubernetes, publish to registry)

GitLab CI

GitLab CI uses a .gitlab-ci.yml file with a stages-based pipeline model:

stages:
  - lint
  - test
  - build
  - deploy

image: python:3.12-slim

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

cache:
  key: $CI_COMMIT_REF_SLUG
  paths:
    - .cache/pip
    - .venv

lint:
  stage: lint
  script:
    - pip install ruff
    - ruff check src/ tests/

test:
  stage: test
  coverage: '/TOTAL.*\s+(\d+%)$/'
  artifacts:
    reports:
      junit: test-results.xml
  script:
    - pip install -r requirements-dev.txt
    - pytest --junitxml=test-results.xml --cov

build:
  stage: build
  services:
    - docker:dind
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

deploy:
  stage: deploy
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
  script:
    - kubectl set image deployment/app app=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  environment:
    name: production

GitLab CI’s auto DevOps feature provides opinionated default pipelines for common stacks (Django, Rails, Spring Boot) with built-in SAST, dependency scanning, and container scanning.

Pipeline Stages (Full Lifecycle)

A mature CI/CD pipeline runs eight stages sequentially or in parallel where dependencies allow:

Lint — static analysis for code style and potential bugs (Ruff, ESLint, Prettier)
Unit tests — fast tests for individual functions and classes, run in parallel
Build — compile code, build container images, produce deployment artifacts
Integration tests — test component interactions, database access, API contracts
Security scanning — SAST (static analysis), SCA (dependency vulnerabilities), container scanning
Deploy to staging — deploy to production-like environment for validation
End-to-end tests — full workflow tests in staging with real services
Deploy to production — gradual rollout with canary or blue-green deployment

Deployment Strategies

Strategy	Description	Risk	Rollback	Complexity
Rolling	Replace instances gradually	Low	Moderate	Low
Blue/Green	Switch traffic between old and new environment	Very low	Instant (switch back)	Medium
Canary	Route small % of traffic to new version	Very low	Instant	High
Feature flags	Toggle features on/off without deployment	Lowest	Instant (toggle off)	Medium

Blue/green deployment with Kubernetes:

# Blue (current)
apiVersion: apps/v1
kind: Service
metadata:
  name: app-service
spec:
  selector:
    app: app
    version: blue
  ports:
  - port: 80
    targetPort: 8080

# Green (new) — deploy, verify, then switch label selector

After switching the Service selector to version: green, traffic flows to the new deployment. If issues arise, switch back to version: blue. No code redeployment needed.

Testing in Pipelines

The test pyramid approach optimizes for feedback speed:

# Run different test types with increasing scope
pytest tests/unit/              # Fast, many tests — run first
pytest tests/integration/       # Slower, moderate count — run on pass
pytest tests/e2e/               # Slow, few tests — run last

Service virtualization — isolate tests from external dependencies using test doubles, wiremock, or LocalStack for AWS services. Integration tests should run against real databases but within the pipeline environment (Ephemeral PostgreSQL container).

Test reporting — publish test results in JUnit format for trend analysis and failure categorization. GitHub Actions and GitLab CI both support JUnit reporting with historical dashboards.

Secrets Management

Pipeline secrets (API keys, cloud credentials, database passwords) require secure handling:

Never hardcode — secrets in repository source code are the most common CI/CD security breach
Repository secrets — GitHub Actions encrypted secrets, GitLab CI variables (masked in logs)
Cloud-native secret stores — read secrets from AWS Secrets Manager, GCP Secret Manager, Azure Key Vault at pipeline runtime, not stored as variables
OpenID Connect (OIDC) — exchange pipeline identity for cloud provider credentials without storing any long-lived secret. GitHub Actions and GitLab CI support OIDC with AWS, GCP, and Azure:

# GitHub Actions with AWS OIDC
permissions:
  id-token: write   # Required for OIDC
  contents: read

jobs:
  deploy:
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-deploy-role
          aws-region: us-east-1

Best Practices

Fast pipelines — every commit should complete the pipeline in under 10 minutes. Parallelize test jobs, cache dependencies, and use matrix builds. A 45-minute pipeline encourages developers to bypass it.

Fail fast — run the fastest checks first (lint, unit tests). If they fail, abort the pipeline immediately. Don’t waste time building and deploying code that doesn’t compile.

Immutable artifacts — build once, promote through environments. The same artifact (container image, compiled binary) deployed to staging must be the exact same artifact deployed to production. Never rebuild for deployment.

Pipeline as code — the pipeline configuration lives in the repository alongside the application code. Changes to the pipeline follow the same review and testing process as code changes.

DORA metrics tracking — instrument your CI/CD system to track deployment frequency, lead time for changes, change failure rate, and time to restore service (MTTR). Aim for elite performance: daily deployments, less than one hour lead time, under 5% change failure rate, and under one hour MTTR.

FAQ

What is the difference between Continuous Delivery and Continuous Deployment? Continuous Delivery means every commit goes through the full pipeline and is ready for production deployment, but deployment requires a manual approval. Continuous Deployment means every commit that passes all pipeline stages is automatically deployed to production. Most organizations start with Continuous Delivery.

How do I handle database migrations in CI/CD? Run backward-compatible migrations (add columns, not remove) before deploying new application code. Use tools like Flyway, Alembic, or Prisma Migrate in a pre-deployment stage. Never run destructive migrations (DROP COLUMN) until the old code is no longer running.

Should I use a dedicated CI/CD tool or a cloud provider’s managed service? GitHub Actions and GitLab CI are excellent for most organizations. Jenkins provides maximum customization but requires significant maintenance. Cloud-native services (AWS CodePipeline, GCP Cloud Build, Azure Pipelines) integrate deeply with their ecosystem but lock you into that provider.

How do I test infrastructure changes in CI/CD? Use Terraform plan output validation, Infracost for cost estimation, and Checkov or tfsec for security scanning of IaC templates. Create ephemeral environments per pull request using Pulumi or Terraform workspaces — destroy them when the PR closes.

What is a change failure rate and how is it measured? Change failure rate is the percentage of deployments that cause a production incident requiring remediation (rollback, hotfix, or patch). Calculate it as: (number of failed deployments / total deployments) × 100. Elite teams maintain a change failure rate under 5%, while low-performing teams see rates above 30%.

Docker Containers Guide — Kubernetes Guide — Cloud-Native Development