CI/CD Pipelines: A Complete Guide
Continuous Integration and Continuous Delivery (CI/CD) is the practice of automating the building, testing, and deployment of software. A CI/CD pipeline catches bugs early, enforces quality gates, and ships features faster. Instead of manually running tests and copying files to servers, you define a pipeline once and let it run on every change.
This guide covers the core concepts, popular tools, and practical patterns for building effective CI/CD pipelines.
What Is a CI/CD Pipeline?
A CI/CD pipeline is a series of automated steps that your code goes through from commit to production. Each step validates or transforms the code, and if any step fails, the pipeline stops.
Continuous Integration (CI) means merging code changes frequently — several times a day — and running automated tests on every merge. The goal is to detect integration problems immediately.
Continuous Delivery (CD) means every passing CI build is automatically prepared for release. Deployment to production may still be a manual decision, but the release artifact is always ready.
Continuous Deployment goes further — every passing build is automatically deployed to production with no human intervention.
The Pipeline Stages
A typical CI/CD pipeline has these stages:
- Source — Triggered by a push or pull request
- Build — Compile code, install dependencies
- Test — Run unit, integration, and linting checks
- Package — Build a Docker image or release artifact
- Deploy — Push to staging or production
GitHub Actions
GitHub Actions is a CI/CD platform integrated directly into GitHub repositories. Pipelines are defined as YAML workflows in .github/workflows/.
Basic Workflow
name: CI
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
- run: npm ci
- run: npm testThis workflow runs tests on every push and pull request to main. It checks out the code, sets up Node.js, installs dependencies, and runs tests.
Multi-Job Pipeline
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npm ci
- run: npm run lint
test:
needs: lint
runs-on: ubuntu-latest
strategy:
matrix:
node: [18, 20, 22]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node }}
- run: npm ci
- run: npm test
deploy:
needs: test
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
steps:
- run: echo "Deploying..."The needs keyword creates dependencies between jobs. The matrix strategy runs tests across multiple Node versions in parallel.
Jenkins
Jenkins is a self-hosted automation server with a rich plugin ecosystem. Pipelines are defined either through the web UI or as code using a Jenkinsfile.
Jenkinsfile (Declarative Pipeline)
pipeline {
agent any
stages {
stage('Checkout') {
steps {
checkout scm
}
}
stage('Build') {
steps {
sh 'npm ci'
sh 'npm run build'
}
}
stage('Test') {
steps {
sh 'npm test'
}
post {
always {
junit 'reports/**/*.xml'
}
}
}
stage('Deploy') {
when {
branch 'main'
}
steps {
sh 'docker build -t myapp .'
sh 'docker push myapp:latest'
}
}
}
---Jenkins pipelines are Groovy-based. The post block runs after each stage, which is useful for collecting test reports or cleaning up resources.
Blue Ocean and Plugins
Jenkins excels in environments that need custom plugins, complex build matrices, or integration with on-premise tools. The Blue Ocean UI provides a modern visualization of pipeline runs. Popular plugins include Docker Pipeline, Kubernetes, and Slack Notification.
GitLab CI
GitLab CI is built into GitLab repositories. Pipelines are defined in a .gitlab-ci.yml file at the project root.
Basic Configuration
stages:
- build
- test
- deploy
image: node:20
cache:
paths:
- node_modules/
build:
stage: build
script:
- npm ci
- npm run build
artifacts:
paths:
- dist/
test:
stage: test
script:
- npm test
coverage: '/Statements\s*:\s*(\d+\.\d+)%/'
deploy:
stage: deploy
script:
- echo "Deploying $CI_COMMIT_SHORT_SHA"
only:
- mainGitLab CI uses runners — agents that execute jobs. You can use GitLab’s shared runners or install your own. The artifacts keyword passes build outputs between stages.
Best Practices
Keep Pipelines Fast
Long pipelines discourage developers from pushing frequently. Optimize by running fast tests first, using dependency caching, and parallelizing independent jobs.
# GitHub Actions — cache dependencies
- uses: actions/cache@v4
with:
path: ~/.npm
key: ${{ runner.os }}-node-${{ hashFiles('package-lock.json') }}Fail Fast
Design your pipeline so the fastest, most likely-to-fail checks run first. Linting and unit tests should run before integration tests and deployment.
Secure Secrets
Never hardcode credentials in pipeline files. Use secret management:
- GitHub Actions: Repository secrets via Settings → Secrets
- Jenkins: Credentials plugin with encrypted storage
- GitLab CI: CI/CD Variables in project settings
Use Environment-Specific Config
Separate staging and production deployments with different variables and approval gates:
deploy-staging:
stage: deploy
script: deploy staging
only:
- main
deploy-production:
stage: deploy
script: deploy production
only:
- main
when: manual # requires manual approvalThe when: manual directive in GitLab CI creates a gate that requires human approval before deployment.
Common Pipeline Patterns
Monorepo with Path Filters
When multiple projects live in one repository, only trigger pipelines for changed paths:
# GitHub Actions — path filtering
on:
push:
paths:
- 'frontend/**'
- 'backend/**'
- '.github/workflows/**'Preview Environments
Deploy every pull request to an isolated environment for review. Tear it down when the PR is merged or closed. This pattern is especially useful for web applications where reviewers want to see changes live.
Dependency Scanning
Integrate security scanning into your pipeline:
# GitLab CI — dependency scanning
include:
- template: Jobs/Dependency-Scanning.gitlab-ci.ymlSummary
CI/CD pipelines automate the path from code commit to production deployment. Choose GitHub Actions for tight GitHub integration, Jenkins for maximum flexibility and on-premise control, or GitLab CI for an all-in-one DevOps platform. Regardless of the tool, focus on fast feedback, secure practices, and incremental deployment strategies.
Pipeline Stages and Best Practices
Source and Build Stage
The source stage triggers the pipeline when changes are pushed to version control. Branch policies enforce that all changes go through pull requests and pass required checks before merging. The build stage compiles the application and runs unit tests. Dependency caching — storing downloaded packages between runs — reduces build time significantly, especially for languages with large dependency trees. For compiled languages, incremental compilation (only recompiling changed files) speeds iterations. Docker-based builds ensure reproducibility: the same commit always produces the same artifact. Build artifacts should be immutable — each build produces a uniquely identified artifact (Docker image tag with commit SHA, JAR with version) that is never overwritten. Immutable artifacts guarantee that what was tested in staging is exactly what is deployed to production.
Test Stage
The test stage should operate as a funnel: fast, high-confidence tests run first, and slower tests run only after faster tests pass. Unit tests execute in milliseconds and provide the first feedback loop. Integration tests verify interactions between components — database queries, API contracts, message queue operations — and typically take seconds to minutes. End-to-end tests simulate real user workflows and take minutes to tens of minutes. The test pyramid principle guides this structure: many unit tests, fewer integration tests, and few end-to-end tests. Parallel test execution across multiple runners or shards reduces total pipeline duration. Flaky tests — those that fail nondeterministically — must be quarantined immediately, as they erode trust in the pipeline and mask real failures.
Deploy and Release Stage
Progressive delivery deploys changes gradually to reduce blast radius. Canary releases route a small percentage of traffic to the new version, monitoring error rates and performance before increasing the percentage. Blue-green deployments maintain two identical environments — the new version is deployed to the inactive environment, then traffic switches over. Feature flags decouple deployment from release: code can be deployed to production but remain disabled behind a flag until the team is ready. Rollback strategies must be automated and tested. Database migrations are the highest-risk deployment operation — they require careful sequencing, backward-compatible changes, and automated rollback plans. Health checks and smoke tests after deployment verify that the running application is functional and responsive.
Frequently Asked Questions
What is the difference between a CI/CD pipeline and a build pipeline? A build pipeline focuses on compiling code and running tests, producing deployable artifacts. A CI/CD pipeline extends this with automated deployment through multiple environments (staging, production), approval gates, rollback capabilities, and integration with deployment infrastructure. Every CI/CD pipeline includes a build pipeline, but not every build pipeline includes deployment automation.
How do I handle database migrations in CI/CD? Database migrations require backward-compatible changes — add new columns as nullable, never rename columns, and only remove deprecated columns after the old code is fully retired. Use migration tools (Flyway, Liquibase, Alembic) that are executed as part of the deployment script. Test migrations against a staging database clone before applying to production.
Should I deploy on Fridays? Deploying on Friday afternoon is risky because if something goes wrong, the on-call team may be short-staffed over the weekend. Teams with mature CI/CD practices, comprehensive automated testing, and effective rollback mechanisms can deploy any day. Teams with less mature practices should deploy early in the week during business hours.
For a comprehensive overview, read our article on Blue Green Deployments.
For a comprehensive overview, read our article on Ci Cd Pipeline Tools.