CI/CD Pipeline: Tools and Best Practices
Continuous Integration and Continuous Deployment (CI/CD) automates the build, test, and deployment pipeline. A well-designed CI/CD pipeline catches bugs early, enforces code quality, and deploys reliably. While the tools differ in syntax and hosting, the pipeline stages and principles remain consistent across platforms.
Pipeline Stages
Every CI/CD pipeline follows a similar structure:
- Trigger — code push, pull request, or scheduled event
- Checkout — fetch the source code
- Install Dependencies — install required packages and tools
- Lint/Format — check code style and formatting
- Build — compile, bundle, or transpile
- Test — run unit, integration, and end-to-end tests
- Security Scan — SAST, dependency scanning, secret detection
- Package — create artifacts (binaries, Docker images, packages)
- Deploy — push to staging or production
- Notify — report results via Slack, email, or other channels
GitHub Actions
GitHub Actions uses YAML workflows stored in .github/workflows/:
name: CI Pipeline
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: '1.22'
- run: go mod download
- run: go vet ./...
- run: go test -race -coverprofile=coverage.out ./...
- uses: codecov/codecov-action@v4
with:
file: ./coverage.out
deploy:
needs: test
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: docker build -t myapp:latest .
- run: docker push myapp:latestGitLab CI
GitLab CI uses .gitlab-ci.yml:
stages:
- test
- build
- deploy
variables:
DOCKER_IMAGE: registry.gitlab.com/$CI_PROJECT_PATH
go-test:
stage: test
image: golang:1.22
script:
- go mod download
- go vet ./...
- go test -race -coverprofile=coverage.out ./...
artifacts:
paths:
- coverage.out
docker-build:
stage: build
image: docker:latest
services:
- docker:dind
script:
- docker build -t $DOCKER_IMAGE:$CI_COMMIT_SHA .
- docker push $DOCKER_IMAGE:$CI_COMMIT_SHA
deploy-production:
stage: deploy
image: alpine
before_script:
- apk add --no-cache curl
script:
- curl -X POST $DEPLOY_URL
only:
- mainJenkins
Jenkins uses a Jenkinsfile with Declarative Pipeline syntax:
pipeline {
agent any
stages {
stage('Checkout') {
steps {
checkout scm
}
}
stage('Test') {
parallel {
stage('Unit Tests') {
steps {
sh 'go test -short ./...'
}
}
stage('Lint') {
steps {
sh 'golangci-lint run'
}
}
}
}
stage('Build') {
steps {
sh 'docker build -t myapp:${BUILD_NUMBER} .'
}
}
stage('Deploy') {
when { branch 'main' }
steps {
sh 'kubectl set image deployment/myapp myapp=myapp:${BUILD_NUMBER}'
}
}
}
post {
failure {
slackSend(
color: '#FF0000',
message: "Pipeline failed: ${env.BUILD_URL}"
)
}
}
---CircleCI
CircleCI uses .circleci/config.yml:
version: 2.1
orbs:
go: circleci/go@1.7
jobs:
test:
docker:
- image: cimg/go:1.22
steps:
- checkout
- go/load-cache
- go/mod-download
- run: go test -race -cover ./...
- go/save-cache
deploy:
docker:
- image: cimg/base:current
steps:
- checkout
- setup_remote_docker
- run: docker build -t myapp:latest .
- run: docker push myapp:latest
workflows:
version: 2
build-test-deploy:
jobs:
- test
- deploy:
requires:
- test
filters:
branches:
only: mainPipeline Design Patterns
Fast Feedback
Run the fastest checks first. Linting and unit tests should complete in seconds. Integration and end-to-end tests run later or in parallel.
Parallel Stages
Run independent jobs in parallel to reduce total pipeline time:
# GitHub Actions
jobs:
lint:
unit-test:
integration-test:
needs: [lint, unit-test]Artifact Caching
Cache dependencies to speed up subsequent runs:
# GitHub Actions
- uses: actions/cache@v4
with:
path: ~/.cache/go
key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}Environment Promotion
Promote the same artifact through environments. Build once, deploy many times:
# Build stage produces artifact
- run: docker build -t myapp:$CI_COMMIT_SHA .
- run: docker push myapp:$CI_COMMIT_SHA
# Deploy uses the same artifact hash
- run: kubectl set image deployment/myapp myapp=myapp:$CI_COMMIT_SHATesting Stages
| Stage | Purpose | Speed | When | |
Best Practices
- Build once, deploy many — promote immutable artifacts through environments
- Fail fast — run the fastest checks first, stop the pipeline on failure
- Keep secrets in CI variables — never hardcode credentials in pipeline files
- Use status checks — block PR merges on pipeline failures
- Monitor pipeline duration — optimize slow stages with parallelism
- Version your CI configuration — treat
.github/workflows/*like application code - Run security scans — integrate SAST, dependency scanning, and secret detection
Conclusion
A robust CI/CD pipeline automates testing, building, and deployment while providing fast feedback to developers. Whether you choose GitHub Actions, GitLab CI, Jenkins, or CircleCI, the principles remain the same: fail fast, build once, and promote through environments. Start with a simple pipeline and add stages as your project grows.
Related: Learn about configuration management and infrastructure as code.
CI/CD Tool Selection Criteria
Self-Hosted vs. Cloud-Managed
The choice between self-hosted CI/CD and cloud-managed services involves trade-offs across cost, control, and maintenance. Self-hosted runners (GitHub Actions self-hosted, GitLab runners on your infrastructure) provide complete control over hardware specifications, network access, and security policies. They are essential for workloads requiring GPU access, specific hardware, or air-gapped environments. However, self-hosted runners require ongoing maintenance — updating runner software, patching the host OS, managing scaling during peak load, and handling failures. Cloud-managed services (GitHub Actions hosted, CircleCI, Travis CI) eliminate this overhead and provide automatic scaling, but impose limits on build minutes, concurrent jobs, and available machine types. Most organizations start with cloud-managed CI/CD and migrate to self-hosted when cost or requirements justify the operational burden.
Pipeline as Code and Versioning
Modern CI/CD systems treat pipeline definitions as code — YAML files stored in version control alongside application code. This practice enables code review for pipeline changes, audit trails for who changed what, and the ability to test pipeline changes in feature branches before merging. GitHub Actions uses workflow YAML files in .github/workflows/, GitLab CI uses .gitlab-ci.yml, and Jenkins supports Pipeline DSL in Jenkinsfiles. Pipeline-as-code reduces configuration drift — the pipeline definition is versioned and deployed with the application, ensuring that older branches use the correct pipeline configuration. Reusable workflow components (composite actions, GitLab includes, Jenkins shared libraries) promote consistency across multiple repositories and reduce duplication.
Security and Compliance Considerations
CI/CD pipelines are a high-value attack target — compromising the pipeline can inject malicious code into production. The SolarWinds and Codecov breaches demonstrated that trusted pipeline infrastructure can be exploited to distribute malware. Best practices include signing all pipeline artifacts, storing secrets in dedicated secret management tools (HashiCorp Vault, AWS Secrets Manager, GitHub Secrets), scanning dependencies for vulnerabilities during the build, and implementing approval gates for production deployments. Supply chain security frameworks like SLSA (Supply-chain Levels for Software Artifacts) provide maturity levels for build integrity. SBOM (Software Bill of Materials) generation during CI/CD enables downstream consumers to understand what dependencies are included in each release.
CI/CD Tool Ecosystem Overview
The CI/CD tool landscape has consolidated around a few dominant platforms. GitHub Actions has emerged as the most widely adopted CI/CD platform due to its native GitHub integration, generous free tier for public repositories, and extensive marketplace of community actions. GitLab CI offers a complete DevOps lifecycle within a single application — from source control through CI/CD to monitoring. Jenkins remains the most flexible option with over 1,800 plugins but requires significant operational investment. CircleCI and Travis CI provide cloud-managed services with fast execution through parallel test splitting. Newer entrants like Buildkite offer hybrid models where you manage your own build agents while Buildkite handles coordination and UI. For container-native workflows, Tekton provides Kubernetes-native CI/CD pipelines with custom resource definitions. Argo Workflows extends CI/CD to complex DAG-based workflows for data processing and ML pipelines. The trend is toward integration with cloud provider services (AWS CodePipeline, Azure DevOps, Google Cloud Build) and toward Git-native configuration where pipeline definitions live alongside application code.
Build Optimization Techniques
Build caching is the single most effective optimization for CI/CD pipelines. Docker layer caching reuses unchanged image layers across builds, avoiding repeated downloads and compilation. Dependency caching stores downloaded packages (npm node_modules, Maven .m2, pip cache) between runs — restoring them at the start of each job avoids re-downloading the internet on every commit. Content-based caching uses checksums of input files as cache keys, invalidating only when dependencies change. Incremental compilation compiles only files that have changed since the last build, dramatically reducing build times for large monorepos. Tools like Nx, Turborepo, and Bazel compute dependency graphs and only rebuild affected packages. Distributed builds across multiple machines (using tools like Buck, Pants) parallelize compilation across CPU cores on different machines. Remote execution (BuildBuddy, EngFlow) runs build actions on cloud infrastructure with abundant resources, reducing local bottlenecks.
Frequently Asked Questions
What is the difference between CI and CD? CI (Continuous Integration) automatically builds and tests code changes when they are pushed to a shared repository, ensuring that integration issues are caught early. CD (Continuous Delivery/Deployment) automatically deploys validated code to environments. Continuous Delivery requires manual approval for production deployment; Continuous Deployment automates the entire pipeline to production.
Which CI/CD tool is best for my team? The best tool depends on your ecosystem. GitHub Actions integrates natively with GitHub repositories and has a large marketplace. GitLab CI is tightly integrated with GitLab’s DevOps platform. Jenkins offers maximum flexibility through plugins but requires significant maintenance. For new projects, choose the tool that integrates most naturally with your existing version control and infrastructure.
How do I optimize CI/CD pipeline speed? Parallelize independent jobs, cache dependencies between runs, use incremental builds, run only the tests affected by changed code, optimize Docker image layers, and use faster hardware. Pre-build Docker images with dependencies installed to avoid re-downloading packages on every run. Split long test suites into parallel shards running on concurrent runners.