Jenkins Pipeline Guide: Declarative and Scripted Pipelines

CI/CD CI/CD 8 min read 1620 words Beginner ExcellentWiki Editorial Team

Jenkins remains the most widely deployed self-hosted CI/CD automation server. Its Plugin ecosystem of over 1,800 extensions integrates with virtually every development and deployment tool. Jenkins Pipelines — defined as code in Jenkinsfiles — are the foundation of Jenkins-based CI/CD. This guide covers declarative and scripted pipeline syntax, shared libraries, multi-branch pipelines, and production operational practices.

Declarative vs. Scripted Pipelines

Jenkins supports two pipeline syntax styles. Both are written in Groovy and stored in a Jenkinsfile at the repository root, but they differ in structure, flexibility, and learning curve.

Declarative Pipeline

Declarative syntax provides a structured, opinionated pipeline model:

pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                echo 'Building...'
                sh 'mvn compile'
            }
        }
        stage('Test') {
            steps {
                echo 'Testing...'
                sh 'mvn test'
            }
        }
        stage('Deploy') {
            when {
                branch 'main'
            }
            steps {
                echo 'Deploying...'
                sh 'deploy.sh'
            }
        }
    }
    post {
        always {
            echo 'Pipeline completed'
        }
        failure {
            emailext subject: "Pipeline Failed: ${env.JOB_NAME}",
                     body: "Check ${env.BUILD_URL}"
        }
    }
---

Declarative pipelines enforce a structure: pipeline block → agent → stages → stage → steps. The post section handles cleanup and notifications. The when condition controls stage execution. This structure makes declarative pipelines easier to read, validate, and review — especially for teams with mixed experience levels.

Scripted Pipeline

Scripted pipelines use the full Groovy language, offering maximum flexibility at the cost of structure:

node {
    stage('Checkout') {
        checkout scm
    }
    stage('Build') {
        sh 'mvn compile'
    }
    stage('Test') {
        try {
            sh 'mvn test'
        } catch (e) {
            currentBuild.result = 'UNSTABLE'
        }
    }
    stage('Deploy') {
        if (env.BRANCH_NAME == 'main') {
            sh 'deploy.sh'
        }
    }
---

node allocates an executor and workspace. Stages are blocks with arbitrary Groovy code inside. Error handling uses standard try/catch. This flexibility is powerful but creates pipelines that are harder to review and maintain.

Which One to Choose?

Declarative is the recommended starting point for all new pipelines. The structured format enables Pipeline Stage View visualization, built-in error handling (post), and declarative directives (when, environment, parameters). Use scripted only when declarative’s opinionated structure prevents a required pattern — for example, complex parallel branch logic with dynamic stage generation or custom error-handling flows that span multiple stages.

Shared Libraries

Shared libraries are Jenkins’ mechanism for reusing pipeline code across repositories. A shared library is a Groovy script repository with a defined directory structure:

vars/
  buildJar.groovy
  deployToK8s.groovy
src/
  org/myorg/PipelineUtils.groovy
resources/
  templates/
    deployTemplate.groovy

Defining a Shared Library

The vars/ directory contains global functions callable from any pipeline:

// vars/buildJar.groovy
def call(String projectName) {
    sh "mvn -f ${projectName}/pom.xml package"
---

Using a Shared Library

// Jenkinsfile
@Library('my-pipeline-library') _

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                buildJar('payment-service')
            }
        }
    }
---

Libraries are configured in Jenkins under Manage Jenkins → Configure System → Global Pipeline Libraries. Version control (Git, GitHub, GitLab) is mandatory, and versions are pinned for reproducibility. Shared libraries should be treated as production software — they need their own CI/CD pipeline, testing, and release process.

Structured Libraries

For libraries with significant logic, use src/ for object-oriented Groovy classes:

// src/org/myorg/PipelineUtils.groovy
package org.myorg

class PipelineUtils {
    static String buildVersion(String branch) {
        return branch == 'main' ? '1.0.0' : "${branch}-SNAPSHOT"
    }
---

Called as org.myorg.PipelineUtils.buildVersion(env.BRANCH_NAME) from the pipeline. Structured libraries enable unit testing with tools like Jenkins Pipeline Unit, which can mock Jenkins steps and validate shared library behavior in isolation.

Multi-Branch Pipelines

Multi-branch pipelines automatically create a pipeline for every branch in a repository that contains a Jenkinsfile. Jenkins scans the repository, discovers branches, and creates pipeline entries for each. The visual stage view in Jenkins shows results for every branch, making it easy to identify failing branches.

Branch-Specific Behavior

stage('Deploy') {
    when {
        branch 'main'
    }
    steps {
        deployToProduction()
    }
---
stage('Deploy Staging') {
    when {
        branch 'develop'
    }
    steps {
        deployToStaging()
    }
---

Multi-branch pipelines also handle pull requests. The changeRequest condition runs pipelines for PR builds:

when {
    changeRequest()
---

Organization Folders

For multi-repository setups, Organization Folders (formerly GitHub Organization, Bitbucket Team, GitLab Group) scan all repositories owned by a GitHub organization, GitLab group, or Bitbucket team. Jenkins auto-discovers repositories with Jenkinsfiles and creates corresponding pipelines. This eliminates per-repository Jenkins job creation. Combined with GitHub Branch Source plugin or Bitbucket Branch Source plugin, Organization Folders provide a self-service CI onboarding experience — developers add a Jenkinsfile to their repository and CI is automatically configured.

Production Jenkins Operations

High Availability

Jenkins’ default architecture uses a single controller. Production deployments require high-availability patterns.

Active-passive failover with a shared filesystem (NFS) for JENKINS_HOME. The standby controller starts when the primary fails. This prevents data loss but has a recovery time measured in minutes.

Kubernetes-based Jenkins runs the controller as a Deployment with persistent volume claims. Kubernetes liveness probes restart unhealthy controllers automatically. Build agents run as ephemeral pods, scaling based on demand. The Jenkins Kubernetes plugin dynamically provisions agent pods, eliminating the need for pre-configured agent machines.

External build caches (Nexus, Artifactory) store plugin downloads and build outputs, enabling fast agent startup without downloading the internet on every new agent. Configuration as code further accelerates recovery — a destroyed Jenkins controller can be fully recreated from version-controlled YAML in minutes.

Security Configuration

Jenkins’ default security settings are insufficient for production. Required hardening:

Matrix-based security for fine-grained permission control across teams and projects
Agent-to-controller security — filesystem access controls that prevent agents from reading controller files or accessing other builds
Credential provider plugins — HashiCorp Vault, AWS Secrets Manager, Azure Key Vault instead of Jenkins-internal credential storage
Role-based access for multi-team Jenkins instances to prevent cross-team visibility of sensitive jobs
Audit trails via the Audit Trail plugin or external logging shipping to a SIEM

Jenkins Configuration as Code

The Jenkins Configuration as Code (JCasC) plugin defines Jenkins master configuration in YAML files. Instead of configuring Jenkins through the web UI — which is error-prone, unreproducible, and difficult to audit — JCasC declares the configuration in version-controlled files:

jenkins:
  systemMessage: "Jenkins managed by JCasC"
  securityRealm:
    ldap:
      configurations:
        - server: ldap.example.com
          rootDN: "dc=example,dc=com"
  authorizationStrategy:
    globalMatrix:
      permissions:
        - "Overall/Administer:admin-team"
        - "Overall/Read:authenticated"

JCasC configuration is applied at Jenkins startup and reloaded on demand. Combined with Docker-based Jenkins controller images, this enables fully reproducible Jenkins deployments — destroy and recreate the controller from code with no manual steps.

Pipeline Optimization

Build time optimization for Jenkins follows the same principles as other CI platforms with Jenkins-specific considerations:

Parallel stages using parallel keyword in declarative or parallel block in scripted to run independent jobs concurrently
Stash/unstash for passing files between stages on different agents without external artifact storage
Pipeline caching of dependency directories using tools and custom cache logic to preserve ~/.m2, ~/.npm, or vendor/bundle between builds
Incremental builds triggered only for changed modules in monorepos using tools like Bazel or custom change detection

Troubleshooting Common Issues

Pipeline Syntax Parsing Errors

Jenkins parses the entire Jenkinsfile before execution. Syntax errors in when conditions, environment blocks, or post conditions fail the pipeline immediately. Validate syntax locally using the Jenkins Pipeline Linter available at http://jenkins-url/pipeline-syntax/validate or the command-line -pipeline-lint tool.

Groovy Sandbox Restrictions

Untrusted Jenkinsfiles run in a Groovy sandbox that restricts method calls. Script approval is required for methods outside the approved whitelist. Use @NonCPS annotations for methods that must bypass serialization. For shared libraries, run them outside the sandbox by trusting the library source in the Jenkins global configuration.

Jenkins Out-of-Memory

Jenkins controllers manage all pipeline execution state in memory. Hundreds of concurrent pipelines can exhaust heap. Configure JVM heap settings (-Xmx) based on workload: 4 GB for moderate use, 8–16 GB for high-throughput environments. Monitor heap usage through the Jenkins Monitoring plugin or JMX metrics, and configure GC logging to diagnose memory issues before they cause outages.

Frequently Asked Questions

What is the difference between a Jenkins freestyle job and a pipeline?

Freestyle jobs are configured through the Jenkins web UI — checkboxes, text fields, and dropdowns. Pipelines are code defined in a Jenkinsfile. Freestyle jobs cannot be version-controlled and require UI interaction to modify. Pipelines are Git-native, support pull request workflows, and provide the Stage View visualization. All new Jenkins automation should use pipelines.

How do I manage Jenkins plugins?

Use the Plugin Management API or Jenkins Configuration as Code (JCasC) to version plugin sets. Pin plugin versions in a YAML file and apply through JCasC. Use the Plugin Stats plugin to identify unused plugins. Remove plugins that are not actively used — each plugin adds attack surface and upgrade risk. Establish a regular plugin update cadence and test updates in a staging Jenkins instance before applying to production.

Is Jenkins still relevant with GitHub Actions and GitLab CI?

Yes, for specific use cases. Jenkins excels in environments requiring extreme customization, air-gapped infrastructure, or integration with legacy tools that lack modern CI APIs. Jenkins also remains the default choice for organizations with multi-year Jenkins investments and mature pipeline libraries. For greenfield projects without these constraints, cloud-native CI is typically a better fit.

Conclusion

Jenkins Pipelines provide the most mature, flexible CI/CD automation framework available. Declarative syntax offers structure and safety for standard pipelines. Scripted syntax enables complex orchestration when needed. Shared libraries multiply pipeline patterns across entire organizations. Multi-branch pipelines eliminate per-branch configuration. For teams that need maximum control over their CI/CD infrastructure and are willing to invest in operational management, Jenkins remains unmatched in capability.

For a comprehensive overview, read our article on Artifact Management.

For a comprehensive overview, read our article on Ci Cd Best Practices.

Share this article

X LinkedIn Facebook Email