GitLab CI/CD: Advanced Pipeline Configuration

GitLab CI/CD is an integral part of the GitLab DevOps platform. Unlike standalone CI tools that require integration with separate version control, GitLab CI/CD is built into the same application that hosts repositories, manages issues, and tracks deployments. This tight integration enables features like pipeline security scanning, container registry, and environment management without third-party tools. This guide covers advanced GitLab CI/CD concepts beyond the basic .gitlab-ci.yml setup.

Pipeline Architecture in GitLab

GitLab pipelines are defined in .gitlab-ci.yml at the repository root. The configuration file specifies stages, jobs, and execution rules. Each job belongs to a stage, and all jobs in a stage must complete before the next stage begins — unless DAG (Directed Acyclic Graph) is used to override this default behavior.

The Default Pipeline Model

The standard pipeline model runs stages sequentially with parallel jobs within each stage:

stages:
  - build
  - test
  - deploy

build-frontend:
  stage: build
  script: npm run build

build-backend:
  stage: build
  script: mvn package

test:
  stage: test
  script: npm test

Both build-frontend and build-backend run in parallel. The test job waits for both to complete before starting. The deploy stage requires all previous stages to pass.

This sequential stage model is simple and predictable but introduces idle time: test must wait for both build jobs even if it only needs one artifact. DAG pipelines solve this.

DAG Pipelines with needs

The needs keyword defines explicit dependencies between jobs, overriding the sequential stage model:

test-frontend:
  stage: test
  needs: ["build-frontend"]
  script: npm test

test-backend:
  stage: test
  needs: ["build-backend"]
  script: mvn test

deploy:
  stage: deploy
  needs: ["test-frontend", "test-backend"]
  script: deploy.sh

test-frontend starts as soon as build-frontend completes, without waiting for build-backend. test-backend starts when build-backend finishes. Both test jobs run in parallel. deploy waits for both test jobs. The total pipeline time is reduced from max(build) + test + deploy to max(build + test) + deploy.

DAG pipelines are essential for monorepos where independent components can be built and tested in parallel. GitLab automatically renders the DAG in the pipeline graph view.

Include Files for Configuration Reuse

GitLab CI’s include keyword imports configuration from external files, enabling DRY pipeline definitions across projects. Four include methods are supported:

Local Includes

include:
  - local: templates/deploy.yml

This imports jobs from templates/deploy.yml in the same repository. Local includes are the most common approach for monorepo pipeline organization.

Project Includes

include:
  - project: devops/pipeline-templates
    ref: v2.1
    file: /templates/ruby.yml

Project includes reference files in other repositories, enabling an organization-wide pipeline template library. Version pinning (ref) ensures stable, predictable includes.

Remote Includes

include:
  - remote: https://gitlab.com/example/pipeline-templates/-/raw/main/lint.yml

Remote includes fetch from any URL. This is useful for consuming community-provided templates.

Template Includes

include:
  - template: Jobs/SAST.gitlab-ci.yml

GitLab provides built-in templates for security scanning, code quality, and dependency scanning. Including these templates enables industry-standard checks with zero configuration.

Include Strategy

Includes are recursive — included files can include other files. The strategy keyword controls error handling:

include:
  - local: templates/optional.yml
    strategy: never

strategy: never prevents pipeline failure if the included file does not exist, enabling optional optional templates.

Cache and Artifact Strategies

GitLab CI distinguishes between caches (dependency directories preserved for build speed) and artifacts (build outputs passed between stages).

Cache Optimization

The cache keyword preserves package manager directories across pipeline runs:

cache:
  key:
    files:
      - package-lock.json
  paths:
    - node_modules/

The files cache key uses a checksum of the lock file, automatically invalidating the cache when dependencies change. For monorepos, cache per job:

frontend-cache:
  cache:
    key: frontend-$CI_COMMIT_REF_SLUG
    paths:
      - frontend/node_modules/

Artifacts for Stage Handoff

Artifacts pass files between stages:

build:
  stage: build
  artifacts:
    paths:
      - dist/
    expire_in: 30 days

deploy:
  stage: deploy
  needs: ["build"]
  script: deploy dist/

Artifacts are automatically downloaded in subsequent stages. expire_in prevents storage accumulation. GitLab’s artifact retention policies can be configured at the project, group, or instance level.

Dependency Control

The dependencies keyword limits which artifacts a job downloads:

test-frontend:
  stage: test
  dependencies: ["build-frontend"]
  script: npm test

Without dependencies, a job downloads artifacts from all preceding jobs in the DAG. Explicit dependencies reduce network transfer and job startup time.

Child-Parent Pipelines

Child-parent pipelines (also called multi-project pipelines) enable pipeline composition across repositories. A parent pipeline triggers child pipelines in other projects and aggregates their results.

Triggering Child Pipelines

staging:
  stage: deploy
  trigger:
    project: myorg/infrastructure
    branch: main
    strategy: depend

The strategy: depend keyword makes the parent pipeline wait for the child to complete. The child pipeline’s status determines the parent’s status. Without depend, the parent continues immediately — useful for fire-and-forget notification pipelines.

Dynamic Child Pipelines

Child pipelines can be generated dynamically using a job that creates a YAML file:

generate-config:
  stage: build
  script: generate-pipeline-config.sh
  artifacts:
    paths:
      - generated-config.yml

dynamic-pipeline:
  stage: deploy
  trigger:
    include:
      - artifact: generated-config.yml
        job: generate-config
    strategy: depend

Dynamic child pipelines enable build-matrix generation based on repository contents. A monorepo with 50 microservices can generate a pipeline with 50 child jobs — one per service — without hardcoding each job in .gitlab-ci.yml.

Cross-Project Pipeline Triggers

The CI/CD pipeline in one project can trigger pipelines in other projects. This is the foundation for GitLab’s multi-project pipeline visualizations.

deploy-shared-library:
  stage: deploy
  trigger: myorg/shared-library

The trigger keyword accepts a project path. The downstream pipeline appears in the upstream pipeline’s graph, providing end-to-end visibility across the delivery chain.

Passing Variables to Downstream Pipelines

deploy-infrastructure:
  stage: deploy
  trigger:
    project: myorg/infrastructure
    branch: main
  variables:
    IMAGE_TAG: $CI_COMMIT_SHORT_SHA
    ENVIRONMENT: staging

Variables propagate to downstream pipelines using standard GitLab CI variable syntax.

Environment Management

GitLab environments track deployments with rich metadata. Each deployment records the commit, job, and external URL. The environment dashboard shows deployment history, rollback options, and current state across environments:

deploy-production:
  stage: deploy
  environment:
    name: production
    url: https://app.example.com
    on_stop: stop-production
  script: deploy.sh

stop-production:
  stage: deploy
  environment:
    name: production
    action: stop
  script: teardown.sh
  when: manual

The on_stop action defines a cleanup job that runs manually or on pipeline failure. GitLab environments integrate with Prometheus for deployment-annotated metrics, overlaying release events on performance graphs.

Rules and Conditions

GitLab CI’s rules keyword provides fine-grained control over job execution, replacing the deprecated only/except syntax. Rules evaluate conditions and determine whether a job runs, its variables, and its behavior:

deploy-production:
  stage: deploy
  rules:
    - if: '$CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"'
      when: manual
      allow_failure: false
    - if: '$CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "schedule"'
      when: never
    - when: never

Rules support complex boolean expressions combining CI variables, branch names, pipeline sources, and changes to specific files. The last rule with no conditions acts as the default fallback.

Pipeline Efficiency Optimization

Parallel Job Execution

Large test suites benefit from parallel execution across multiple jobs:

test:
  parallel: 7
  script: rspec -- $CI_NODE_INDEX/$CI_NODE_TOTAL

GitLab splits the job into 7 parallel instances, each running independently. CI_NODE_INDEX and CI_NODE_TOTAL environment variables distribute the workload. Test suites can be split by file patterns or test timing data.

Resource Groups

Resource groups prevent concurrent job execution on shared resources — useful for database migrations or deployment to environments that cannot handle parallel updates:

deploy-prod:
  stage: deploy
  resource_group: production
  script: deploy.sh

Only one job in the production resource group runs at a time. Subsequent jobs queue until the running job completes.

Recommended Internal Links

• CI/CD Guide: Continuous Integration and Deployment Explained — foundational CI/CD concepts for GitLab pipeline design
• CI/CD Tools Comparison: Jenkins vs GitHub Actions vs GitLab vs CircleCI — how GitLab CI compares to other platforms
• Security in CI/CD: Secrets Management, SAST, and Supply Chain Security — GitLab’s built-in security scanning features

Frequently Asked Questions

What is the difference between GitLab CI and GitLab SaaS runners?

GitLab CI is the pipeline execution engine. GitLab SaaS runners are the infrastructure that executes jobs — managed by GitLab and offered as a hosted service. Self-managed GitLab instances can also run their own runners. The engine and the runner are separate: GitLab CI coordinates job execution, while runners execute the actual script commands.

How do I pass variables between GitLab CI jobs?

Variables are isolated per job. Use artifacts to pass files between jobs. Use the dotenv report type to pass key-value pairs:

job1:
  artifacts:
    reports:
      dotenv: build.env
  script: echo "VERSION=v2.0" > build.env

job2:
  needs: ["job1"]
  script: echo $VERSION

dotenv artifacts are loaded into the environment of dependent jobs.

How do I debug a GitLab CI job?

Enable debug logging by setting CI_DEBUG_TRACE=true as a CI variable. GitLab provides interactive web terminals for running jobs (when enabled by the administrator). Use before_script to inspect the environment:

before_script:
  - env | sort
  - pwd
  - ls -la

What is the GitLab CI/CD component registry?

Introduced in GitLab 16, the CI/CD component registry is a repository for reusable pipeline components — similar to GitLab CI includes but with versioning, discovery, and dependency management. Components are published to the GitLab project component registry and consumed with include: component:.

Conclusion

GitLab CI/CD’s advanced features — DAG pipelines, include files, child-parent pipelines, and parallel execution — enable sophisticated CI/CD architectures that scale across monorepos, multi-project systems, and large organizations. The tight integration with GitLab’s platform eliminates the integration overhead of mixing separate tools. As GitLab continues investing in the CI/CD component registry and pipeline visualization, its position as an all-in-one DevOps platform strengthens.

For a comprehensive overview, read our article on Artifact Management.

For a comprehensive overview, read our article on Ci Cd Best Practices.