GitLab CI: Caching Policy

What is cache?

It's a set of files that a job can download before running and upload after execution. By default, the cache is stored in the same place where GitLab Runner is installed. If the distributed cache is configured, S3 works as storage.

GitLab Cache

Let's suppose you run a Pipeline for the first time with a local cache. The job will not find the cache but will upload one after the execution to runner01. The second job will execute on runner02, it won't find the cache on it either and will work without it. The result will be saved to runner02. Lint, the third job, will find the cache on runner01 and use it (pull). After execution, it will upload the cache back (push).

What are artifacts?

Artifacts are files stored on the GitLab server after a job is executed. Subsequent jobs will download the artifact before script execution.

GitLab artifacts

Build job creates a DEF artifact and saves it on the server. The second job, Test, downloads the artifact from the server before running the commands. The third job, Lint, similarly downloads the artifact from the server.

Uses

=======

image: docker:dind

test:

? stage: test

? script: gradle check

? cache:

??? key: "$CI_COMMIT_REF_NAME"

??? policy: pull

??? paths:

????? - node_modules/

? only:

??? refs:

????? - /^feature[0-9]+$/ # https://regex101.com/r/SMeXVe/1

build:

? stage: build

? script:

??? - npm install

? cache:

??? key: "$CI_COMMIT_REF_NAME"

??? policy: push

??? paths:

????? - build

?

deploy:

? image: alpine

? stage: deploy

? before_script:

??? # Get ssh keys from target server and add them to known_hosts

??? # Add the private key defined in PRIVATE_KEY variable

? script:

??? - echo "Deploying the system"

??? - ssh root@$PRODUCTION_HOST -p 22 -4 "mkdir -p /opt/webapps"

??? - scp . $PRODUCTION_HOST:/opt/webapps/

??? - ssh root@$PRODUCTION_HOST -p 22 -4 "docker-compose -f /opt/webapps/docker-compose-prod.yml up -d --force-recreate"

? cache:

??? key: "$CI_COMMIT_REF_NAME"

??? policy: pull

??? paths:

????? - build

? only:

??? - master

?

Uses of cache in Test Stage

key: "$CI_COMMIT_REF_NAME": Caches are named based on the branch name or commit reference.

policy: pull: Only pulls the cache (does not push updates to the cache).

paths: - node_modules/: Specifies the node_modules/ directory to cache, which is useful for speeding up builds by reusing dependencies.

?

Uses of cache in Build Stage

key: "$CI_COMMIT_REF_NAME": Caches are named based on the branch name or commit reference.

policy: push: Both pulls and pushes cache updates (useful for saving build artifacts).

paths: - build: Specifies the build directory to cache.

?

Uses of cache in Deploy Stage

key: "$CI_COMMIT_REF_NAME": Caches are named based on the branch name or commit reference.

policy: pull: Only pulls the cache (does not push updates to the cache).

paths: - build: Specifies the build directory to cache.

?

One More Example

----------------------

image: node: 16.3.0 # (1)

stages:

? - setup

? - test

variables:

? npm_config_cache: "$CI_PROJECT_DIR/.npm" (5)

.dependencies_cache:

? cache:

??? key:

????? files:

??????? - package-lock.json

??? paths:

????? - .npm

??? policy: pull

?

setup:

? stage: setup

? script:

??? - npm ci

? extends: .dependencies_cache

? cache:

??? policy: pull-push

? artifacts:

??? expire_in: 1h

??? paths:

????? - node_modules

?

test_async:

? stage: test

? script:

??? - node ./specs/start.js ./specs/async.spec.js

?

test_db:

? stage: test

? script:

??? - node ./specs/start.js ./specs/db-postgres.spec.js

?

Uses of cache in dependencies_cache

Configures caching for npm dependencies.

key: Uses the package-lock.json file to determine the cache key, which helps ensure the cache is updated when dependencies change.

paths: Specifies the .npm directory to cache, where npm stores its cache.

policy: pull: Indicates that the cache should only be pulled (i.e., read) and not updated.

?

Uses of cache and artifacts in setup

cache:

policy: pull-push: Indicates that the cache should be both pulled (read) and pushed (updated).

artifacts:

expire_in: 1h: Specifies that the node_modules directory will be saved as an artifact and expire after 1 hour.

paths: Defines node_modules as an artifact, which means it will be available for use in later stages or jobs.

?