Cache Files

File caching may improve build times and save resources. It works by storing project dependencies locally in a Space file repository. This way, when the same build steps are run again, the referenced packages can be quickly retrieved from the cache instead of being downloaded from a remote server.

Default cache repository

Projects in Space can have associated default file repositories for storing cached files for the subsequent job runs.

Each project has its own default file repository for cache with corresponding access restrictions (only for project members).
The repository is created automatically when you run the job that references the default repository (e.g., uploads cached files) for the first time.
In a script, you can get the name of the default file repository from the run:file-caches.default-repository parameter.
You can replace the default file repository with any other file repository in Packages.
The default name of the default repository is default-automation-caches. You can change this name in Jobs | Settings under File Storage | Cache Repository.
By default, if you upload a file that already exists in the default repository, the file is overwritten. Learn more
By default, cache files are deleted if they were not accessed in 7 days. You can change the default retention policy in the repository settings.

Upload and reuse cached files

To work with file cache, depending on your run environment, use either the job.container.cache or the job.host.cache block. For example, to cache dependencies for an npm project:

job("Build frontend") {
    container(displayName = "Build npm", image = "node") {
        cache {
            // To upload to another repo (e.g., 'my-file-repo'), uncomment the next line
            // location = CacheLocation.FileRepository(name = "my-file-repo", remoteBasePath = "caches/{{ run:job.repository }}")

            // Generate cache file name
            // Using a hash of the build file ensures all job runs with the
            // same package.json will share the cached dependencies
            storeKey = "npm-{{ hashFiles('package.json') }}"

            // Fallback option
            // If the right cache file is not found, get cache from 'npm-master.tar.gz'
            restoreKeys {
                +"npm-master"
            }

            // Local path to the cache file directory
            localPath = "node_modules"
        }

        shellScript {
            content = "npm install"
        }
    }
}

How caching works

The default cache location is caches/{{ run:job.repository }} directory in the default cache repository. Note how the default location is defined on the repository level – by default, all jobs in the same repository reuse the same cache keys. You can override the default location using the location parameter.

On job run, Automation checks whether the {storeKey}.tar.gz archive exists in the default location or in the user-specified location:

If the file exists, Automation:
1. Downloads the file and extracts it to the localPath.
2. Runs the script.
If the file doesn't exist, Automation:
1. Checks location (or the default repository) for archives specified in restoreKeys.
2. If a fallback cache archive is found, extracts it to localPath and runs the script.
3. If a fallback cache archive is not found, Automation:
  1. Runs the script.
  2. Creates a .tar.gz archive with the directory specified in localPath. As a file name, Automation uses the storeKey string.
  3. Uploads the .tar.gz archive to the default cache repository or to the repository specified in location.

Upload path and cache key names

If you don't specify location, Automation uploads the caches .tar.gz archive to the caches/{{ run:job.repository }} directory of the default default-automation-caches repository.

The cache file name is defined by the storeKey parameter value: {storeKey}.tar.gz. The name must be unique and must be unambiguously determined by the required project dependencies. For example, in a Node.js project, dependencies are specified in the packages.json file, so it makes sense to generate the storeKey based on the contents of packages.json. To help you solve this task, Automation provides the hashFiles(files: Collection<String>): String function. This function generates hash based on the contents of a file or number of files. To call the function, you must use the parameter syntax (double curly braces).

For example: storeKey = "npm-{{ hashFiles('package.json') }}" or storeKey = "cache_{{ hashFiles('server/build.gradle.kts', 'client/build.gradle.kts') }}".

How to specify localPath

localPath defines the location of the directory to be cached. You can specify it with:

An absolute path, for example, /home/root/.m2.
A path relative to the current home directory, for example, ~/.m2. You can use it only in job.host steps. Learn more
A path relative to the step working directory (by default, the project root), for example, node_modules.

Share cache between repositories

The default cache location is defined on the Git repository level as it uses a relative path which includes the repository name: caches/{{ run:job.repository }}.

If you want to share caches between Git repositories in the project, use the location parameter to specify an absolute path to caches. For example:

cache {
    location = CacheLocation.FileRepository(name = CacheLocation.DefaultRepositoryName, remoteBasePath = "my-shared-caches")
    storeKey = "npm-{{ hashFiles('package.json') }}"
    localPath = "node_modules"
}

Cache multiple directories

To cache multiple directories, you can create multiple caching rules. For example:

job("Example") {
    container(displayName = "Get caches from two dirs", image = "ubuntu") {
        cache {
            // all files and dirs from caches/dir1
            localPath = "caches/dir1"
            // other cache settings
        }

        cache {
            // all files and dirs from caches/dir2
            localPath = "caches/dir2"
            // other cache settings
        }

        // here goes the script...
    }
}

Cache files outside the default mount directory

If you run a step in a self-hosted or a cloud machine using job.host, you can simply specify the required absolute path to a directory. For example, localPath = "~/.m2/repository".

Things are different if you run a step in a container using job.container. Here you can reference files only under the mount directory, which is, /mnt/space by default. Therefore, if you want to cache directories outside /mnt/space, you must change the default mount directory to a particular home directory. As most of the containers used for CI/CD run under the root user, this will be the root directory. For example:

job("Build backend") {
    container(displayName = "Maven build", image = "maven") {
        // Change mount directory
        // This will overwrite all files in this dir in the image!
        mountDir = "/root"

        cache {
            storeKey = "maven-{{ hashFiles('pom.xml') }}"
            // now, we can reference files and dirs
            // inside /root/...
            localPath = "/root/.m2/repository"
        }

        shellScript {
            content = "mvn install"
        }
    }
}

Fallback cache

If Automation cannot find a {storeKey}.tar.gz archive, it looks for files specified in the restoreKeys list. We recommend creating such a fallback cache of project-wide dependencies whose versions change relatively rarely.

restoreKeys {
    +"npm-base-frontend"
    +"npm-base"
}

Overwrite cache files in the file repository

File repositories in Space have the Immutable files option. If it's enabled, the repository prohibits uploading a file with the same path more than once. If it's disabled, files can be overwritten. By default, this option is disabled for the default file repository, so files will be updated when changed.

Upload caches to a non-default repository

To upload caches to a repository different from the default one, use the cache.location parameter. Note that this repository must belong to the same project where you run the job.

// to upload cache files to 'my-file-repo'
cache {
    location = CacheLocation.FileRepository(name = "my-file-repo", remoteBasePath = CacheLocation.DefaultRemoteBasePath)
    storeKey = "npm-{{ dashify('run:git-checkout.ref') }}"
    restoreKeys {
        +"npm-refs-heads-master"
    }
    localPath = "node_modules"
}

To upload caches to a file repository created in another project, you should first attach this repository to your current project.

Disable cache uploads on file change

In many scenarios, re-uploading a changed file to the cache is unnecessary. For example, a JavaScript project may change files in the node_modules directory on each npm install, even though no dependencies were changed.

To prevent unnecessary cache uploads in such cases, use the cache.reuploadWhenFilesChange parameter. For example:

cache {
    // do not upload changed files to cache
    // only newly created files will be uploaded
    reuploadWhenFilesChange = false

    storeKey = "npm-{{ hashFiles('package.json') }}"
    localPath = "node_modules"
}

View project cache files

Open the desired project.
On the sidebar menu, choose Packages.
Open the file repository that contains project caches. In case of the default file repository, that would be default-automation-caches.

Last modified: 15 December 2023