Skip to content

Instantly share code, notes, and snippets.

@belgattitude
Last active April 30, 2024 05:58
Show Gist options
  • Star 26 You must be signed in to star a gist
  • Fork 4 You must be signed in to fork a gist
  • Save belgattitude/838b2eba30c324f1f0033a797bab2e31 to your computer and use it in GitHub Desktop.
Save belgattitude/838b2eba30c324f1f0033a797bab2e31 to your computer and use it in GitHub Desktop.
Composite github action to improve CI time with pnpm

Why

Although @setup/node as a built-in cache option, it lacks an opportunity regarding cache persistence. Depending on usage, the action below might give you faster installs and potentially reduce carbon emissions (♻️🌳❤️).

Requirements

pnpm v7 or v8 (not using pnpm ? see the corresponding yarn action gist)

Bench

Based on the nextjs-monorepo-example with pnpm.

A cold cache install on the ci is around ±1m20s.

With warmed cache: ±40s + (add ±10s for compression). Crafted from benchmarks results in https://gist.github.com/belgattitude/0ecd26155b47e7be1be6163ecfbb0f0b. Depending on repo (renovatebot...), the slight complexity increase in ci setup might worth it.

Structure

.
└── .github
    ├── actions
    │   └── pnpm-install/action.yml (composite action)    
    └── workflows
        └── ci.yml (uses: ./.github/actions/pnpm-install)    

Composite action

Create a file in .github/actions/pnpm-install/action.yml and paste

########################################################################################
# "pnpm install" composite action for pnpm 7/8+                                        #
#--------------------------------------------------------------------------------------#
# Requirement: @setup/node should be run before                                        #
#                                                                                      #
# Usage in workflows steps:                                                            #
#                                                                                      #
#      - name: 📥 Monorepo install                                                     #
#        uses: ./.github/actions/pnpm-install                                          #
#        with:                                                                         #
#          enable-corepack: false # (default)                                          #
#          cwd: ${{ github.workspace }}/apps/my-app # (default = '.')                  #
#                                                                                      #
# Reference:                                                                           #
#   - latest: https://gist.github.com/belgattitude/838b2eba30c324f1f0033a797bab2e31    #
#                                                                                      #
# Versions:                                                                            #
#   - 1.1.0 - 15-07-2023 - Add project custom directory support.                       #
########################################################################################


name: 'PNPM install'
description: 'Run pnpm install with cache enabled'

inputs:
  enable-corepack:
    description: 'Enable corepack'
    required: false
    default: 'false'
  cwd:
    description: "Changes node's process.cwd() if the project is not located on the root. Default to process.cwd()"
    required: false
    default: '.'

runs:
  using: 'composite'

  steps:
    - name: ⚙️ Enable Corepack
      if: ${{ inputs.enable-corepack == 'true' }}
      shell: bash
      working-directory: ${{ inputs.cwd }}
      run: |
        corepack enable
        echo "corepack enabled" 

    - uses: pnpm/action-setup@v2.2.4
      if: ${{ inputs.enable-corepack == 'false' }}
      with:
        run_install: false
        # If you're not setting the packageManager field in package.json, add the version here
        # version: 8.6.7

    - name: Expose pnpm config(s) through "$GITHUB_OUTPUT"
      id: pnpm-config
      shell: bash
      run: |
        echo "STORE_PATH=$(pnpm store path)" >> $GITHUB_OUTPUT

    - name: Cache rotation keys
      id: cache-rotation
      shell: bash
      run: |
        echo "YEAR_MONTH=$(/bin/date -u "+%Y%m")" >> $GITHUB_OUTPUT

    - uses: actions/cache@v3
      name: Setup pnpm cache
      with:
        path: ${{ steps.pnpm-config.outputs.STORE_PATH }}
        key: ${{ runner.os }}-pnpm-store-cache-${{ steps.cache-rotation.outputs.YEAR_MONTH }}-${{ hashFiles('**/pnpm-lock.yaml') }}
        restore-keys: |
          ${{ runner.os }}-pnpm-store-cache-${{ steps.cache-rotation.outputs.YEAR_MONTH }}-

    # Prevent store to grow over time (not needed with yarn)
    # Note: not perfect as it prune too much in monorepos so the idea
    #       is to use cache-rotation as above. In the future this might work better.
    #- name: Prune pnpm store
    #  shell: bash
    #  run: pnpm prune store

    - name: Install dependencies
      shell: bash
      working-directory: ${{ inputs.cwd }}
      run: pnpm install --frozen-lockfile --prefer-offline
      env:
        # Other environment variables
        HUSKY: '0' # By default do not run HUSKY install

Workflow action

To use it in the workflows

    steps:
      - uses: actions/checkout@v3

      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v3
        with:
          node-version: ${{ matrix.node-version }}

      - name: 📥 Monorepo install
        uses: ./.github/actions/pnpm-install

Recommended .npmrc

Ensure you're running a recent pnpm version (or adapt)

## PNPM related ###############
## https://pnpm.io/npmrc      #
###############################

# Not always possible to be strict, but if it works for you, keep it to true.
# https://pnpm.io/next/npmrc#strict-peer-dependencies
strict-peer-dependencies=false

# Auto install peers should be false to avoid downloading
# extraneous deps. If the install fails, try first to explicitly add
# the missing deps in your package. Set it to true at last resort
# (when the problem comes from upstream dependencies). The best is false.
# https://pnpm.io/npmrc#auto-install-peers
auto-install-peers=false

# Helps locating duplicates, default in v8
# https://pnpm.io/next/npmrc#use-lockfile-v6
use-lockfile-v6=true

# Will fix duplicates due to peer-dependencies (>=7.29.0), default in v8
# https://github.com/pnpm/pnpm/releases/tag/v7.29.0
dedupe-peer-dependents=true

# Helps with peer-deps (>=7.23.0), default in v8
# https://pnpm.io/npmrc#resolve-peers-from-workspace-root
resolve-peers-from-workspace-root=true

# default to 'lowest' in v8.5.0
# set to highest for reasons specified here: https://github.com/pnpm/pnpm/issues/6463
# https://pnpm.io/npmrc#resolution-mode
resolution-mode=highest

# Default in 8.1.0 to fix issues with root/workspaces hoisting
# https://pnpm.io/npmrc#dedupe-direct-deps
dedupe-direct-deps=false

# Pinlock to exact version (default is '^')
# https://pnpm.io/npmrc#save-prefix
# see also how save-workspace-protocol affect this https://pnpm.io/npmrc#save-workspace-protocol
save-prefix=''

# Most of the time, you want to use the rolling protocol for monorepos
# https://pnpm.io/npmrc#save-workspace-protocol
save-workspace-protocol=rolling

Notes

Install

image

Post-install

image

Cleanup caches

When a PR is closed or merged the best is to remove install cache rather than letting github reach the max (10GB) and prune.

image

Link: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#force-deleting-cache-entries

Here's an example (feel free to adapt if you need to preserse some things, ie gh actions-cache list -R $REPO -B $BRANCH | cut -f 1 | grep pnpm will only clear pnpm related caches)

# https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#force-deleting-cache-entries
name: Cleanup caches for closed branches

on:
  pull_request:
    types:
      - closed
  workflow_dispatch:

jobs:
  cleanup:
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
        uses: actions/checkout@v3

      - name: Cleanup
        run: |
          gh extension install actions/gh-actions-cache

          REPO=${{ github.repository }}
          BRANCH="refs/pull/${{ github.event.pull_request.number }}/merge"

          echo "Fetching list of cache key"
          cacheKeysForPR=$(gh actions-cache list -R $REPO -B $BRANCH | cut -f 1 )

          ## Setting this to not fail the workflow while deleting cache keys. 
          set +e
          echo "Deleting caches..."
          for cacheKey in $cacheKeysForPR
          do
              gh actions-cache delete $cacheKey -R $REPO -B $BRANCH --confirm
          done
          echo "Done"
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@brainsaysno
Copy link

Worked great, thank you so much! ❤️

@belgattitude
Copy link
Author

Happy to hear ❤️

@theoephraim
Copy link

adding echo "STORE_PATH=$(pnpm store path | tr -d '\n')" >> $GITHUB_OUTPUT should fix some issues with pnpm store path having an extra line break

@navidemad
Copy link

https://github.com/actions/setup-node/blob/main/docs/advanced-usage.md#caching-packages-data
actions/setup-node#530 (comment)

pnpm should be preinstalled before the setup-node action

We should change it to that right ?

- uses: actions/checkout@v3
- uses: pnpm/action-setup@v2
  with:
    version: 6.32.9
- uses: actions/setup-node@v3
  with:
    node-version: '14'
    cache: 'pnpm'
- run: pnpm install --frozen-lockfile
- run: pnpm test

@belgattitude
Copy link
Author

belgattitude commented May 22, 2023

thanks @navidemad I've updated the comment and added corepack input. Note that it was there: https://gist.github.com/belgattitude/838b2eba30c324f1f0033a797bab2e31#workflow-action

@sinisastanic
Copy link

if: ${{ inputs.enable-corepack }} == 'true' will never be evaluated correctly. See more.

So you should either do if: ${{ inputs.enable-corepack == 'true' }} or if: inputs.enable-corepack == 'true' .

Also inputs.cwd is being referenced in Enable corepack step, but cwd is not listed in the inputs block.

@lotyp
Copy link

lotyp commented Jul 15, 2023

inputs:
  cwd:
    description: "Changes node's process.cwd() if the project is not located on the root. Default to process.cwd()"
    required: false
    default: '.'

@belgattitude
Copy link
Author

belgattitude commented Jul 16, 2023

@lotyp Thanks, note that the cwd will only work with corepack (setup/pnpm does not allow to set cwd without running install)

@sinisastanic you're totally right, tested in belgattitude/pnpm-yarn-ci-what-to-cache#1 for comparison

Updated the gist 🙏

@sbue
Copy link

sbue commented Jul 19, 2023

This is great 🙏

I'm a bit new to Github Actions. Is there a way to run pnpm install as it's own job, and downstream jobs that also need pnpm install re-use the cache created by the first job?

Here's an example of what I'm thinking. My impression is that this won't work because these jobs run on independent machines. Given this - how can we leverage this pnpm caching such that in a workflow with many jobs, PNPM install is ideally only called once (wherein a cache exists)?

jobs:
  install-deps:
    name: Install Dependencies
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Install Node.js
        uses: actions/setup-node@v3
        with:
          node-version: 18

      - name: PNPM Install
        uses: ./.github/actions/pnpm-install

  build:
    name: Build
    needs: [install-deps]
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Run build
        uses: ./.github/actions/builds/web
        with:
          vercel-token: ${{ secrets.VERCEL_TOKEN }}
          turbo-token: ${{ secrets.TURBO_TOKEN }}
          turbo-team: ${{ vars.TURBO_TEAM }}
          environment: preview

@oliveryasuna
Copy link

Anyone know how to accomplish the cleanup with GitHub Script? In organizations, we don't want to use personal access tokens. I tried:

uses: actions/github-script@v6.4.1
with:
  script: |
    const ownerRepoRef = {
      owner: context.repo.owner,
      repo: context.repo.repo,
      ref: `refs/pull/${{ github.event.pull_request.number }}/merge`,
    };
    
    const caches = await github.rest.actions.getActionsCacheList(ownerRepoRef);
    
    for(const cache of caches.data.actions_caches) {
      const cacheKey = cache.key;
    
      console.log(`Deleting cache ${cacheKey}...`);
    
      await github.rest.actions.deleteActionsCacheByKey({
          ...ownerRepoRef,
          key: cacheKey,
      });
    }

Got an error:

RequestError [HttpError]: Resource not accessible by integration
    at /home/runner/work/_actions/actions/github-script/v6.4.1/dist/index.js:6842:21
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async eval (eval at callAsyncFunction (/home/runner/work/_actions/actions/github-script/v6.4.1/dist/index.js:15143:16), <anonymous>:9:16)
    at async main (/home/runner/work/_actions/actions/github-script/v6.4.1/dist/index.js:15236:20) {
  status: 403,
  response: {
    url: 'https://api.github.com/repos/panama-ui/panama/actions/caches?ref=refs%2Fpull%2F25%2Fmerge',
    status: 403,
    headers: {
      'access-control-allow-origin': '*',
      'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset',
      connection: 'close',
      'content-encoding': 'gzip',
      'content-security-policy': "default-src 'none'",
      'content-type': 'application/json; charset=utf-8',
      date: 'Sun, 30 Jul 2023 15:47:27 GMT',
      'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
      server: 'GitHub.com',
      'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
      'transfer-encoding': 'chunked',
      vary: 'Accept-Encoding, Accept, X-Requested-With',
      'x-content-type-options': 'nosniff',
      'x-frame-options': 'deny',
      'x-github-api-version-selected': '2022-11-28',
      'x-github-media-type': 'github.v3; format=json',
      'x-github-request-id': '8782:3ABA:3F38198:80DC772:64C6860F',
      'x-ratelimit-limit': '1000',
      'x-ratelimit-remaining': '944',
      'x-ratelimit-reset': '1690732978',
      'x-ratelimit-resource': 'core',
      'x-ratelimit-used': '56',
      'x-xss-protection': '0'
    },
    data: {
      message: 'Resource not accessible by integration',
      documentation_url: 'https://docs.github.com/rest/actions/cache#list-github-actions-caches-for-a-repository'
    }
  },
  request: {
    method: 'GET',
    url: 'https://api.github.com/repos/panama-ui/panama/actions/caches?ref=refs%2Fpull%2F25%2Fmerge',
    headers: {
      accept: 'application/vnd.github.v3+json',
      'user-agent': 'actions/github-script octokit-core.js/3.6.0 Node.js/16.16.0 (linux; x64)',
      authorization: 'token [REDACTED]'
    },
    request: { agent: [Agent], hook: [Function: bound bound register] }
  }
}
Error: Unhandled error: HttpError: Resource not accessible by integration

@sinisastanic
Copy link

@sbue If we assume you have need for 2 separate jobs that both need to install deps, the part of using

      - name: PNPM Install
        uses: ./.github/actions/pnpm-install

would do caching by itself, since this is inside it contains

    - uses: actions/cache@v3
      name: Setup pnpm cache
      with:
        path: ${{ steps.pnpm-config.outputs.STORE_PATH }}
        key: ${{ runner.os }}-pnpm-store-cache-${{ steps.cache-rotation.outputs.YEAR_MONTH }}-${{ hashFiles('**/pnpm-lock.yaml') }}
        restore-keys: |
          ${{ runner.os }}-pnpm-store-cache-${{ steps.cache-rotation.outputs.YEAR_MONTH }}-

If the example you provided is your starting point, in general, job which just installs something doesn't make much sense, since just like you said, they run in separate machines. Just use the ./.github/actions/pnpm-install when you need it:

jobs:
  build:
    name: Build
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Install Node.js
        uses: actions/setup-node@v3
        with:
          node-version: 18

      - name: PNPM Install
        uses: ./.github/actions/pnpm-install

      - name: Run build
        uses: ./.github/actions/builds/web
        with:
          vercel-token: ${{ secrets.VERCEL_TOKEN }}
          turbo-token: ${{ secrets.TURBO_TOKEN }}
          turbo-team: ${{ vars.TURBO_TEAM }}
          environment: preview

@oliveryasuna maybe this would help you.

@oliveryasuna
Copy link

oliveryasuna commented Jul 30, 2023

@sinisastanic Thank you, but the page uses the CLI. As far as I know, GitHub CLI does not support any organization-specific authentication (without SAML).

@matchai
Copy link

matchai commented Mar 21, 2024

# default to 'lowest' in v8.5.0
# set to highest for reasons specified here: https://github.com/pnpm/pnpm/issues/6463
# https://pnpm.io/npmrc#resolution-mode
resolution-mode=highest

A heads up that it now defaults to "highest" in v8.7.0: pnpm/pnpm#6987

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment