Monorepo sparse-checkout tricks

tricks.md

References

• https://git-scm.com/docs/git-sparse-checkout
• https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---reference-if-ableltrepositorygt
• https://git-scm.com/docs/gitrepository-layout#Documentation/gitrepository-layout.txt-objectsinfoalternates
• https://about.gitlab.com/blog/2020/03/13/partial-clone-for-massive-repositories/

Tricks

Sparse checkout using CODEOWNERS

git clone --reference-if-able /on/disk/cache/repo/.git \
          --filter=tree:0 \
          --single-branch \
          --no-checkout \
          https://git.store.com/group/repo.git my-repo

cd my-repo

git sparse-checkout init --cone

git checkout master

grep $USER CODEOWNERS |\
  grep -v '^#' |\ # Ignore comments
  awk '{print $1}' |\ # Only print file path
  git sparse-checkout set --stdin

Sparse checkout using on-disk manifest

git clone --reference-if-able /on/disk/cache/repo/.git \
          --filter=tree:0 \
          --single-branch \
          --sparse \
          https://git.store.com/group/repo.git my-repo
          
cd my-repo

git sparse-checkout set dir/to/manifests

cat dir/to/manifests/*.manifest |\
  git sparse-checkout add --stdin

Generate sparse checkout manifest with bazel

1. In a full-checkout copy, run bazel query

cd my-repo

bazel query --output=package 'buildfiles(deps(//projects/my-awsome-service/...))' |\
  awk '!/^@/ {print "/"$0}' \ # Ignore external deps, prepend all paths with '/'
  > dir/to/manifests/my-awsome-service/baze-deps.manifest
  
# Upload the file to some place
curl -XPOST dir/to/manifests/my-awsome-service/baze-deps.manifest

# Or add it into the git repo
git add dir/to/manifests/my-awsome-service/baze-deps.manifest
git commit -m 'manifest update' -m 'Update details: blah blah'
git push

2. Checkout using bazel manifest

git clone --reference-if-able /on/disk/cache/repo/.git \
          --filter=tree:0 \
          --single-branch \
          --sparse \
          https://git.store.com/group/repo.git my-repo
          
cd my-repo

curl -s 'https://some-http-app/service/name/bazel-deps.manifest' |\
  git sparse-checkout set --stdin

Create and maintain on disk cache of repo

git clone --mirror https://git.store.com/group/repo.git /on/disk/cache/repo/.git

while true; do
  git -C /on/disk/cache/repo/.git fetch --prune --prune-tags origin;
  git -C /on/disk/cache/repo/.git gc --auto;
done

Using host local git cache in a container

# On baremetal/VM host
docker run --rm -it -v /on/disk/cache:/on/disk/cache:ro alpine /bin/bash

# Inside container
git clone --reference-if-able /on/disk/cache/repo/.git \
          --filter=tree:0 \
          --single-branch \
          --sparse \
          https://git.store.com/group/repo.git my-repo

@malomarrec FYI this is what we talked about today.

Thanks!