Skip to content

Instantly share code, notes, and snippets.

@martin-braun
Last active March 10, 2024 14:03
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save martin-braun/e10b7cafdd55feea44bc2d06d5c2c972 to your computer and use it in GitHub Desktop.
Save martin-braun/e10b7cafdd55feea44bc2d06d5c2c972 to your computer and use it in GitHub Desktop.
vs - Git Vendor Sources - A POSIX compliant Unix tool to manage third-party git sources

Git Vendor Sources

Usage

Usage: vs [COMMAND] [REPO] [OPTIONS]
Adds, updates or deletes git source files of third party vendors in the local source storage at $HOME/.sources/.vendor.
This tool is meant to clone third party sources to compile or use without further modification.
It will try to checkout the latest tag and fallback to the latest branch, unless a prefered branch/tag is given.
If you want to load a workspace, consider forking and cloning using the normal git utility instead.

COMMAND
  ls                              Lists all locally added repositories/branches.
  dir                             Outputs the directory of the given repository/branch.
  sh                              Switches to the given repository/branch in a new shell.
  add                             Clones the given repository/branch with all its submodules with depth of 1 into the localized source storage and outputs its path.
  up                              Resets and pulls the latest changes on the given repository/branch. Will run ../.hooks/up.pre and ../.hooks/up.post.
  up-all                          Resets and pulls the latest changes on all repositories/branches. Will run ../.hooks/up.pre and ../.hooks/up.post.
  del                             Deletes the given repository/branch from the system. Will run ../.hooks/del.pre and ../.hooks/del.post.

REPO
  Unused on the ls and up-all command.
  For the add-command, needs to be a HTTPS URL or SSH destination ending with .git.
  For any other command it can also be just the sub path (without branch) on the file system.

OPTIONS
  -V,   --version                 Prints the version of this script.
  -v,   --verbose                 Prints verbose information.
  -p=*, --path=*                  Sets the working directory path (root) of the source storage (default: $path).
  -b=*, --branch=*                Sets the branch/tag of the repo to work with. Tags lock to a specific version. The 'ls' command will accept -b=/ to inline branches into paths.
  -c=*, --cmd=*                   Automatically runs the given command expression instead of launching an interactive shell when using the 'sh' command.
  -f,   --force                   Forces the operation that will potentially result in destructive behavior to ensure a successful operation.
  -h,   --help                    Prints this help message.

Installation

The following commands will download the script and use it to install itself. It will also link the binary accordingly:

mkdir -p "$HOME/.local/bin"
ln -s "$(curl --proto '=https' --tlsv1.2 -sSf "https://gist.githubusercontent.com/martin-braun/e10b7cafdd55feea44bc2d06d5c2c972/raw/vs" | sh -s -- add "https://gist.github.com/e10b7cafdd55feea44bc2d06d5c2c972.git")/vs" "/usr/local/bin/vs"

Please make sure that $HOME/.local/bin is part of your PATH environment variable.

You can also install vs globally as root:

su
VS_PATH="/var/opt"
ln -s "$(curl --proto '=https' --tlsv1.2 -sSf "https://gist.githubusercontent.com/martin-braun/e10b7cafdd55feea44bc2d06d5c2c972/raw/vs" | sh -s -- add "https://gist.github.com/e10b7cafdd55feea44bc2d06d5c2c972.git" --path="$VS_PATH")/vs" "/usr/local/bin/vs"

Then you should alias vs on all users (i.e. in .bashrc) to point to use the global repository instead:

export VS_PATH="/var/opt"
alias vs='/usr/local/bin/vs --path="$VS_PATH"'

To pick a different path regardless of the alias, use \vs -p="/your/custom/path".

To confirm a successful installation, simply run vs ls. It should list this gist. By installing this tool with itself, it will also auto update itself when running vs up-all. If you ever install repositories to a custom path, you need to \vs -p="/your/custom/path" up-all as well.

NOTE: It's generally not adviced to incorparate this into server deployment because of potentially changing APIs. I recommend to look into ansible for that.

# Created by https://www.toptal.com/developers/gitignore/api/windows,macos,linux,vim
# Edit at https://www.toptal.com/developers/gitignore?templates=windows,macos,linux,vim
### Linux ###
*~
# temporary files which can be created if a process still has a handle open of a deleted file
.fuse_hidden*
# KDE directory preferences
.directory
# Linux trash folder which might appear on any partition or disk
.Trash-*
# .nfs files are created when an open file is removed but is still being accessed
.nfs*
### macOS ###
# General
.DS_Store
.AppleDouble
.LSOverride
# Icon must end with two \r
Icon
# Thumbnails
._*
# Files that might appear in the root of a volume
.DocumentRevisions-V100
.fseventsd
.Spotlight-V100
.TemporaryItems
.Trashes
.VolumeIcon.icns
.com.apple.timemachine.donotpresent
# Directories potentially created on remote AFP share
.AppleDB
.AppleDesktop
Network Trash Folder
Temporary Items
.apdisk
### macOS Patch ###
# iCloud generated files
*.icloud
### Vim ###
# Swap
[._]*.s[a-v][a-z]
!*.svg # comment out if you don't need vector files
[._]*.sw[a-p]
[._]s[a-rt-v][a-z]
[._]ss[a-gi-z]
[._]sw[a-p]
# Session
Session.vim
Sessionx.vim
# Temporary
.netrwhist
# Auto-generated tag files
tags
# Persistent undo
[._]*.un~
### Windows ###
# Windows thumbnail cache files
Thumbs.db
Thumbs.db:encryptable
ehthumbs.db
ehthumbs_vista.db
# Dump file
*.stackdump
# Folder config file
[Dd]esktop.ini
# Recycle Bin used on file shares
$RECYCLE.BIN/
# Windows Installer files
*.cab
*.msi
*.msix
*.msm
*.msp
# Windows shortcuts
*.lnk
# End of https://www.toptal.com/developers/gitignore/api/windows,macos,linux,vim
#!/usr/bin/env sh
# shellcheck disable=SC2065
version="0.0.15"
path="$HOME/.sources/.vendor"
usage="$(cat << EOF
Usage: $(basename "$0") [COMMAND] [REPO] [OPTIONS]
Adds, updates or deletes git source files of third party vendors in the local source storage at $path.
This tool is meant to clone third party sources to compile or use without further modification.
It will try to checkout the latest tag and fallback to the latest branch, unless a prefered branch/tag is given.
If you want to load a workspace, consider forking and cloning using the normal git utility instead.
COMMAND
ls Lists all locally added repositories/branches.
dir Outputs the directory of the given repository/branch.
sh Switches to the given repository/branch in a new shell.
add Clones the given repository/branch with all its submodules with depth of 1 into the localized source storage and outputs its path.
up Resets and pulls the latest changes on the given repository/branch. Will run ../.hooks/up.pre and ../.hooks/up.post.
up-all Resets and pulls the latest changes on all repositories/branches. Will run ../.hooks/up.pre and ../.hooks/up.post.
del Deletes the given repository/branch from the system. Will run ../.hooks/del.pre and ../.hooks/del.post.
REPO
Unused on the ls and up-all command.
For the add-command, needs to be a HTTPS URL or SSH destination ending with .git.
For any other command it can also be just the sub path (without branch) on the file system.
OPTIONS
-V, --version Prints the version of this script.
-v, --verbose Prints verbose information.
-p=*, --path=* Sets the working directory path (root) of the source storage (default: $path).
-b=*, --branch=* Sets the branch/tag of the repo to work with. Tags lock to a specific version. The 'ls' command will accept -b=/ to inline branches into paths.
-c=*, --cmd=* Automatically runs the given command expression instead of launching an interactive shell when using the 'sh' command.
-f, --force Forces the operation that will potentially result in destructive behavior to ensure a successful operation.
-h, --help Prints this help message.
EOF
)"
while [ "$#" -gt 0 ]; do
case "$1" in
-V|--version) echo "$version" && exit 0 ;;
-v|--verbose) verbose=1 ;;
-p=*|--path=*) path="${1#*=}" ;;
-b=*|--branch=*) branch="${1#*=}" ;;
-c=*|--cmd=*) cmd="${1#*=}" ;;
-f|--force) force=1 ;;
-h|--help) echo "$usage" >&2 && exit 129 ;;
*)
if [ "$command" = "" ]; then
command="$1"
elif [ "$repo" = "" ]; then
repo="$1"
else
echo "Invalid command line flag $1." >&2;
exit 1
fi
;;
esac
shift
done
test -n "$verbose" && set -x
set -e
command -v "git" >/dev/null 2>&1 || { echo Missing git. >&2; exit 1; }
test ! "$command" = "" || { echo Missing command. >&2; exit 1; }
linksubpath="share"
if [ ! -d "$path/$linksubpath" ]; then
mkdir -p "$path"
chown -R "$(whoami)" "$path"
mkdir -p "$path/$linksubpath"
fi
if [ "$command" = "ls" ]; then
if [ "$branch" = "/" ]; then
find "$path/$linksubpath" -maxdepth 1 -type l -exec basename {} \; | sed 's/,,/\//' | tr , /
find "$path/$linksubpath" -maxdepth 1 -type f -name '*.lnk' -exec basename {} \; | sed 's/^\.\///' | sed 's/,,/\//' | sed 's/\.lnk$//' | tr , / # lnk support for network sync to msys2
else
find "$path/$linksubpath" -maxdepth 1 -type l -exec basename {} \; | sed 's/,,/ -b=/' | tr , /
find "$path/$linksubpath" -maxdepth 1 -type f -name '*.lnk' -exec basename {} \; | sed 's/^\.\///' | sed 's/,,/ -b=/' | sed 's/\.lnk$//' | tr , / # lnk support for network sync to msys2
fi
exit
elif [ "$command" = "up-all" ]; then
"$0" -p="$path" ls | xargs -L1 "$0" -p="$path" up
exit
fi
test -n "$repo" || { echo Missing repo. >&2; exit 1; }
test "$repo" = "${repo% *}" || { echo Cannot have spaces in repo. >&2; exit 1; }
reposubpath="$repo"
reposubpath="${reposubpath#*//}" # remove ^https://
reposubpath="${reposubpath#*@}" # remove ^git@
reposubpath="${reposubpath%.git}" # remove .git$
reposubpath=$(printf '%s' "$reposubpath" | tr : /) # replace : with /
test ! "$(printf '%s' "$reposubpath" | tr -d / | tr -d '.')" = "" || { echo Local reposubpath results in empty or invalid path. >&2; exit 1; }
if [ "$branch" = "" ]; then
reposubpathslashes=$(echo "$reposubpath" | tr -cd '/')
if [ "${#reposubpathslashes}" = "3" ]; then # path has branch/tag, so remove it from reposubpath and set it in branch
branch="${reposubpath##*/}"
reposubpath="${reposubpath%/*}"
fi
fi
if [ "$branch" = "" ]; then
link="$path/$linksubpath/$(printf '%s' "$reposubpath" | tr / ,)" # no branch/tag for rolling release
else
link="$path/$linksubpath/$(printf "%s,,%s" "$reposubpath" "$branch" | tr / ,)"
fi
if [ "$command" = "dir" ]; then
if [ -f "$link.lnk" ]; then
# Ugly msys2 work around for VMs with network storage, i.e. Parallels turns symlinks into lnk files.
# Just parse the name, but the branch needs to derived from the default branch, if it has not been supplied.
# This will break things if the default branch changes, which will happen for tagged releases.
# TODO: Probably also create a database file for each repo, which points to the right branch.
if [ "$branch" = "" ]; then
branch="$(git ls-remote --tags --sort="v:refname" "https://$reposubpath.git" | tail -n1 | awk -F '[/\\^]' '{print $3}')" # get latest tag
test ! "$branch" = "" || branch="$(git ls-remote --symref "https://$reposubpath.git" HEAD | awk '/^ref:/ {sub(/refs\/heads\//, "", $2); print $2}')" # or get default branch
test ! "$branch" = "" || { echo "Repository $repo doesn't exist or it has no tags or no default branch." >&2; exit 3; }
fi
echo "$path/$(basename "$link.lnk" | sed 's/^\.\///' | sed 's/,,/ -b=/' | sed 's/\.lnk$//' | tr , /)/$branch" && exit; # lnk support for network sync to msys2
fi
test -h "$link" || { echo /dev/null; echo "Repository $repo doesn't exist on the local system, because $link doesn't exist." >&2; exit 2; }
readlink -f "$link"
exit
elif [ "$command" = "sh" ]; then
( cd "$("$0" -p="$path" dir "$repo")"; { test "$cmd" = "" && "$SHELL" -i; } || "$SHELL" -c "$cmd" )
exit
fi
if [ "$branch" = "" ]; then
branch="$(git ls-remote --tags --sort="v:refname" "https://$reposubpath.git" | tail -n1 | awk -F '[/\\^]' '{print $3}')" # get latest tag
test ! "$branch" = "" || branch="$(git ls-remote --symref "https://$reposubpath.git" HEAD | awk '/^ref:/ {sub(/refs\/heads\//, "", $2); print $2}')" # or get default branch
test ! "$branch" = "" || branch="master"
echo "Autopicked branch $branch" >&2
fi
fullrepopath="$path/$reposubpath/$branch"
echo "Full path to repository: $fullrepopath" >&2
fullhookspath="$fullrepopath/../.hooks"
echo "Full path to its hooks: $fullhookspath" >&2
if [ "$command" = "add" ]; then
if [ -n "$force" ]; then
rm -rf "$fullrepopath" || true
rm -rf "$link" || true
fi
test ! -d "$link" || { echo "Repository $repo already added to the local system, consider updating it using the up-command instead or add the -f flag to forcefully re-install." >&2; exit 2; }
echo "Adding $repo ($branch) to $fullrepopath ..." >&2
if [ ! -d "$fullrepopath" ]; then
mkdir -p "$fullrepopath" >&2
echo "Cloning $repo ($branch) to $fullrepopath ..." >&2
git clone --recursive --depth 1 --branch "$branch" "$repo" "$fullrepopath" >&2
else
echo "Repository $repo already found, relinking..." >&2
fi
ln -sf "../$reposubpath/$branch" "$link" >&2
echo "Linked $fullrepopath to $link" >&2
test -h "$link" || { echo /dev/null; echo "Repository $repo doesn't exist on the local system, because $link doesn't exist." >&2; exit 2; }
readlink -f "$link"
mkdir -p "$fullhookspath" # makes it easier to find by the user
elif [ "$command" = "up" ]; then
test -d "$link" || { echo "Repository $repo doesn't exist on the local system." >&2; exit 2; }
echo Updating "$fullrepopath" ... >&2
if [ ! -d "$fullrepopath" ]; then # link exists but determined fullrepopath does not? This can happen when the latest tag or default branch changed on non-locked packages.
test -x "$fullhookspath/up.pre" && sh -c "$fullhookspath/up.pre"
oldlinkdest="$(readlink "$link")"
rm "$link"
if find "$path/$linksubpath" -maxdepth 1 -not -name '.' -exec readlink {} \; | grep -q "$oldlinkdest\$"; then # no other source links to the old version?
rm -rf "${path:?}/$linksubpath/$oldlinkdest" || echo Warning: Failed to remove old version. >&2
fi
mkdir -p "$fullrepopath"
git clone --recursive --depth 1 --branch "$branch" "$repo" "$fullrepopath"
ln -sf "../$reposubpath/$branch" "$link"
test -x "$fullhookspath/up.post" && sh -c "$fullhookspath/up.post"
else
if [ -n "$(git -C "$fullrepopath" diff HEAD "origin/$branch" 2>/dev/null || exit 0)" ]; then # exits on tags, because it cannot find the branch. Tags are locked anyways.
test -x "$fullhookspath/up.pre" && sh -c "$fullhookspath/up.pre"
if ! git -C "$fullrepopath" pull --rebase; then
if [ "$force" ]; then
git -C "$fullrepopath" reset --hard
git -C "$fullrepopath" pull --ff-only
else
git rebase --abort
echo Update failed, because incoming changes cannot be merged with local changes. Use the -f flag to force a reset of local changes or manually rebase and solve the conflicts. >&2
exit 3
fi
fi
test -x "$fullhookspath/up.post" && sh -c "$fullhookspath/up.post"
fi
fi
elif [ "$command" = "del" ]; then
test -d "$link" || test -n "$force" || { echo "Repository $repo doesn't exist on the local system. Use the -f flag to forcefully try delete and clean up anyways." >&2; exit 2; }
test -d "$link" && fullrepopath="$path$(readlink "$link" | sed 's/^..//')" # get real fullrepopath in case of older rolling
echo "Deleting $fullrepopath ..." >&2
test -x "$fullhookspath/del.pre" && sh -c "$fullhookspath/del.pre"
rm -rf "$fullrepopath" || test -n "$force"
echo Cleaning up ... >&2
rm -rf "$link" || test -n "$force"
rmdir "$path/$linksubpath" 2>/dev/null || true
find "$path/" -depth -mindepth 1 -type d -exec rmdir {} + 2>/dev/null || true
test -x "$fullhookspath/del.post" && sh -c "$fullhookspath/del.post"
else
echo "Unknown command $command." >&2
exit 1
fi
set +e && set +x
_vs()
{
COMPREPLY=()
cur="${COMP_WORDS[COMP_CWORD]}"
prev="${COMP_WORDS[COMP_CWORD-1]}"
if [[ ${cur} == --* ]] ; then
COMPREPLY=( $(compgen -W "--version --verbose --path= --branch= --cmd= --force --help" -- ${cur}) )
return 0
fi
if [[ ${cur} == -* ]] ; then
COMPREPLY=( $(compgen -W "-V -v -p= -b= -c= -f -h" -- ${cur}) )
return 0
fi
if [[ ${#COMP_WORDS[@]} == 2 ]] ; then
COMPREPLY=( $(compgen -W "ls dir sh add up up-all del" -- ${cur}) )
return 0
fi
if [[ ${prev} == "dir" || ${prev} == "sh" || ${prev} == "up" || ${prev} == "del" ]] ; then
COMPREPLY=( $(compgen -W "$(vs ls -b=/ | xargs echo)" -- ${cur}) )
return 0
fi
}
complete -o nospace -F _vs vs
# vi: ft=bash
@martin-braun
Copy link
Author

martin-braun commented Jan 18, 2024

// reserved

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment