Skip to content

Instantly share code, notes, and snippets.

@tomdavidson

tomdavidson/README.md

Last active Jun 3, 2020
Embed
What would you like to do?
mono repo tool to id dirs with files that have changed

lolaus

A simple monorepo lifecycle/pipeline tool for running one or more commands on one or more directories that have diffs compared to an ancestor. The primary use case is selective CI jobs within a trunk based workflow such Git Flow.

Usage

lolaus has for optional positional arguments: lolaus [glob] [command] [target_ref] [source_ref]

  • glob defaults to **

  • command does not default

  • target_ref defaults to next if available and then master

  • source_ref defaults to the current HEAD

Examples:

  lolaus | parallel "wc -l {} > {}.out"
  lolaus "./tests/* :(top,exclude)**requirements.txt" ls
  lolaus "*/*/package.json" "npm test" release_branch
docker_images:
  stage: build
  script:
    - lolaus "services/*/Dockerfile" "docker build -t ${PWD##*/} ."

py_tests:
  image: python
  stage: test
  script:
    - lolaus "apps/*/requirements.txt" "python3 tests.py"

Install

lolaus is a Bash script and should be fairly universal but not tested outside of GNU/Linux. OSX users will have a better shell experience (and perhaps life in general) if they install GNU utilities. Contribs increasing portability or with other packaging (pip|curl|gem|other) would be great!

npm install -g lolaus

Design

lolaus is just a convenience shell wrapper of host's git client. The targeted integration branch is used with git merge-base --fork-point to determine the common ancestor node (CA). This is one of several common ancestry node strategies and others could be supported in lolaus, but this seems to provide the most benefit to the pre-integration, monorepo use case.

git diff from the current git ref (Fh) to the CA to determine which files have changes, filtered by the provided glob. From the list of modified files the command is invoked from a sorted and unique list of directories sequentially via eval in a loop.

graph LR
  m1(("Sc"))
  ca(("CA"))
  mh(("Mh"))
  f1(("Fc"))
  f2(("Fc"))
  fh(("Fh"))

  m1---ca
  ca-- master ---mh
  ca-- feature ---f1
  ca-. git diff .-fh
  f1---f2
  f2---fh

  classDef gitNode fill:#4ED1A1,stroke:#555,stroke-width:4px;
  class m1,m2,mh,f1,f2,fh,ca gitNode

  classDef gitDiff fill:#4E63D1,stroke:#555,stroke-width:4px;
  class fh,ca gitDiff

The design also assumes that each microservice or component is contained in its own directory and can be identified by some sort of file such as a package.json, pom.xml, or requirements.txt via the glob.

There is no parallelism, lolaus is intended to be used pre-integration where only one or two components that are effected in the merge request. In cases where target_ref and source_ref defaults work, the command parameter can be omitted and the list of effected directors can be piped to parallel.

#!/usr/bin/env bash
help() {
cat <<EOF
Simple monorepo lifecycle/pipeline tool for running one or more commands on one
or more directories that have diffs compared to an ansector. The primary
use case is for selective CI jobs within a trunk based workflow.
Takes two arguments, <glob> <command>. The command is invoked from each
directory context matching the glob.
Usage:
lolaus "./tests/* :(top,exclude)**requirements.txt" ls
lolaus "**" pwd
lolaus "*/*/package.json" npm test & lolaus "*/*/requirements.txt" "python test.py"
lolaus "apps/*/index.js"
lolaus "**" ls target-branch other-branch
EOF
exit 1
}
# lolaus <cmd> [glob] [target_ref] [source_ref]
function main() {
[[ $1 == "-h" || $1 == "--help" ]] && help
local GLOB
GLOB=${1:-'**'}
local CMD
CMD="$2"
local TARGET_REF
TARGET_REF=$(get-target-ref "$3")
[[ ${DEBUG} =~ ^(true|TRUE) ]] && echo target: $TARGET_REF
local SOURCE_REF
SOURCE_REF=$(get-source-ref "$4")
[[ ${DEBUG} =~ ^(true|TRUE) ]] && echo source: $SOURCE_REF
local CONCESTOR
CONCESTOR=$(get-concestor $TARGET_REF $SOURCE_REF)
[[ ${DEBUG} =~ ^(true|TRUE) ]] && echo concestor: $CONCESTOR
local DIFFS
DIFFS=$(get-diffs $CONCESTOR $SOURCE_REF)
[[ ${DEBUG} =~ ^(true|TRUE) ]] && echo diffs: $DIFFS
local DIRS
DIRS=$(files-to-dirs "$DIFFS")
[[ ${DEBUG} =~ ^(true|TRUE) ]] && echo dirs: $DIRS
[[ x"$2" == "x" ]] && echo "$DIRS" || cmd-runner "$CMD" "$DIRS"
}
# Invokes the provided command in each provided direcotry
# cmd-runner <cmd> <dirs>
cmd-runner() {
: ${1? "Please provide comands to run."}
: ${2? "Please provide directories to run in."}
local cmd
cmd="$1"
declare -a local dirs
dirs=($2)
for d in "${dirs[@]}"; do
[[ ! -d $d ]] && echo "cmd-runner only works on dirs, not: $d" && exit 1
local cmds="cd $d; $cmd"
output="$(eval "$cmds" 2>&1)"
[[ x"$output" == "x" ]] && output="$cmd produced no output"
[[ $d == "$PWD" ]] && d='./'
relative=${d/$(echo "$PWD/")/}
echo -e "$(tput smso) $relative $(tput sgr0)\n$output\n"
done
}
# Returns list of sorted and unique containing directories from a list of files.
# files-to-dirs <files>
files-to-dirs() {
declare -a local files
files=($1)
local path
path=$(pwd)
declare -a local dirs
declare -a local sorted
for file in "${files[@]}"; do
if [ -n "${file##*/*}" ]; then
dirs+=("$path")
else
dirs+=("$path/${file%/*}")
fi
done
IFS=$'\n' sorted=($(sort -u <<<"${dirs[*]}"))
unset IFS
echo "${sorted[*]}"
}
# Determines the closest common ansester from two git refs with a default
# strategy of 'fork-point'
# get-concestor <LEFT_REF> <RIGHT_REF> [STRATEGY]
get-concestor() {
local left_ref=${1}
local right_ref=${2}
local strategy=${3:-'fp'}
local concestor
case ${strategy} in
fp) concestor=$(git merge-base --fork-point $left_ref $right_ref) ;;
esac
echo $concestor
}
# Returns a list of files with diffs between two git refs with optional glob.
# get-diffs <left_ref> <right_ref> [glob]
get-diffs() {
local left_ref=$1
local right_ref=$2
local glob=${3:-"**"}
local diffs
diffs=$(git diff --name-only $left_ref..$right_ref -- $glob)
echo $diffs
}
# Returns the verified git reference to target. Defaults to current ref.
# get-source-ref [branch name]
get-source-ref() {
local ref=$1
if [ ! -z "$ref" ]; then
if ! git rev-parse --quiet --verify $ref &>/dev/null; then
echo "Source git ref, $ref, is not valid"
exit 1
fi
else
ref=$(git symbolic-ref --short HEAD)
fi
echo $ref
}
# Returns the verified git reference to target. Default to 'next' then 'master'
# get-target-ref [branch name]
get-target-ref() {
local REF=$1
if [ ! -z "$ref" ]; then
if ! git rev-parse --quiet --verify $ref &>/dev/null; then
echo "Targeted git ref, $ref, is not valid"
return 1
fi
else
if git rev-parse --quiet --verify next &>/dev/null; then
ref='next'
elif git rev-parse --quiet --verify master &>/dev/null; then
ref='master'
else
return 1
fi
fi
echo $ref
}
main "$@"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.