Example how to compute diffs of more than 300 files between branches/commits/tags server side using a pre-receive hook in GitHub Enterprise

diff-branches-client.sh

#!/usr/bin/env bash

# Client side script to trigger pre-receive hook and scan its output
# usage: diff-branches-client.sh <repo URL> <compare branch> [base branch] [magic trigger branch]
# Examples:
# diff-branches.client.sh https://jonico@octodemo.com/jonico/large-branch-diff-with-prereceive-hook.git compare-branch feature-branch # diffs compare-branch with feature branch
# diff-branches.client.sh https://jonico@octodemo.com/jonico/large-branch-diff-with-prereceive-hook.git compare-branch # diffs compare-branch with master
# diff-branches.client.sh https://jonico@octodemo.com/jonico/large-branch-diff-with-prereceive-hook.git f75b18abbb06 f75b18abbb06^1 # diffs commit f75b18abbb06 with its parent
# diff-branches.client.sh https://jonico@octodemo.com/jonico/large-branch-diff-with-prereceive-hook.git master master~2 # diffs latest commit on master with its grandparent

if [ "$#" -lt "2" ]; then
  echo "Usage: diff-branches-client.sh <repo URL> <compare branch> [base branch] [magic trigger branch]"
  exit 1
fi

WORK_DIR=`mktemp -d`

repo=$1
compareBranch=$2
baseBranch=${3:-master}
magicTrigger=${4:-diff-branches}

# check if tmp dir was created
if [[ ! "$WORK_DIR" || ! -d "$WORK_DIR" ]]; then
  echo "Could not create temp dir"
  exit 1
fi

# deletes the temp directory
function cleanup {
  rm -rf "$WORK_DIR"
}

# register the cleanup function to be called on the EXIT signal
trap cleanup EXIT

cd $WORK_DIR
git init --quiet
echo "Commit to trigger something" >> trigger.txt
git add trigger.txt
git commit --quiet -am "Trigger pre-receive hook"
git push -f "${repo}" "HEAD:${magicTrigger}" "--push-option=${compareBranch}" "--push-option=${baseBranch}"  2>rawOutput

startMarker="${magicTrigger}: Starting diff"
endMarker="${magicTrigger}: Finished diff"

grep -q "$endMarker" rawOutput

if [ $? -ne "0" ]; then
  echo Could not find magic marker.
  echo Printing output from git push "${repo}" "HEAD:${magicTrigger}" "--push-option=${compareBranch}" "--push-option=${baseBranch}":
  cat rawOutput
  exit 1
fi

# Extract everything between start and end marker and remove "remote: " - prefix
awkPattern='/remote: /{sub("remote: ","")'"}/$startMarker/"'{flag=1;next}'"/$endMarker/"'{flag=0}flag'

awk "$awkPattern" rawOutput

diff-branches.sh

#!/usr/bin/env bash

#
# Pre-receive hook that will show the output of the diff of two branches computed server side
#
# Most GitHub API calls are designed to limit their output to a number of objects for performance reasons.
# For instance, https://developer.github.com/enterprise/v3/pulls/#list-pull-requests-files or
# https://developer.github.com/v3/repos/commits/#get-a-single-commit
# are paged with a batch size of 30 and will not list more than 300 files.
#
# If GitHub Enterprise customers need to diff branches/commits/tags with more than 300 files changed and understand the impact this has on the load of their instance, this script shows how to do it.
# This script intercepts pushes to a magic branch, called diff-large-branch (can be changed) and will output the files changed between two (real) branches of the same repository
# The two branche/commits/tags to compare can be specified using --push-options (see usage example)
#
# If no base branch is specified, master is assumed
# What you actually push does not matter as it will be rejected anyways

#
# Usage: git push origin HEAD:diff-branches --push-option=<compareBranch> [--push-option=<baseBranch>]
#
# Examples:
# git push origin HEAD:diff-branches --push-option=compare-branch --push-option=feature-branch # compare compare-branch with feature-branch
# git push origin HEAD:diff-branches --push-option=compare-branch # compare compare-branch with master
# git push origin HEAD:diff-branches --push-option=master --push-option=master^1 # compare latest commit on master with its parent
# git push origin HEAD:diff-branches --push-option=f75b18abbb06 --push-option=f75b18abbb06^1 # compare commit f75b18abbb06 with its parent
#
# If you do not like the idea of pushing to a magic branch, you could also use a magic trigger as first --push-option
#
# More details on pre-receive hooks and how to apply them can be found on
# https://help.github.com/enterprise/admin/guides/developer-workflow/managing-pre-receive-hooks-on-the-github-enterprise-appliance/
#

# trigger branch to push to so that this hook gets active
trigger="diff-branches"

while read oldrev newrev refname; do

  if [ "$refname" = "refs/heads/$trigger" ]; then
    if [ "$GIT_PUSH_OPTION_COUNT" -lt "1" ]; then
      echo $trigger: "Usage: git push origin HEAD:$trigger --push-option=<compareBranch> [--push-option=<baseBranch>]"
      exit 1
    fi
    compareBranch=$GIT_PUSH_OPTION_0
    baseBranch=${GIT_PUSH_OPTION_1:-master}
    echo $trigger: "Starting diff from ${baseBranch} to ${compareBranch}"
    git diff --name-status "$baseBranch" "$compareBranch"
    echo $trigger: "Finished diff"
    exit 42
  else
    continue
  fi
done
exit 0

Example usage:

jonico at pulipan in ~/large-branch-diff-with-prereceive-hook on master
$ echo "Compare branches in a pre-receive hook" >> README.md && git commit -am "Trigger the pre-receive hook" && git push origin HEAD:diff-branches --push-option=compare-branch --push-option=master
[master d3dea93] Trigger the pre-receive hook
 1 file changed, 1 insertion(+)
Counting objects: 6, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 742 bytes | 742.00 KiB/s, done.
Total 6 (delta 2), reused 0 (delta 0)
remote: Resolving deltas: 100% (2/2), completed with 1 local object.
remote: diff-large-branches.sh: failed with exit status 42
remote: diff-branches: Starting diff from master to compare-branch
remote: A	100.binary
remote: A	101.binary
remote: A	102.binary
remote: A	103.binary

... 800+ lines omitted ...

remote: A	997.binary
remote: A	998.binary
remote: A	999.binary
remote: M	README.md
remote: M	block_file_extensions.sh
remote: D	diff-large-branches.sh
remote: diff-branches: Finished diff
To https://octodemo.com/jonico/large-branch-diff-with-prereceive-hook.git
 ! [remote rejected] HEAD -> diff-branches (pre-receive hook declined)
error: failed to push some refs to 'https://octodemo.com/jonico/large-branch-diff-with-prereceive-hook.git'

Example usage for client script (does not need any local clone of the repository):

$ ./diff-branches-client.sh https://jonico@octodemo.com/jonico/large-branch-diff-with-prereceive-hook.git compare-branch

A	100.binary        
A	101.binary        

... 800+ lines omitted ...

A	997.binary        
A	998.binary        
A	999.binary        
M	README.md        
M	block_file_extensions.sh        
D	diff-large-branches.sh

Example on how to diff a specific commit:

$ ./diff-branches-client.sh https://jonico@octodemo.com/jonico/large-branch-diff-with-prereceive-hook.git f75b18abbb06 f75b18abbb06^1
M	README.md    

Example on how to diff the latest commit on master with its predecessor:

$ ./diff-branches-client.sh https://jonico@octodemo.com/jonico/large-branch-diff-with-prereceive-hook.git master master^1
A	diff-branches-client.sh        
M	diff-large-branches.sh