Skip to content

Instantly share code, notes, and snippets.

@zmousm
Last active August 29, 2015 14:07
Show Gist options
  • Save zmousm/1e70ddcf48696a31d694 to your computer and use it in GitHub Desktop.
Save zmousm/1e70ddcf48696a31d694 to your computer and use it in GitHub Desktop.

git push/fetch over stdio

Git uses send-pack and fetch-pack for transferring data over the smart protocol,. These plumbing commands are a bit too smart for their own good, in the sense that they always try to run their peer commands (receive-pack and upload-pack respectively), either on the local host or over ssh to the remote host (for file:// and ssh:// remotes, respectively). In some cases it may not be feasible for git to connect to the remote host and/or run the peer commands, or it may just be preferrable that such processes are not spawned by git itself. Even then it should still be possible to connect the two ends, e.g. over stdio.

The git commands pipe-send-pack and pipe-fetch-pack are wrappers for the respective git commands, which do just that. These commands, along with gfp2gur.awk (which is used by pipe-fetch-pack to update refs fetched from the source), must be installed on the systems where the target repositories live (anywhere in the PATH for a non-interactive command). File descriptor redirection currently requires socat(1), which must also be installed on these systems. You will also need awk and mktemp for pipe-fetch-pack.

git_stdio_client.sh provides (bash) shell functions that setup ssh connections to the sending and receiving hosts and connect the two through a bidirectional pipe. The versions ending in 2 depend on the git commands above; the other versions call socat directly, so the git commands are not necessary, however these implementations are messier and more error prone (but otherwise mostly equivalent in functionality).

Having loaded the functions in your shell, you can then do:

git-pipe-push sending-host:/path/to/source/git/repo receiving-host:/path/to/destination/git/repo

Or for the equivalent of a git fetch:

git-pipe-fetch receiving-host:/path/to/destination/git/repo sending-host:/path/to/source/git/repo

git fetch operations update refs for a (potentially) virtual remote, whose name can be controlled through the --remote option and defaults to the sending-host. If no refs are specified, they default to --all. On the other hand git push operations typically update refs common to the source and the target repo.

Known issues

receive-pack may close the stream too soon, which results in send-pack (the receiving end) complaining like this: fatal: The remote end hung up unexpectedly The socat wrapper may also complain for a broken pipe. As far as I understand, this is harmless. You may verify refs and objects on the receiving end, or you may also repeat the process after setting GIT_TRACE=1 and/or GIT_TRACE_PACKET=1 on the sending-host, in order to check if the transfer completes correctly.

#!/usr/bin/awk -f
BEGIN {
if (remote "x" == "x")
exit
cmd = gitprefix "update-ref"
}
$1 ~ /^[0-9a-f]+$/ && length($1) == 40 {
if ($2 == "HEAD")
sref = "refs/remotes/" remote "/" $2
else {
sref = $2
sub(/heads/, "remotes/" remote, sref)
}
print cmd, sref, $1
}
#!/bin/sh
USAGE='[--write-refs file] [git-fetch-pack options]'
OPTIONS_SPEC=
SUBDIRECTORY_OK=Yes
save () {
what="$1"
shift
for i; do
if test "$what" = opts && printf %s\\n "$i" | grep -q -- "^--"; then
:
elif test "$what" = args && printf %s\\n "$i" | grep -qv -- "^--"; then
:
elif test "$what" = all; then
:
else
continue
fi
# escape : for socat
i=$(printf %s\\n "$i" | sed "s/:/\\\:/g")
printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/"
echo " "
done
unset what i
}
. "$(git --exec-path)/git-sh-setup"
write_refs=
i=1
while test ! $i -gt $# ; do
eval "ival=\$$i"
eval "jval=\$$((i+1))"
case "$ival" in
--write-refs=*)
write_refs=$(printf %s\\n "$ival" | sed 's/--write-refs=//')
shift
;;
--write-refs)
if test -n "$jval" && printf %s\\n "$jval" | grep -qv -- "^-"; then
write_refs="$jval"
shift 2
else
usage
fi
;;
esac
i=$((i+1))
done
unset i ival jval
opts=$(save opts "$@")
args=$(save args "$@")
# fetch-pack apparently needs some refs
if test -z "$args" && printf %s\\n "$opts" | grep -qv -- "--all"; then
eval "set -- $opts --all . $args"
else
eval "set -- $opts . $args"
fi
unset opts args
socat=$(which socat)
if test -z "$socat" || test ! -x "$socat"; then
die "This command requires socat(1)"
fi
gfpcmd="git fetch-pack --upload-pack=\\\"$socat - 5 #\\\" $@"
if test -n "$write_refs"; then
if touch "$write_refs" && test -w "$write_refs"; then
:
else
die "$write_refs not writeable"
fi
gfpcmd="$gfpcmd \>$write_refs"
fi
$socat STDIO SYSTEM:"$gfpcmd",fdin=5
#!/bin/sh
USAGE='[git-send-pack options]'
OPTIONS_SPEC=
SUBDIRECTORY_OK=Yes
save () {
what="$1"
shift
for i; do
if test "$what" = opts && printf %s\\n "$i" | grep -q -- "^--"; then
:
elif test "$what" = args && printf %s\\n "$i" | grep -qv -- "^--"; then
:
elif test "$what" = all; then
:
else
continue
fi
# escape : for socat
i=$(printf %s\\n "$i" | sed "s/:/\\\:/g")
printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/"
echo " "
done
unset what i
}
. "$(git --exec-path)/git-sh-setup"
opts=$(save opts "$@")
args=$(save args "$@")
eval "set -- $opts . $args"
unset opts args
socat=$(which socat)
if test -z "$socat" || test ! -x "$socat"; then
die "This command requires socat(1)"
fi
gspcmd="git send-pack --receive-pack=\\\"$socat - 5 #\\\" $@"
$socat STDIO SYSTEM:"$gspcmd",fdin=5
#!/bin/bash
function git-pipe-push () {
local shost spath dhost dpath opts=() args=() justafifo rc
if [ $# -ge 2 ]; then
eval $(perl -e \
'my @d = qw(s d);
for (my $i=0; $i<=1; $i++) {
if ($ARGV[$i] =~ /^(\[[0-9A-Fa-f:]+\]|[^:]+):(.*)$/) {
printf "%1\$shost=\"%2\$s\" %1\$spath=\"%3\$s\" ", $d[$i], $1, $2 || ".";
}
}' "${@:1:2}")
fi
if [ -z "$shost" -o -z "$dhost" -o "$1" = "-h" ]; then
echo "Usage: ${FUNCNAME[0]} sending-host:[path] receiving-host:[path] [git-send-pack options]" >&2
[ "$1" = "-h" ] && return 0 || return 1
else
shift 2
fi
while [ -n "$1" ]; do
case "$1" in
--*)
opts+=("$1")
;;
*)
args+=("$1")
esac
shift
done
opts+=(".")
opts+=("${args[@]}")
justafifo=$(mktemp -u /tmp/gitpipe.XXXXXX)
mkfifo "$justafifo"
rc=$?
if [ $rc -eq 0 ]; then
trap 'rm -f "$justafifo"' HUP INT QUIT TERM KILL
else
return $rc
fi
ssh $shost \
"cd \"$spath\";"\
"socat - EXEC:'git send-pack --receive-pack=\\\"socat - 5 #\\\" ${opts[@]//:/\:}',fdin=5"\
<"$justafifo" | \
ssh $dhost \
"git receive-pack \"$dpath\"" \
>"$justafifo"
rc=$?
rm -f "$justafifo"
return $rc
}
function git-pipe-fetch () {
local shost spath dhost dpath remote opts=() args=() justafifo rc
if [ $# -ge 2 ]; then
eval $(perl -e \
'my @d = qw(s d);
for (my $i=0; $i<=1; $i++) {
if ($ARGV[$i] =~ /^(\[[0-9A-Fa-f:]+\]|[^:]+):(.*)$/) {
printf "%1\$shost=\"%2\$s\" %1\$spath=\"%3\$s\" ", $d[$i], $1, $2 || ".";
}
}' "${@:1:2}")
fi
if [ -z "$shost" -o -z "$dhost" -o "$1" = "-h" ]; then
echo "Usage: ${FUNCNAME[0]} receiving-host:[path] sending-host:[path] [--remote=remote_name] [git-fetch-pack options]" >&2
[ "$1" = "-h" ] && return 0 || return 1
else
shift 2
fi
if test -n "$1" && echo "$1" | grep -q "^--remote="; then
remote="${1#--remote=}"
shift
fi
while [ -n "$1" ]; do
case "$1" in
--*)
opts+=("$1")
;;
*)
args+=("$1")
esac
shift
done
# fetch-pack apparently needs refs
if test -z "${args[*]}" && echo "${opts[*]}" | grep -qv -- "--all"; then
opts+=("--all")
fi
opts+=(".")
opts+=("${args[@]}")
justafifo=$(mktemp -u /tmp/gitpipe.XXXXXX)
mkfifo "$justafifo"
rc=$?
if [ $rc -eq 0 ]; then
trap 'rm -f "$justafifo"' HUP INT QUIT TERM KILL
else
return $rc
fi
ssh $shost \
"cd \"$spath\";"\
"export refstmp=$(mktemp -u /tmp/gitfetchpack.XXXXXXXX);"\
"socat - SYSTEM:'git fetch-pack --upload-pack=\\\"socat - 5 #\\\" ${opts[@]//:/\:} >\$refstmp',fdin=5;"\
"awk -v remote=\"${remote:-$dhost}\""\
"'\$1 ~ /^[0-9a-f]+$/ && length(\$1) == 40 {
if (\$2 == \"HEAD\") { sref = \"refs/remotes/\" remote \"/\" \$2; }
else { sref = \$2; sub(/heads/, \"remotes/\" remote, sref); };
print \"update-ref\", sref, \$1;
}' \${refstmp} | xargs -L 1 git;"\
"rm \${refstmp}"\
<"$justafifo" | \
ssh $dhost \
"git upload-pack \"$dpath\"" \
>"$justafifo"
rc=$?
rm -f "$justafifo"
return $rc
}
function git-pipe-push2 () {
local shost spath dhost dpath opts=() sudo justafifo rc
if [ $# -ge 2 ]; then
eval $(perl -e \
'my @d = qw(s d);
for (my $i=0; $i<=1; $i++) {
if ($ARGV[$i] =~ /^(\[[0-9A-Fa-f:]+\]|[^:]+):(.*)$/) {
printf "%1\$shost=\"%2\$s\" %1\$spath=\"%3\$s\" ", $d[$i], $1, $2 || ".";
}
}' "${@:1:2}")
fi
if [ -z "$shost" -o -z "$dhost" -o "$1" = "-h" ]; then
echo "Usage: ${FUNCNAME[0]} sending-host:[path] receiving-host:[path] [--sudo[=user]] [git-send-pack options]" >&2
[ "$1" = "-h" ] && return 0 || return 1
else
shift 2
fi
while [ -n "$1" ]; do
case "$1" in
--sudo*)
if [ -z "${1#--sudo}" ]; then
sudo=root
else
sudo="${1#--sudo=}"
fi
;;
*)
opts+=("$1")
;;
esac
shift
done
justafifo=$(mktemp -u /tmp/gitpipe.XXXXXX)
mkfifo "$justafifo"
rc=$?
if [ $rc -eq 0 ]; then
trap 'rm -f "$justafifo"' HUP INT QUIT TERM KILL
else
return $rc
fi
if [ -n "$sudo" ]; then
ssh $shost \
"sudo -u \"$sudo\" -- sh -c '"\
"cd \"$spath\";"\
"git pipe-send-pack ${opts[@]};"\
"'"\
<"$justafifo" | \
ssh $dhost \
"sudo -u \"$sudo\" -- sh -c '"\
"git receive-pack \"$dpath\"" \
"'"\
>"$justafifo"
else
ssh $shost \
"cd \"$spath\";"\
"git pipe-send-pack ${opts[@]};"\
<"$justafifo" | \
ssh $dhost \
"git receive-pack \"$dpath\"" \
>"$justafifo"
fi
rc=$?
rm -f "$justafifo"
return $rc
}
function git-pipe-fetch2 () {
local shost spath dhost dpath opts=() remote sudo justafifo rc
if [ $# -ge 2 ]; then
eval $(perl -e \
'my @d = qw(s d);
for (my $i=0; $i<=1; $i++) {
if ($ARGV[$i] =~ /^(\[[0-9A-Fa-f:]+\]|[^:]+):(.*)$/) {
printf "%1\$shost=\"%2\$s\" %1\$spath=\"%3\$s\" ", $d[$i], $1, $2 || ".";
}
}' "${@:1:2}")
fi
if [ -z "$shost" -o -z "$dhost" -o "$1" = "-h" ]; then
echo "Usage: ${FUNCNAME[0]} receiving-host:[path] sending-host:[path] [--remote=remote_name] [--sudo[=user]] [git-fetch-pack options]" >&2
[ "$1" = "-h" ] && return 0 || return 1
else
shift 2
fi
while [ -n "$1" ]; do
case "$1" in
--remote*)
if [ -z "${1#--remote}" ]; then
echo "missing required argument to --remote" >&2
return 1
else
remote="${1#--remote=}"
fi
;;
--sudo*)
if [ -z "${1#--sudo}" ]; then
sudo=root
else
sudo="${1#--sudo=}"
fi
;;
*)
opts+=("$1")
;;
esac
shift
done
justafifo=$(mktemp -u /tmp/gitpipe.XXXXXX)
mkfifo "$justafifo"
rc=$?
if [ $rc -eq 0 ]; then
trap 'rm -f "$justafifo"' HUP INT QUIT TERM KILL
else
return $rc
fi
if [ -n "$sudo" ]; then
ssh $shost \
"sudo -u \"$sudo\" -- sh -c '"\
"cd \"$spath\";"\
"export refstmp=$(mktemp -u /tmp/gitfetchpack.XXXXXXXX);"\
"git pipe-fetch-pack --write-refs=\${refstmp} ${opts[@]};"\
"gfp2gur.awk -v remote=\"${remote:-$dhost}\" \${refstmp} | xargs -L 1 git;"\
"rm \${refstmp}"\
"'"\
<"$justafifo" | \
ssh $dhost \
"sudo -u \"$sudo\" -- sh -c '"\
"git upload-pack \"$dpath\"" \
"'"\
>"$justafifo"
else
ssh $shost \
"cd \"$spath\";"\
"export refstmp=$(mktemp -u /tmp/gitfetchpack.XXXXXXXX);"\
"git pipe-fetch-pack --write-refs=\${refstmp} ${opts[@]};"\
"gfp2gur.awk -v remote=\"${remote:-$dhost}\" \${refstmp} | xargs -L 1 git;"\
"rm \${refstmp}"\
<"$justafifo" | \
ssh $dhost \
"git upload-pack \"$dpath\"" \
>"$justafifo"
fi
rc=$?
rm -f "$justafifo"
return $rc
}
@zmousm
Copy link
Author

zmousm commented Oct 17, 2014

git-fetch-pack seems to spit refs in stdout, which we need in order to update-ref

@rowanthorpe
Copy link

FWIW: Passing -q to git-fetch-pack might help anyway (in other ways), but seems to only quieten the output of git unpack-objects... Also, what about the --no-progress flag?

@zmousm
Copy link
Author

zmousm commented Oct 17, 2014

The problem is not how to make fetch-pack quiet (I think that stuff goes to stderr) but how to get its' output (refs fetched) in the middle of all this redirection hell.

@zmousm
Copy link
Author

zmousm commented Oct 18, 2014

Fixed! What a mess :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment