Skip to content

Instantly share code, notes, and snippets.

@frioux

frioux/ditch-master.sh

Created June 13, 2020 17:58
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save frioux/6c259c63dda705f9ef9d030ddb880347 to your computer and use it in GitHub Desktop.
Save frioux/6c259c63dda705f9ef9d030ddb880347 to your computer and use it in GitHub Desktop.
Download ZIP
You can run this (using https://github.com/ingydotnet/git-hub) to rename all default branches from master to main; run from a fresh, empty repo or the repo you run it from will get all bloated with garbage refs.
Raw
ditch-master.sh
#!/bin/sh
set -e
for r in $(git hub repos -r --all); do
json=$(git hub repo "$r" --json)
# ignore forks
if [ -n "$(echo "$json" | jq -r '.["source/full_name"]+""')" ]; then
echo "skipping $r; fork"
continue
fi
archived=$(git hub repo-get $r archived)
if [ "$archived" = "true" ]; then
echo "skipping $r; archived"
continue
fi
# find out if HEAD is master:
br=$(git hub repo-get $r default_branch)
if [ "$br" != master ]; then
echo "skipping $r; default_branch=$br"
continue
fi
echo "fetching $r to rename branch..."
git fetch -q git@github.com:$r
git push git@github.com:$r FETCH_HEAD:refs/heads/main
git hub repo-edit $r default_branch main
git push git@github.com:$r --delete master
done
@frioux
Copy link
Author

frioux commented Jun 16, 2020 via email

@spazm
Copy link

spazm commented Jul 15, 2020

So yeah, it's a month later...

TL;DR: possible, but not included in the default git tooling. So I wrote my own: https://github.com/spazm/git-rename-remote-branch

  1. the git tooling could provide a way to send a branch reference update with an empty pack file, in git push / git send-pack
  2. the git tool does not provide a way to send a branch reference update without a full pack file - the code to create and send the message in git send-pack is tightly coupled in a single function.
  3. the current git tooling accepts and validates a branch reference update message with an empty pack file. ( git-receive-pack)
  4. Forcing it is just a simple matter of programming.

empty pack file == valid version 2 pack file containing zero items.

Once I got the message created (it's a pretty simple 4 line message) and encoded (into pck_line format) I was able to verify that git receive-pack would accept it locally, by just passing data around in the shell.

Testing this across an ssh link to a remote server proved much more difficult because much of the process was opaque. After much chasing of fascinating but completely unrelated strands, I found most of my issue was with timing. With a remote server on a slow host, the welcome message took a while to create, and any input sent before that was lost and ignored. So then we enter the classic programming fun of reading from and writing to pipes of the same program.

overview:

  1. connect to remote server:
  2. read welcome message
  3. parse welcome message for branch shas
  4. create message creating new branch and deleting old branch
  5. write message to server
  6. read and verify response

Valid pack file containing 0 items.

# PACK file, version 2, 0 objects + 20 byte sha1 checksum
my $EMPTY_PACK = pack("A4NN", "PACK", 2, 0);
$EMPTY_PACK .= sha1($EMPTY_PACK);

format of the message format to create a new branch and remove an old branch,
both of which use a special 40 char sha placeholder of all zeros.

<40 zeros> <40 bit sha of old branch> <name of new branch>
<40 bit sha of old branch> <40 zeros> <name of old branch>
<empty line>
<empty pack file>

The lines must be encoded in packline format, which prefixes a length count to the line. The pack file is not encoded in packline format and as raw bytes. The empty line is encoded as a sync packet 0000.

The server also expects options to be included on the first line of update message, appended after a \0 separator.

Actual magic message to rename master to main in repository git@github.com:spazm/config.git

00860000000000000000000000000000000000000000 47665e3c9209f44824e076359293fe51906c6317 refs/heads/main\0 report-status agent=git/2.17.1
006847665e3c9209f44824e076359293fe51906c6317 0000000000000000000000000000000000000000 refs/heads/master
0000PAC;بj<>

the first two messages have 4 byte hex header prefixes of 0086 and 0068 encoding the lengths of the messages, including the optional newlines I include for clarity.
the third message is the sync packet 0000 and is followed directly by the pack file blob

Without newlines, the same message would have shorter length prefixes and look like a single long line to a human:

00850000000000000000000000000000000000000000 47665e3c9209f44824e076359293fe51906c6317 refs/heads/main\0 report-status agent=git/2.17.1006747665e3c9209f44824e076359293fe51906c6317 0000000000000000000000000000000000000000 refs/heads/master0000PAC;بj<>

@spazm
Copy link

spazm commented Jul 15, 2020

a perl version, written against v5.14 for wide portability. Attempting to mimic the ui and git aesthetic. Currently only works for ssh remote repos in the scp style: [user@]host:org/project.git

from: https://github.com/spazm/git-rename-remote-branch/blob/main/git-rename-remote-branch

#!/usr/bin/env perl
package GitRename;

use v5.14.0;
use strict;
use warnings;

# Core only for distribution
use Digest::SHA1 qw(sha1);
use FileHandle ();
use Getopt::Long v2.32 qw(:config bundling auto_version);
use IO::Select () ;
use IPC::Open3 qw(open3);
use Pod::Usage qw(pod2usage);
use POSIX qw(:sys_wait_h);

# USER OVERRIDE via environment variables
our $DEBUG = exists $ENV{DEBUG}   ? $ENV{DEBUG}   : 1;
our $SSH   = exists $ENV{GIT_SSH} ? $ENV{GIT_SSH} : 'ssh';

my $GIT_RECEIVE_PACK = "git-receive-pack";
my $BUFFER_READ_SIZE = 4096;
my $EMPTY_SHA        = "0" x 40;

# PACK file, version 2, 0 objects + 20 byte sha1 checksum
my $EMPTY_PACK = pack("A4NN", "PACK", 2, 0);
$EMPTY_PACK .= sha1($EMPTY_PACK);

# hacky manual logging to keep this program self-contained + core
sub trace { say STDERR "TRACE: ", @_ if $DEBUG >= 2 }
sub debug { say STDERR "DEBUG: ", @_ if $DEBUG >= 1 }
sub info  { say STDERR "INFO:  ", @_ if $DEBUG >= 0 }
sub done  { info @_ if @_;    exit(0) }
sub fatal { say STDERR $_[0]; exit($_[1] // 1 ) }

sub main {
  my ($repo, $old_name, $new_name) = parse_args();
  my $git = GitRename->new($repo, $old_name, $new_name);

  info("Renaming '$git->{old_name}' to '$git->{new_name}' on '$git->{remote}'");
  $git->rename_remote_branch();
  info("Complete.  Renamed $git->{old_name} to $git->{new_name}");
}

sub parse_args {
  # Configure option parsing to mimic other git commands
  #
  # Parse command line flags with Getopt::Long and then
  # manually read and verify the three positional arguments:
  #   <repository> <old_branch> <new_branch>
  #
  # returns (repo, old_name, new_name)

  my %opt = (
    'receive-pack-git' => \$GIT_RECEIVE_PACK,
    'verbose'          => 0,
  );
  my $result = GetOptions(
    \%opt,
    "verbose|v+",
    "quiet|q",
    "receive-pack-git|exec=s",
    "man",
    "help",
  );
  pod2usage(1) if $opt{help};
  pod2usage(-exitval => 0, -verbose =>2) if $opt{man};
  pod2usage(2) if !$result;

  $DEBUG += $opt{verbose};
  $DEBUG = -1 if $opt{quiet};

  if (@ARGV < 3 ){
    pod2usage(-msg => "<repository>, <old_branch>, and <new_branch> are all requied", -exitval => 1);
  } elsif (@ARGV > 3) {
    pod2usage(-msg => "Too many arguments", -exitval => 1);
  }

  my ($remote, $old_name, $new_name) = @ARGV;
  $old_name    =~ s,refs/heads/,,;
  $new_name    =~ s,refs/heads/,,;

  ($remote, $old_name, $new_name);
}

### Convenience Functions ###

sub format_line {
  # packet_line format requires a 4 byte hex header containing the length
  # of the line, including header.
  # newlines are not required at the end of lines. If present they must be
  # included in the length
  #
  # 0004 empty message is not allowed
  # 0000 is a special packet_sync message.
  # => we convert empty $line into packet_sync.
  #
  # If using sidechannel, an additional byte is included after the header to indicate which sidechannel

  my ($line, $sidechannel) = @_;
  $sidechannel //= 0;
  my $len = length($line);
  $len += 4 if $len;
  if ($sidechannel) {
    sprintf("%04X%s%s", $len + 1, $sidechannel, $line)
  } else {
    sprintf("%04X%s", $len, $line)
  }
}

sub decode_pack_lines {
  # decode variable length encoded lines.
  # 4 byte header containing the length of the rest of the message
  # newlines may be used between messages, but must be included in the length.

  my $msg = shift;
  my @msgs = ();
  my $offset = 0;
  while ($offset < length($msg) ){
    my $len = hex(substr($msg, $offset, 4));
    $len = 4 if $len == 0;
    if ($len > 4){
      my $m = substr($msg, $offset + 4, $len - 4);
      trace "offset: $offset, len:$len, m:[$m]";
      chomp($m);
      push @msgs, $m;
    }
    $offset += $len;
  }
  return @msgs
}

sub read_io_buffer {
  # Convenience method to do non-blocking sysread on a filehandle.
  # read will be blocking if $delay is 0
  # delay is lowered to 1 after the initial read.

  my ($io, $delay) = @_;
  $delay //= 10;

  my $buffer     = "";
  my $tmp_buffer = "";
  my $done       = 0;

  while (not $done and my @ready = $io->can_read($delay))  {
    $delay = 1;
    foreach my $reader (@ready) {
      my $bytes_read = sysread($reader, $tmp_buffer, $BUFFER_READ_SIZE);
      trace("    read_buffer: read $bytes_read from reader.  partial_buffer:[$tmp_buffer]");
      $buffer .= $tmp_buffer;
      $done = 1 if not defined($bytes_read) or $bytes_read == 0;
    }
  }
  return $buffer;
}


sub new {
  my ($class, $remote, $old_name, $new_name) = @_;

  # normalize name and ref
  $old_name    =~ s,refs/heads/,,;
  $new_name    =~ s,refs/heads/,,;
  my $old_ref  = 'refs/heads/' . $old_name;
  my $new_ref  = 'refs/heads/' . $new_name;

  pod2usage("<old_branch> and <new_branch> must be different") if $old_name eq $new_name;

  my ($user_host, $path) = split(':', $remote);
  pod2usage("path is required in repository") unless defined $path;
  $path .= '.git' unless $path =~ m/.git$/;

  bless my $self = {
    old_name  => $old_name,
    old_ref   => $old_ref,
    new_name  => $new_name,
    new_ref   => $new_ref,
    remote    => $remote,
    user_host => $user_host,
    path      => $path,
    sha       => undef,
  }, $class
}

sub rename_remote_branch {
  # Top level API
  # connects,
  # reads and parses the initial message,
  # sends the rename message
  # reads and parses the unpack response.
 
  my $self = shift;
  my $wh = $self->open_ssh($self);

  $self->read_welcome_message();
  # we only care about errors from ssh at the beginning, e.g. if our repo is rejected.
  $self->check_for_error_message();
  $self->write_rename_message();
  $self->read_confirmation_message();

  close($wh);
  waitpid($self->{pid}, 0);
}

sub open_ssh {
  # Run ssh in a sub-process connected with pipes for input,
  # output and stderr.
  #
  # Creates an IO::Select for each of input, output, stderr
  # Adds io_read, io_write, and io_err IO::Select objects
  # and pid of child process to self.
  #
  # returns write handle so the caller can explicitly close it
  # and then wait for the child to complete.

  my $self = shift;
  my @ssh_cmd = ($SSH, '-x', $self->{user_host}, $GIT_RECEIVE_PACK, "'$self->{path}'");
  debug("ssh_cmd: @ssh_cmd");

  my $pid = open3(
    my $wh = FileHandle->new(),
    my $rh = FileHandle->new(),
    my $eh = FileHandle->new(),
    @ssh_cmd
  );

  # open3 opens the handles, need to set binmode after open and before reading/writing.
  binmode($rh, ':raw');
  binmode($wh, ':raw');
  binmode($eh, ':raw');

  # use separate reader/writer because ->can_read occasionally returns the writer.
  my $io_read  = IO::Select->new($rh);
  my $io_write = IO::Select->new($wh);
  my $io_err   = IO::Select->new($eh);  # error messages appear in can_read() instead of has_exceptions()!

  $self->{io_read}  = $io_read;
  $self->{io_write} = $io_write;
  $self->{io_err}   = $io_err;
  $self->{pid}      = $pid;

  return $wh;
}

sub read_welcome_message {
  # Reads the initial server connect message on io_read, parsing the
  # SHA and branch referenes.
  # uses verify_branches to check that old_ref exists and new_ref does not.
  #
  # returns the message read.

  my $self = shift;
  my $initial_msg = read_io_buffer($self->{io_read}, 10);
  trace("welcome message sysread: $initial_msg");
  $self->verify_branches($initial_msg);
  return $initial_msg;
}

sub check_for_error_message {
  # Checks for error messages on io_err at the start of connection
  # that indicate a rejection of our connection -- most likely an
  # incorrect repo path.
  #
  # returns the message read.

  my $self = shift;
  my $error_msg = read_io_buffer($self->{io_err}, 1);
  trace("error_message sysread: $error_msg") if $error_msg;
  fatal($error_msg) if (waitpid($self->{pid}, WNOHANG));
  return $error_msg;
}

sub write_rename_message {
  # Writes the rename message to io_write and returns
  # the number of bytes written.
  my $self = shift;
  my $cmd = $self->rename_remote_branch_command();
  trace("remote cmd: [$cmd]");

  my $bytes_written = 0;
  debug ("sending cmd to ssh. length:" . length($cmd));
  if (my @ready = $self->{io_write}->can_write(10)){
    $bytes_written = syswrite($ready[0], $cmd);
    if (defined($bytes_written)) {
      debug "success: wrote $bytes_written bytes";
    } else {
      fatal("write failed: $!");
    }
  }
  return $bytes_written;
}

sub read_confirmation_message {
  # Reads the server confirmation message io_read and verifies
  # the expected message pattern with verify_update_response.
  #
  # Expected message:
  #    unpack ok
  #    ok ref1
  #    ok ref2
  #
  # returns the message read.

  my $self = shift;
  my $server_response = read_io_buffer($self->{io_read}, 10);
  trace "confirmation message sysread: [$server_response]";
  $self->verify_update_response($server_response);
  return $server_response;
}

sub verify_branches {
  # Check that $old exists and $new does not
  # return the SHA for $old
  #
  # git-receive-pack initial message, in pck_line format
  # SHA refs/heads/$branch1\0$options
  # SHA refs/heads/$branch2
  # ...
  # SHA refs/heads/$branchn
  # ""

  my ($self, $initial_msg) = @_;
  my ($old_sha, $new_sha);
  foreach my $line (split(/\n/, $initial_msg)){
    if ($line =~ m|(\w{40}) \s+ (refs/heads/[^\x00]+)|x) {
      # we don't need the whole hash of branch names, just the two we want.
      debug("Found branch[$2] with sha[$1]");
      $old_sha = $1 if $2 eq $self->{old_ref};
      $new_sha = $1 if $2 eq $self->{new_ref};
    } elsif ($line == '0000' ) {
      # redundant, the 0000 message should be the last line.
      last;
    }
  }
  $self->{sha} = $old_sha;

  done "already complete" if !$old_sha and $new_sha;
  fatal ("$self->{old_name} does not exist in remote", 2) unless $old_sha;
  fatal ("$self->{new_name} already exists in remote", 2) if $new_sha;
  return 1;
}

sub verify_update_response {
  # Expected response:
  #     unpack ok
  #     ok $ref
  #     ok $ref
  #
  # Note: OK message order is not guaranteed

  my ($self, $msg) = @_;
  my @msgs = decode_pack_lines($msg);

  if (@msgs != 3 or shift(@msgs) ne "unpack ok") {
    debug $_ foreach @msgs;
    fatal("message was not successfully received and unpacked");
  }

  my %ok;
  foreach my $line (@msgs){
    if ($line =~ m/ok (.*)/i ) {
      $ok{$1} = 1;
    } else {
      fatal "invalid OK reponse: $line\n";
    }
  }
  unless ($ok{$self->{new_ref}} and $ok{$self->{old_ref}}) {
    debug $_ foreach @msgs;
    fatal("OK message missing", 2);
  }
  return 1;
}

sub rename_remote_branch_command {
  # Create the custom 4 line message to
  # create new_name and delete old_name
  # using an valid PACK file with 0 elements that
  # we can build without access to the refs in the repo.

  my $self = shift;
  my $options = "\0 report-status agent=git/2.17.1";
  # importantly, we are not enabling sideband. 

  # git-receive-pack message format:
  #   OLD_SHA NEW_SHA BRANCH\0OPTIONS
  #   OLD_SHA NEW_SHA BRANCH
  #   OLD_SHA NEW_SHA BRANCH
  #   SYNC_PACKET
  #   PACKFILE
  #
  # set OLD_SHA to EMPTY_SHA to create a branch
  # set NEW_SHA to EMPTY_SHA to delete a branch

  # create new_ref by setting OLD_SHA to EMPTY
  # delete old_ref by setting NEW_SHA to EMPTY
  my $cmd = format_line("$EMPTY_SHA $self->{sha} $self->{new_ref}$options\n") .
            format_line("$self->{sha} $EMPTY_SHA $self->{old_ref}\n") .
            format_line("") .
            $EMPTY_PACK
            ;
}

main() if not caller();

__END__

=head1 NAME

git rename-remote-branch - Rename a remote branch without requiring a local checkout

=head1 SYNOPSIS

git rename-remote-branch [options] <repository> <old_branch> <new_branch>

  options:
    --verbose|-v        increase verbosity (can be used multiple times)
    --quiet|-q          decrease verbosity completely
    --receive-pack-git  override the executable run on the remote side.
    --help              this message
    --man               show the man page

=head1 DESCRIPTION

B<git rename-remote-branch> renames a branch in a remote repository without requiring a local checkout or downloading refs.

This works similarly to the low-level command B<git upload-pack>, except that B<git rename-remote-branch> creates and sends an empty PACK file.

Only ssh format repositories are supported : [user@]host:path/to/repo.git

Upon connection, the remote sends a welcome message contraining the SHA for each branch.  The remote will ignore any input sent before this message is completed.

This program parses the welcome message and extracts the SHA for <old_branch> and verifies that <new_branch> does not exist.

This program then crafts a message to create the new branch and delete the old branch.  This messages is encoded in pack_line format and sent to the remote along with a specially crafted empty_pack_file.

The server then responds with an "unpack ok" message, followed by and "ok <branch>" message for each branch referenced in the sent message.

=head1 OPTIONS

=over 4

=item B<--verbose|-v>

Increase debug level

=item B<--quiet|-q>

Display no messages (sets DEBUG = -1)

=item B<--receive-pack-git|--exec>

Takes a string argument representing an alternate B<git-receive-pack> exeutable to run on the remote host.

See the documentation for B<git send-pack> for more details.

=item B<--help>

Prints a brief help message and exits.

=item B<--man>

Prints the manual page and exits.

=back

=head1 ENVIRONMENT

=over 4

=item B<GIT_SSH>

Choose an alternate ssh binary.  Defaults to C<ssh>

=item B<DEBUG>

Override the default DEBUG level.  This can also be accomplished using C<--verbose> or C<--quiet>.

=back

=head1 DEFINITIONS

=head2 Rename branch message

    <empty_sha> <old_branch_sha> <new_branch>\0<options>
    <old_branch_sha> <empty_sha> <old_branch>
    ""
    <empty_pack_file>

Example:

    0001

    0000
    PACK

where <empty_sha> is a string of 40 zeros.

=head2 Pack_line format

Each message in pack_line format is prefixed with a 4 byte length in hex, followed by the message.  The length inclues the 4 bytes of prefix.

Messages may end with a newline, but this must be included in the length. The server is required to treat messages with or without newlines equivalently.

Examples:

    0010hello world\n
    000Fhello world

The blank message of C<0004> is not allowed.

C<0000> is used as a sync message.

=head2 Message in pack_line format

Connects to remote Verified that <old_branch> exists and <new_branch> does not.

=head2 Empty pack file

The documentation for the git message protocol specify that an empty pack file must be sent when deleting a branch.  In reality, C<git push> and C<git send-pack> do not include a PACK file at all when sending a delete command.

When creating a branch, a normal PACK file is created showing the state of the current repository.

It is possible to create and construct a valid packfile containing zero entries.  This ability is not exposed anywhere in the git libraries.  The closest is C<git send_pack>, which uses C<git pack-ref> to create the pack file to send with the reference sha messages.  The logic for building the messages and packfile and ending them are all hiddne behind a single api function.  If no refs are provided, C<git pack-ref> will exit without creating a packfile rather than create one with zero items.

Git pack file has a format consisting of a 4 character header "PACK" followed by the version (2 or 3) and the number of objects, both encoded in network byte order.  The pack file then contains a 20 byte SHA hash of that message.  This seems very simple but was very difficult to find and test.

    my $EMPTY_PACK = pack("A4NN", "PACK", 2, 0);
    $EMPTY_PACK .= sha1($EMPTY_PACK);

=cut

@spazm
Copy link

spazm commented Jul 15, 2020

updating here, now, so I don't forget and let it drop off the radar again.

ps. pack and unpack in perl are onerous yet awesome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment