Move a subdirectory from one git repository to a subdirectory of another, without losing commit history.

commands.sh

# Assumptions:
#
# * After the merge is complete, the directory should exist in the target repository and not exist in the source
#   repository. In other words, this is a move, not a cross-reference: we never want to be able to push commits
#   back to the source repository, or merge further changes from the source repository. If you want to do that
#   instead, a subtree merge is what you're looking for.
#
# * The directory doesn't exist in the target repository yet.
#
# In this example, we're moving the "/openstack-swift/" directory from jclouds/jclouds-labs-openstack to
# "/apis/openstack-swift/" in jclouds/jclouds.

# Get a clone of the target repository.
git clone git@github.com:jclouds/jclouds.git
cd jclouds

# Add the source repository as a remote, and perform the initial fetch.
git remote add -f sourcerepo git@github.com:jclouds/jclouds-labs-openstack.git

# Create a branch based on the source repositories' branch that contains the state you want to copy.
git checkout -b staging-branch sourcerepo/master

# Delete everything that isn't the directory we want to move.
# Don't worry, we're not going to push this back ;-)
git rm -r openstack-glance/ openstack-heat/ openstack-marconi/ \
  openstack-neutron/ openstack-autoscale/ rackspace-* \
  *.md pom.xml
git commit -m "Delete everything we don't want to merge"

# Move the directory into its final position.
mkdir apis/
git mv openstack-swift/ apis/
git commit -m "Lock S-foils in attack position"

# Change back to master and merge it in.
# You could also do this by submitting a pull request, of course. I had one conflict, in /.mailmap.
git checkout master
git merge staging-branch

# I had one conflict, in /.mailmap. Fix it, stage it, and commit it to finalize the merge.
${EDITOR} .mailmap
git add .mailmap
git commit

# Voila! The directory has moved, and you still have full history:
#
#  ~/samples/jclouds (master>) $ git log --follow --oneline apis/openstack-swift/pom.xml
#  cb930d9 Move openstack-swift into apis/.
#  491dc97 Revert "Fix poms so that modernizer doesn't fail on snapshot."
#  889243a Fix poms so that modernizer doesn't fail on snapshot.
#  d65048d Fix Maven parent.relativePath warnings
#  f33e90e Updating project versions to 2.0.0-SNAPSHOT
#  be8bc22 Up to 2.0.0-SNAPSHOT after the 1.8.0 release
#
# (Notice that you do need the --follow to trace through the git mv I did.)

# Next: be sure to promptly delete the directory from the source repository by normal means
# (git rm -r dir/ && git commit && git push && pull request), because you don't have an
# easy way to update the target repository with further changes.

One doubt: with this approach you have to add the --follow to the git log command. Do you have to add it always? Or is it possible to show the logs of any file or directory tree just with a normal git log?

I wrote a procedure to do the same thing, based on jclouds-chef here: https://wiki.apache.org/jclouds/PromoteProvider
Although it can seem complicated, it is pretty straightforward and most of it requires is to copy/paste the commands. That allows you to git log normally. WDYT about it?

I'm glad I checked this, gists don't notify when someone leaves a comment ☝️

One doubt: with this approach you have to add the --follow to the git log command. Do you have to add it always? Or is it possible to show the logs of any file or directory tree just with a normal git log?

The --follow is necessary because I used a regular git mv to shuffle the source directory to the right place. If the directory happens to be in the same place within both repositories then it won't be needed. It's the same as if you'd moved the directory around within the same repository.

I wrote a procedure to do the same thing, based on jclouds-chef here: https://wiki.apache.org/jclouds/PromoteProvider
Although it can seem complicated, it is pretty straightforward and most of it requires is to copy/paste the commands. That allows you to git log normally. WDYT about it?

Ah, using filter-branch to make history looks like the directory had always been in the right place? Clever! It should also cut down on repository bloat.

You shouldn't need to do work in a temporary clone and remove the remotes and everything, of course. The filtering is isolated to specific branches so even if you push them by mistake, it won't break anything. Also, rather than copy-paste, this is the kind of thing I'd try to keep around as a script for projects that are likely to need it often (like JClouds).

You can also do some trickery with subtrees to make this happen - I did it the way I did partially because it uses only fairly common git commands, so you can see what it's doing at a glance. The only real leap is realizing that a git repository's remotes can also point to clones that don't even share history.

I'm glad I checked this, gists don't notify when someone leaves a comment

:) Agree. I also checked this without having been notified!

You can also do some trickery with subtrees to make this happen

If I'm not wrong (haven't tried, just seen comments after googling for the best process) when using subtrees you're not able to rebase normally the branch. It presents several issues.

Also, rather than copy-paste, this is the kind of thing I'd try to keep around as a script for projects that are likely to need it often (like JClouds).

+1, and thanks for the feedback! :)