Copying my root filesystem

gistfile1.md

My root filesystem is on an SD card, and when I got a Raspberry Pi I decided to swap that SD card for a larger one and use the original in the Pi. This post chronicles my attempts to copy the filesystem.

Day One

I only have one SD card slot, which means I need to copy through an intermediate storage device. I have a backup disk, so that's not a problem. I also need another operating system, since I can't use my normal one while copying things to the new SD card. (I also no longer have the original OS on my netbook, because I've been using that partition for swap space.) Years ago I put a copy of Ubuntu on a USB stick "in case of emergencies" (I think this is the first time I've needed it), so that's not a problem either.

The new SD card already has one full-size partition, so the first step is to install ext3 on it:

mkfs.ext3 /dev/sdd1

I've been using rdiff-backup to make backups, which the USB stick doesn't have. But it does have rsync, so I copy the backup over:

rsync -av --exclude /rdiff-backup-data /media/backup/root/ /media/new-sd

(-a is archive mode, i.e. preserve permissions, times, etc. -v is verbose. rdiff-backup-data is the directory where rdiff-backup stores statistics and previous versions of files. /media/backup/root is where I've backed up the original SD card to, and the trailing slash tells rsync not to create a directory /media/new-sd/root.)

After the first time I do this, I try to remount the SD card and it takes several minutes. Further investigation with strace reveals that the open syscall on /dev/sdd1 blocks. I have no idea why. I try it on a couple of friends' computers and it works fine, and then I try again with mine and it works again.

Now I install extlinux, which I've been using as a bootloader. Again, it's not on the USB stick, so I use the binary that I've just copied.

/media/new-sd/sbin/extlinux --install /media/new-sd/boot/extlinux

(I'm not sure if this is actually necessary, but it probably won't hurt.) Then I have to install the syslinux MBR. I can't find it on the SD card, but there's a possibly-different version of it on the USB stick:

dd if=/usr/lib/syslinux/mbr.bin of=/dev/sdd

(Later it turns out that mbr.bin is in /usr/share/syslinux/ on the SD card. Admittedly I didn't look very hard.)

I also set /dev/sdd1 to be bootable using fdisk.

At this point I can boot from the new SD card. When I do, it asks me to select a video mode, which my current one doesn't. I assume that's because it's a different MBR.

Unfortunately it hangs during boot, after a message about freeing kernel memory or something. Googling for that message, I learn that I need to create the device files /dev/{console,null,zero}. They weren't copied because when I make backups, I use --exclude-other-filesystems and /dev is mounted as udev. (This sort of thing might be the reason that making backups from a mounted filesystem isn't advised.)

After fixing that it gets a little further, but soon gives me an error about /lib/rc/init.d being on a read-only filesystem. Googling for this error doesn't immediately tell me what's likely to be the problem, except that it's trying to write to the root filesystem before remounting read-write.

I decide to compare the filesystems and see what's different between the two SD cards. Booting back into the USB stick, I mount the two cards in turn and pipe ls -lR to a file.

Running diff -u on these files has 66,000 lines (which is not the same as 66,000 differences, because diff provides context, but it's still a lot to trawl through). An initial examination tells me:

• I'm missing /proc, /sys, /run and /tmp from the new SD card. Again, these directories host mounted filesystems when I'm booted on the old card. They're probably not the current problem, but I'll probably want to create them anyway.

• Some directories (in particular, /dev and /mnt) have a different number of links. The link number seems to be "number of subdirectories plus two".

• Symlinks don't seem to have their modification times preserved. The dates on the new SD card all seem to be older. My first guess was that they were given the dates of the files they linked to, but that doesn't seem to be the case. I'm not sure why it's happening, but it probably isn't a problem, and it's causing problems with the diff output.

I pipe the results through a filter to blank out the dates: perl -pe 's/\d{4}-\d\d-\d\d \d\d:\d\d/xxxx-xx-xx xx:xx/'. Running diff again, I get about 45,000 lines.

I seem to have run into a problem with column alignment. If one file in a directory has more than 10 hard links, all files in that directory get a width-two column for link number. So if a file has a different number of links on the two SD cards, that can cause every file in that directory to show up in the diff output.

Using another filter to normalise padding, I'm down to about 15,000 lines. Looking again, I learn:

• A surprising number of files have multiple hard links on the old SD card. I knew rsync wasn't preserving hard links (rdiff-backup does), but I had assumed there wouldn't be any. Again, this is isn't going to be the cause of my current problems, but it is something I think I should fix.

• The new SD card is missing /lib/rc/init.d, which adds about 130 lines to the diff (so at a guess, indirectly contains about 80 files on the old SD card). Presumably the init process was noticing this directory doesn't exist and trying to create it.

(It turns out that lib/rc/init.d is mounted as a tmpfs by rc-svcdir, which presumably is a thing that happens during boot. Its mounted contents seem similar to what's on the actual filesystem.)

• Different file sizes are causing the same problem as link numbers did.

I go back to the original ls output and fix file sizes, times and link numbers with a single filter, f -d ' +' -o '\t' 1:5,8:. This leaves 10,000 lines of diff output. At this point I decide I'm done for the day. Overnight I'll restore the backup again, this time preserving hard links, then I'll copy /lib/rc/init.d and see what I'm left with.

(f is a replacement for cut that I wrote. In this case it replaces multiple spaces with a single tab, and leaves out fields 6 and 7 which are the date and time. Symlinks and files with spaces in them aren't displayed properly, but that doesn't cause problems in the diff. cut might have been adequate in this case, I don't know.)

Actually, it seems like it should be possible to copy the filesystem directly instead of just the files on it. I'll look into that; it might be as simple as dd if=/dev/sdd1 of=/media/backup/temp to backup and dd if=/media/backup/temp of=/dev/sdd1 to restore, which would save a lot of hassle, but there might be complications I'm not aware of.

Day Two

It turns out that it's not quite as simple as that, but not much harder either. My main worry was that after copying to the new card, the filesystem would still only be 16GB (the size it was on the old card). tune2fs doesn't have an option to set size, so I decided that probably wouldn't happen. It turns out I was wrong about that, and right to be worried; but a quick google search revealed resize2fs which fixed the problem.

So after copying the filesystem, and reinstalling the MBR just to be sure (and to get the version that doesn't ask me to set a video mode), it seems to work fine.

Lesson learned: copying the contents of a filesystem, even if you preserve permissions and hard links and all that, is likely to work less well than just copying the filesystem. Also, it's sometimes better to try a completely new approach than just to keep fixing small problems in your current one; if I'd copied /lib/rc/init.d, then everything might have worked, but who knows?