gistfile1.txt

In term one: 

core@localhost ~ $ sudo /usr/bin/systemd-nspawn --link-journal=no --private-network -D /mnt/stateful_partition/var_overlay/lib/docker/graph/e9aa60c60128cad1/layer/ /bin/sh -c 'I=0; while [ 1 ] ; do I=$(($I+1)); echo foo $I; sleep 1; done'
Spawning namespace container on /mnt/stateful_partition/var_overlay/lib/docker/graph/e9aa60c60128cad1/layer (console is /dev/pts/2).
Init process in the container running as PID 7531.
/etc/localtime does not point into /usr/share/zoneinfo/, not updating container timezone.
foo 1
foo 2
...


In other terminal, note PID "7531": 

Problem one,  boot_id mount stuff:

localhost ~ # criu dump -D /tmp/criu/ -t 7531
(00.000054) Error (mem.c:67): Can't reset 7638's dirty memory tracker: Invalid argument
(00.001934) Error (kerndat.c:107): Dirty tracking support is OFF
(00.006727) Error (mem.c:338): Can't open /proc/7531/pagemap2: No such file or directory
(00.008998) Error (mem.c:338): Can't open /proc/7637/pagemap2: No such file or directory
(00.011752) Error (mount.c:236): Can't open /proc/sys/kernel/random/boot_id: Not a directory
(00.012786) Error (namespaces.c:437): Namespaces dumping finished with error 65280
(00.013221) Error (cr-dump.c:1731): Dumping FAILED.

WTF, OK fix with nsenter:

localhost ~ # nsenter -m -p -t 7531 /bin/sh


BusyBox v1.19.3 (Ubuntu 1:1.19.3-7ubuntu1.1) built-in shell (ash)
Enter 'help' for a list of built-in commands.

/ # mount
rootfs on / type rootfs (rw)
/dev/root on / type ext2 (ro,relatime)
devtmpfs on /dev type devtmpfs (rw,relat
...[snip]...
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /proc/kmsg (deleted) type tmpfs (rw,nosuid,mode=755)
tmpfs on /proc/sys/kernel/random/boot_id (deleted) type tmpfs (ro,relatime,mode=755)
/ # umount /proc/sys/kernel/random/boot_id
/ # umount /proc/kmsg

No idea what that was about... but ... OK, try again:

localhost ~ # criu dump -D /tmp/criu/ -t 7531
(00.000126) Error (mem.c:67): Can't reset 7800's dirty memory tracker: Invalid argument
(00.002190) Error (kerndat.c:107): Dirty tracking support is OFF
(00.013801) Error (mem.c:338): Can't open /proc/7531/pagemap2: No such file or directory
(00.015239) Error (mem.c:338): Can't open /proc/7799/pagemap2: No such file or directory
Command "save" is unknown, try "ip addr help".
(00.033319) Error (util.c:576): exited, status=255
(00.033560) Error (net.c:310): IP tool failed on addr save
(00.034118) Error (namespaces.c:437): Namespaces dumping finished with error 65280
(00.034258) Error (cr-dump.c:1731): Dumping FAILED.

Aw F, need a new version of IP tool. I copied mine from the SDK, but you could build a whole new image:

localhost ~ # export CR_IP_TOOL=/mnt/stateful_partition/ip  # copied binary from SDK
localhost ~ # criu dump -D /tmp/criu/ -t 7531
(00.000017) Error (mem.c:67): Can't reset 7864's dirty memory tracker: Invalid argument
(00.000568) Error (kerndat.c:107): Dirty tracking support is OFF
(00.005633) Error (mem.c:338): Can't open /proc/7531/pagemap2: No such file or directory
(00.015587) Error (mem.c:338): Can't open /proc/7863/pagemap2: No such file or directory
(00.030875) Error (tty.c:177): tty: Found slave peer index 2 without correspond master peer
(00.031757) Error (cr-dump.c:1731): Dumping FAILED.

... now no idea

localhost ~ # criu restore -D /tmp/criu/ -r /mnt/stateful_partition/var_overlay/lib/docker/graph/e9aa60c60128cad1/layer/ -n net -n mnt -n ipc -n pid --shell-job --evasive-devices
RTNETLINK answers: File exists
RTNETLINK answers: File exists
(00.004952) 1: Error (mount.c:654): pivot_root(., crtools-put-root.2ioikG) failed: Invalid argument
(00.007512) Error (cr-restore.c:967): 8433 exited, status=1
(00.007668) Error (cr-restore.c:1322): Restoring FAILED.

pivot_root does not work if the new_root is on a different filesystem than the original (my case). You have to bind mount the directory to your root filesystem, then try a restore. After all of that...

localhost ~ # criu restore -r /usr/lib/lxc/rootfs --shell-job --evasive-devices --file-locks -D /tmp/criu2/ RTNETLINK answers: File exists
RTNETLINK answers: File exists
(00.012194) 1: Error (cr-restore.c:960): BUG at cr-restore.c:960
(00.017906) Error (cr-restore.c:969): 12040 killed by signal 11
(00.017984) Error (cr-restore.c:1322): Restoring FAILED.