Merge and extract tgz files from Google Takeout

using-google-takeout.md

Recently found some clowny gist was the top result for 'google takeout multiple tgz', where it was using two bash scripts to extract all the tgz files and then merge them together. Don't do that. Use brace expansion, cat the TGZs, and extract:

$ cat takeout-20201023T123551Z-{001..011}.tgz | tar xzivf -

You don't even need to use brace expansion. Globbing will order the files numerically:

$ cat takeout-20201023T123551Z-*.tgz | tar xzivf -

tar has been around forever, they didn't design it to need custom scripts to deal with multipart archives. Since it's extracting the combined archive, there's no 'mess of partial directories' to be merged. It just works, as intended.

An additional tip, courtesy of Dmitriy Otstavnov (@bvc3at): if you have pv available, you can track the progress of the extraction:

> pv takeout-* | tar xzif -
 190GiB 2:37:54 [18.9MiB/s] [==============>                                   ] 30% ETA 5:03:49

1. The scripts are also pointless for zip files, which have a similar one liner. At the time, zip downloads from takeout were limited to 2GB per file, versus 50GB per tgz file, so using zip was already a poor choice.
2. If someone makes a group of scripts that replicate the basic function of unix commands, badly, and people find them and use them because they're the best search result, the world becomes a dumber place. That deserves some slight ridicule. The original gist author removed the gist; only forks of it persist.

Protip for MacOS users: use gnu tar/gtar, not the built-in bsdtar. This just gave me a few days of headaches! It also appears that the built in archive utility won't correctly extract these files if they've first been cated.

Using pv instead of cat can help tracking progress of extracting archives:

> pv takeout-* | tar xzif -
 190GiB 2:37:54 [18.9MiB/s] [==============>                                   ] 30% ETA 5:03:49

This is a useful enhancement, though I note I didn't have pv installed by default in Ubuntu, so it's perhaps less portable. I'll add it regardless.

On MacOS (Mojave or Ventura+), there's a small mod:

pv takeout-* | gtar -xzif -

(if you don't have pv, then install with Homebrew: brew install pv. Also, MacOS uses bsdtar which doesn't support -i ignore zeroes, so use gnutar; installable with brew install gnu-tar)

On MacOS (Mojave or Ventura+), there's a small mod:

pv takeout-* | gtar -xzif -

(if you don't have pv, then install with Homebrew: brew install pv. Also, MacOS uses bsdtar which doesn't support -i ignore zeroes, so use gnutar; installable with brew install gnu-tar)

This worked for me seamlessly. Thank you!
(MacBook Pro M1 Max Ventura 13.0.1)

On MacOS (Mojave or Ventura+), there's a small mod:

pv takeout-* | gtar -xzif -

(if you don't have pv, then install with Homebrew: brew install pv. Also, MacOS uses bsdtar which doesn't support -i ignore zeroes, so use gnutar; installable with brew install gnu-tar)

Worked great! Thanks all!

Arrived here from the previously mentioned page... What's the best option for macOS if I went with the zip option? I went with unzip '*.zip' -d complete-takeout but it only processed two of the files because one had "fatal errors" but it didn't provide any details. I can unzip them individually without issue but I would prefer a method that combines them so I don't have to do it manually.

What's the best option for macOS if I went with the zip option?

Re-request the Takeout with tar.gz. Otherwise, Stack Overflow: https://unix.stackexchange.com/a/40565/19124