(parallel) gzip qith a tqdm progress bar

demo.md

gzip with tqdm progress bar

Prereq

pip install tqdm

Minimalistic

cat "${FILE}" | tqdm --bytes | gzip -c - > "${FILE}.gz"

Output sample:

22.6MB [00:03, 7.74MB/s]

Fancy bar

In order to have a progress bar with completeness, throughput and ETA, you need to provide the total size --total=XX. As the output size of the gzipped file is not known, I rely on the input size.

cat "${FILE}"  \
  | tqdm --total=$(stat --printf="%s" "${FILE}") --bytes  \
  | gzip -c -  \
  > "${FILE}.gz"

Output sample:

 22%|██████████████▎                                                  | 168M/760M [00:00<00:01, 367MB/s]
 56%|████████████████████████████████████▍                            | 426M/760M [00:01<00:00, 379MB/s]
 80%|███████████████████████████████████████████████████▊             | 606M/760M [00:01<00:00, 377MB/s]
100%|█████████████████████████████████████████████████████████████████| 760M/760M [00:02<00:00, 372MB/s]

Wrap as a bash function

Autopromo autopromo

function gzipbar {
  cat "${1}"  \
    | tqdm --total=$(stat --printf="%s" "${1}") --bytes  \
    | gzip -c -  \
    > "${1}.gz"
}

Notes

1. You may use your favorite compression software. Briefly tested with

   • gzip -c -
   • bzip2 -c -
   • xz -c -
   • python -m pgzip -t 32 - (some delays are observed)

2. You may custom the tqdm progressbar with

   • --desc
   • --mininterval
   • --maxinterval

   See tqdm -h for more.

3. You may also be interrestd in pv for progress reporting.