Skip to content

Instantly share code, notes, and snippets.

@ruslangrimov
Last active February 11, 2024 16:28
Show Gist options
  • Save ruslangrimov/0e922a96f4af5531fab7ba078a32ba32 to your computer and use it in GitHub Desktop.
Save ruslangrimov/0e922a96f4af5531fab7ba078a32ba32 to your computer and use it in GitHub Desktop.
Extract tar into memory, remove the original tar file and save files to the same disk if there is no space for usual extraction
import os
import tarfile
from tqdm.auto import tqdm
tar_file = "/data/images.tar.gz"
extract_to = "/data/"
files = {}
with tarfile.open(tar_file, "r:gz") as file:
for member in tqdm(file):
if member.isfile():
files[member.name] = file.extractfile(member.name).read()
os.unlink(tar_file)
for name, cnt in tqdm(files.items()):
f_name = os.path.join(extract_to, name)
os.makedirs(os.path.dirname(f_name), exist_ok=True)
with open(f_name, 'wb') as f:
f.write(cnt)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment