Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
script for ImageNet data extract.
#!/bin/bash
#
# script to extract ImageNet dataset
# ILSVRC2012_img_train.tar (about 138 GB)
# ILSVRC2012_img_val.tar (about 6.3 GB)
# make sure ILSVRC2012_img_train.tar & ILSVRC2012_img_val.tar in your current directory
#
# https://github.com/facebook/fb.resnet.torch/blob/master/INSTALL.md
#
# train/
# ├── n01440764
# │ ├── n01440764_10026.JPEG
# │ ├── n01440764_10027.JPEG
# │ ├── ......
# ├── ......
# val/
# ├── n01440764
# │ ├── ILSVRC2012_val_00000293.JPEG
# │ ├── ILSVRC2012_val_00002138.JPEG
# │ ├── ......
# ├── ......
#
#
# Extract the training data:
#
mkdir train && mv ILSVRC2012_img_train.tar train/ && cd train
tar -xvf ILSVRC2012_img_train.tar && rm -f ILSVRC2012_img_train.tar
find . -name "*.tar" | while read NAME ; do mkdir -p "${NAME%.tar}"; tar -xvf "${NAME}" -C "${NAME%.tar}"; rm -f "${NAME}"; done
cd ..
#
# Extract the validation data and move images to subfolders:
#
mkdir val && mv ILSVRC2012_img_val.tar val/ && cd val && tar -xvf ILSVRC2012_img_val.tar
wget -qO- https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh | bash
#
# Check total files after extract
#
# $ find train/ -name "*.JPEG" | wc -l
# 1281167
# $ find val/ -name "*.JPEG" | wc -l
# 50000
#
@jimmykimmy68
Copy link

jimmykimmy68 commented Mar 30, 2021

Really appreciate it

@AjinkyaBankar
Copy link

AjinkyaBankar commented May 5, 2021

Thank you.

@RonDen
Copy link

RonDen commented May 7, 2021

This is Great

@developer0hye
Copy link

developer0hye commented May 13, 2021

Thanks!

@ninfueng
Copy link

ninfueng commented May 13, 2021

Thank you very much, this saves me a lot of time.

@superli3
Copy link

superli3 commented Jul 6, 2021

Thank you!

@AdityaKane2001
Copy link

AdityaKane2001 commented Jul 11, 2021

Truly a lifesaver!

@chongzhou96
Copy link

chongzhou96 commented Jul 26, 2021

Thanks a lot!

@immrz
Copy link

immrz commented Aug 2, 2021

Many thanks!

@ChengHan111
Copy link

ChengHan111 commented Dec 19, 2021

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment