Fuchai
/ train_valid_split.py
Last active Apr 10, 2020
This is a pytorch generic function that takes a data.Dataset object and splits it to validation and training efficiently.
View train_valid_split.py
| import np | |
| from torch.utils.data import Dataset | |
| class GenHelper(Dataset): | |
| def __init__(self, mother, length, mapping): | |
| # here is a mapping from this index to the mother ds index | |
| self.mapping=mapping | |
| self.length=length | |
| self.mother=mother |
soumith
/ gist:01da3874bf014d8a8c53406c2b95d56b
Last active Feb 26, 2021
Install PillowSIMD+libjpeg-turbo on Conda
View gist:01da3874bf014d8a8c53406c2b95d56b
| conda uninstall --force pillow -y | |
| # install libjpeg-turbo to $HOME/turbojpeg | |
| git clone https://github.com/libjpeg-turbo/libjpeg-turbo | |
| pushd libjpeg-turbo | |
| mkdir build | |
| cd build | |
| cmake .. -DCMAKE_INSTALL_PREFIX:PATH=$HOME/turbojpeg | |
| make | |
| make install |
kevinzakka
/ data_loader.py
Last active Mar 2, 2021
Train, Validation and Test Split for torchvision Datasets
View data_loader.py
| """ | |
| Create train, valid, test iterators for CIFAR-10 [1]. | |
| Easily extended to MNIST, CIFAR-100 and Imagenet. | |
| [1]: https://discuss.pytorch.org/t/feedback-on-pytorch-for-kaggle-competitions/2252/4 | |
| """ | |
| import torch | |
| import numpy as np |