Skip to content

Instantly share code, notes, and snippets.

@mrm8488
Forked from kabirahuja2431/iterable_dataset_v0.py
Created February 23, 2020 03:20
Show Gist options
  • Save mrm8488/65ce7e58cf8ee8b3dc41342692c028e5 to your computer and use it in GitHub Desktop.
Save mrm8488/65ce7e58cf8ee8b3dc41342692c028e5 to your computer and use it in GitHub Desktop.
from torch.utils.data import IterableDataset
class CustomIterableDataset(IterableDataset):
def __init__(self, filename):
#Store the filename in object's memory
self.filename = filename
#And that's it, we no longer need to store the contents in the memory
def __iter__(self):
#Create an iterator
file_itr = open(self.filename)
return file_itr
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment