Skip to content

Instantly share code, notes, and snippets.

@pbloem
Created May 29, 2022 12:29
Show Gist options
  • Save pbloem/78ba9ef9a5f42d5688fb5bc1092aa1f4 to your computer and use it in GitHub Desktop.
Save pbloem/78ba9ef9a5f42d5688fb5bc1092aa1f4 to your computer and use it in GitHub Desktop.
Pytorch: use an IterableDataset to create an infinite stream of batched data
from torch.utils.data import IterableDataset, DataLoader
# The dataset has no length
class TestDataset(IterableDataset):
def __iter__(self): # __iter__() can be a generator
while True:
yield 5
# The loader is called as normal
loader = DataLoader(TestDataset(), batch_size=3)
for b in loader:
print(b)
# Result: repeats [5, 5, 5] forever
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment