A simple example of how to use generators in Python 3. Explanations on when it's a good idea to use generators.

Python 3 generator example.md

In Python, generators are helper objects for managing long sequences of data. When list or tuple is created, it loads all its items into memory at once. A generator, by contrast, only loads and returns 1 item at a time.

In this example, we'll access a list of common Dutch surnames via a generator!

Side note: The for x in y construct actually makes a generator of the iterable y.

dutch_surnames.txt

De Jong
Jansen
De Vries
Van den Berg
Van Dijk
Bakker
Janssen
Visser
Smit
Meijer
De Boer
Mulder
De Groot
Bos
Vos
Peters
Hendriks
Van Leeuwen
Dekker
Brouwer
De Wit
Dijkstra
Smits
De Graaf
Van der Meer

generator_example.py

"""Let's try to access all the names in the text file "dutch_surnames.txt".
Pretend we wanted to do some analysis or processing on each name.
First we'll load all names into a list at once which consumes a lot of memory if the list is long.
Then we'll construct a generator that gives a one name at a time.
With the generator it takes less time until the actual processing can start (since only 1 name is fetched at a time).
In addition less memory is used and
there's no unnecessary loading in the case we abort early (because we found what we were looking for)."""

# First let's load all the names at once
with open('dutch_surnames.txt', 'r') as text_file:
    # names = [line.rsplit() for line in text_file.readlines()]
    names = [line.rstrip() for line in text_file.readlines()]

# Let's print the length of each name
print("Names and their length:")
for name in names:
    print(name, len(name))


# Now we'll use a generator which gives us 1 name at a time,
# instead of loading them all at once.
def give_next_name(names_file):
    with open(names_file, 'r') as names:
        while True:
            name = names.readline()
            if name:
                yield name.rstrip()
            else:
                break


# For processing, let's only print those name that contain a prefix such as De or Van
print('------------------------------------')
print("Only names with prefixes:")
for name in give_next_name('dutch_surnames.txt'):
    if ' ' in name:
        print(name)

# Imagine our list would contain millions of names!
# Then using a generator would certainly be faster and save memory too!