Profile the memory usage of your Python scripts easily

Profiling memory usage in Python.md

This is a easy example of how to profiling memory usage by memory_profiler library in 5 steps:

1.) Install memory_profiler libraries

pip install -U memory_profiler matplotlib

2.) Insert @profile decorator in your script

Example of script to measure memory usage using @profile decorator

from memory_profiler import profile
from datetime import datetime
from time import sleep

# implementation: just import and decorate your function

@profile
def your_function():
    print('-> (sleeping) 1 second...')
    sleep(1)

    print('-> (creating) first list...')
    first_list = [1] * (3 * 10 ** 7)  # 30000000 elements (using 228.9 MiB of RAM memory)
    print(f'(elements created) in first list: {len(first_list)}')
    
    print('-> (sleeping) 1 second...')
    sleep(1)
    
    print('-> (creating) second list...')
    second_list = [2] * (7 * 10 ** 7)  # 70000000 elements (using 533.9 MiB of RAM memory)
    print(f'(elements created) in second list: {len(second_list)}')
    
    print('-> (sleeping) 1 second...')
    sleep(1)
    
    print('-> (removing) first list...')
    del first_list
    
    print('-> (sleeping) 1 second...')
    sleep(1)
    
    print('-> (removing) second list')
    del second_list
    
    print('-> (sleeping) 1 second...')
    sleep(1)
    
    print('-> (finishing) the script...')
    
if __name__ == '__main__':
    # get time of start
    start = datetime.now()
    
    # execute 
    your_function()
    
    # show how much time was taken
    time_took = datetime.now() - start
    print(f'Time took: {time_took}')

Step 3.) Execute your script with memory_profiler runner

Using the example above as script: profiling-memory-usage-in-python.py just execute:

mprof run profiling-memory-usage-in-python.py

mprof: Sampling memory every 0.1s
running new process
running as a Python program...

-> (sleeping) 1 second...
-> (creating) first list...
(elements created) in first list: 30000000
-> (sleeping) 1 second...
-> (creating) second list...
(elements created) in second list: 70000000
-> (sleeping) 1 second...
-> (removing) first list...
-> (sleeping) 1 second...
-> (removing) second list
-> (sleeping) 1 second...
-> (finishing) the script...
Filename: profiling-memory-usage-in-python.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     7     43.5 MiB     43.5 MiB           1   @profile
     8                                         def your_function():
     9     43.5 MiB      0.0 MiB           1       print('-> (sleeping) 1 second...')
    10     43.5 MiB      0.0 MiB           1       sleep(1)
    11                                         
    12     43.5 MiB      0.0 MiB           1       print('-> (creating) first list...')
    13    272.4 MiB    228.9 MiB           1       first_list = [1] * (3 * 10 ** 7)  # 30000000 elements (using 228.9 MiB of RAM memory)
    14    272.4 MiB      0.0 MiB           1       print(f'(elements created) in first list: {len(first_list)}')
    15                                             
    16    272.4 MiB      0.0 MiB           1       print('-> (sleeping) 1 second...')
    17    272.4 MiB      0.0 MiB           1       sleep(1)
    18                                             
    19    272.4 MiB      0.0 MiB           1       print('-> (creating) second list...')
    20    806.4 MiB    534.0 MiB           1       second_list = [2] * (7 * 10 ** 7)  # 70000000 elements (using 533.9 MiB of RAM memory)
    21    806.4 MiB      0.0 MiB           1       print(f'(elements created) in second list: {len(second_list)}')
    22                                             
    23    806.4 MiB      0.0 MiB           1       print('-> (sleeping) 1 second...')
    24    806.4 MiB      0.0 MiB           1       sleep(1)
    25                                             
    26    806.4 MiB      0.0 MiB           1       print('-> (removing) first list...')
    27    577.6 MiB   -228.8 MiB           1       del first_list
    28                                             
    29    577.6 MiB      0.0 MiB           1       print('-> (sleeping) 1 second...')
    30    577.6 MiB      0.0 MiB           1       sleep(1)
    31                                             
    32    577.6 MiB      0.0 MiB           1       print('-> (removing) second list')
    33     43.5 MiB   -534.1 MiB           1       del second_list
    34                                             
    35     43.5 MiB      0.0 MiB           1       print('-> (sleeping) 1 second...')
    36     43.5 MiB      0.0 MiB           1       sleep(1)
    37                                             
    38     43.5 MiB      0.0 MiB           1       print('-> (finishing) the script...')


Time took: 0:00:05.400252

4.) Open created chart by mprof tool

mprof plot

5.) Analyse the execution:

• The memory usage taken by python script
• The time spent executing

References:

• https://github.com/pythonprofilers/memory_profiler
• https://stackoverflow.com/questions/67650841/how-to-find-memory-usage-with-memory-profiler-python

profiling-memory-usage-in-python.py

from memory_profiler import profile
from datetime import datetime
from time import sleep

# implementation: just import and decorate your function

@profile
def your_function():
    print('-> (sleeping) 1 second...')
    sleep(1)

    print('-> (creating) first list...')
    first_list = [1] * (3 * 10 ** 7)  # 30000000 elements (using 228.9 MiB of RAM memory)
    print(f'(elements created) in first list: {len(first_list)}')
    
    print('-> (sleeping) 1 second...')
    sleep(1)
    
    print('-> (creating) second list...')
    second_list = [2] * (7 * 10 ** 7)  # 70000000 elements (using 533.9 MiB of RAM memory)
    print(f'(elements created) in second list: {len(second_list)}')
    
    print('-> (sleeping) 1 second...')
    sleep(1)
    
    print('-> (removing) first list...')
    del first_list
    
    print('-> (sleeping) 1 second...')
    sleep(1)
    
    print('-> (removing) second list')
    del second_list
    
    print('-> (sleeping) 1 second...')
    sleep(1)
    
    print('-> (finishing) the script...')
    
if __name__ == '__main__':
    # get time of start
    start = datetime.now()
    
    # execute 
    your_function()
    
    # show how much time was taken
    time_took = datetime.now() - start
    print(f'Time took: {time_took}')