gistfile1.txt

Partition of a set of 392998 objects. Total size = 64348264 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0 274344  70 23091456  36  23091456  36 str
     1  24662   6 22809712  35  45901168  71 dict (no owner)
     2  33974   9  3459328   5  49360496  77 tuple
     3   9802   2  2176232   3  51536728  80 unicode
     4  12987   3  2069000   3  53605728  83 list
     5   9847   3  1339192   2  54944920  85 function
     6    556   0  1042400   2  55987320  87 dict of module
     7   6622   2   953568   1  56940888  88 types.CodeType
     8   1031   0   951184   1  57892072  90 type
     9   1031   0   825112   1  58717184  91 dict of type
<679 more rows. Type e.g. '_.more' to view.>

gistfile2.txt

# After adding adding python -OO
Partition of a set of 390903 objects. Total size = 63186064 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  24661   6 22809416  36  22809416  36 dict (no owner)
     1 273299  70 22672472  36  45481888  72 str
     2  33962   9  3442848   5  48924736  77 tuple
     3  12985   3  2068888   3  50993624  81 list
     4   8784   2  1451080   2  52444704  83 unicode
     5   9847   3  1339192   2  53783896  85 function
     6    556   0  1042400   2  54826296  87 dict of module
     7   6622   2   953568   2  55779864  88 types.CodeType
     8   1031   0   951184   2  56731048  90 type
     9   1031   0   825496   1  57556544  91 dict of type

Making exercises.json database-backed had an underwhelming improvement in memory usage and significantly slowed down the exercise caching . Next steps:

1. Try to refactor get_exercise_cache to use multiple worker threads;
2. Process khan exercises and perseus exercises differently, and in batches
3. Try to avoid using too much I/O (or bunching up all the IO together).

Retain get_exercise_data API!