Skip to content

Instantly share code, notes, and snippets.

@rsnape

rsnape/check_arr_fail.py

Created August 27, 2015 11:27
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save rsnape/bd1f30db4b789a5f7665 to your computer and use it in GitHub Desktop.
Save rsnape/bd1f30db4b789a5f7665 to your computer and use it in GitHub Desktop.
Download ZIP
Example of how the IncrementalPCA MemoryError failure can occur.
Raw
check_arr_fail.py
import numpy as np
from sklearn.utils import check_array
ut = np.memmap('D:\\my_array2.mmap', dtype=np.float16, mode='w+', shape=(140000,3504))
print repr(ut), ut.shape
res = check_array(ut)
print 'Memory check passed',repr(res),res.shape #This will print the same contents and shape as above
res = check_array(ut, dtype=np.float) # This is the actual line used in IncrementalPCA
# https://github.com/scikit-learn/scikit-learn/blob/0.16.X/sklearn/decomposition/incremental_pca.py#L165
print 'Never reaches here' # You'll get a memory error - this will never print
@abhigenie92
Copy link

Amazing spot!!! Thanks! It seems for the time being as I might use np.float64 on a machine with a larger RAM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment