nicMSlesions MemoryError

MemoryError.log

 WM lesion segmentation                      #
#                                                #
# -------------------------------                #
# (c) Sergi Valverde 2019                        #
# Neuroimage Computing Group                     #
# -------------------------------                #
##################################################

DEBUG:  1
Using TensorFlow backend.
WARNING:tensorflow:From /localhome/jstutters/.virtualenvs/nicmslesionspy2/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:349: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /localhome/jstutters/.virtualenvs/nicmslesionspy2/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:3147: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From /localhome/jstutters/.virtualenvs/nicmslesionspy2/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:99: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

WARNING:tensorflow:From /localhome/jstutters/.virtualenvs/nicmslesionspy2/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:1062: calling reduce_prod_v1 (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From /localhome/jstutters/.virtualenvs/nicmslesionspy2/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:2683: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
WARNING:tensorflow:From /localhome/jstutters/.virtualenvs/nicmslesionspy2/lib/python2.7/site-packages/keras/optimizers.py:675: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /localhome/jstutters/.virtualenvs/nicmslesionspy2/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:2550: calling reduce_sum_v1 (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From /localhome/jstutters/.virtualenvs/nicmslesionspy2/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:2554: The name tf.log is deprecated. Please use tf.math.log instead.

> CNN: loading weights from datatraining configuration
/localhome/jstutters/projects/nicmslesions/nicMSlesions/nets/datatraining/nets/model_1.hdf5
/localhome/jstutters/projects/nicmslesions/nicMSlesions/nets/datatraining/nets/model_2.hdf5
WARNING:tensorflow:From /localhome/jstutters/.virtualenvs/nicmslesionspy2/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:140: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

2019-09-30 14:02:09.128806: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-09-30 14:02:09.141781: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-30 14:02:09.154301: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2019-09-30 14:02:09.154750: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: euler.nmr.ion.ucl.ac.uk
2019-09-30 14:02:09.154787: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: euler.nmr.ion.ucl.ac.uk
2019-09-30 14:02:09.154892: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 435.21.0
2019-09-30 14:02:09.154989: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 435.21.0
2019-09-30 14:02:09.155018: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 435.21.0
2019-09-30 14:02:09.164181: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2394515000 Hz
2019-09-30 14:02:09.166758: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x67dac70 executing computations on platform Host. Devices:
2019-09-30 14:02:09.166814: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-09-30 14:02:09.357367: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
> PRE: 10071 identifying input modalities
> INFO: 10071 elapsed time:  2.0 sec
> PRE: 10071 registering FLAIR  --> T1 space
> PRE: 10071 registering MOD3  --> T1 space
> PRE: 10071 registering MOD4  --> T1 space
> INFO: 10071 elapsed time:  334.0 sec
/localhome/jstutters/projects/nicmslesions/nicMSlesions/libs/CNN/base.py:515: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  patches = [new_image[idx] for idx in slices]
Traceback (most recent call last):
  File "nic_infer_segmentation_batch.py", line 149, in <module>
    out_seg = test_cascaded_model(model, test_x_data, options)
  File "/localhome/jstutters/projects/nicmslesions/nicMSlesions/libs/CNN/base.py", line 157, in test_cascaded_model
    save_nifti=save_nifti)
  File "/localhome/jstutters/projects/nicmslesions/nicMSlesions/libs/CNN/base.py", line 557, in test_scan
    candidate_mask)
  File "/localhome/jstutters/projects/nicmslesions/nicMSlesions/libs/CNN/base.py", line 472, in load_test_patches
    Xs = np.stack(X, axis=1)
  File "/localhome/jstutters/.virtualenvs/nicmslesionspy2/lib/python2.7/site-packages/numpy/core/shape_base.py", line 423, in stack
    return _nx.concatenate(expanded_arrays, axis=axis, out=out)
MemoryError