Janelia inference, Workstation ANNA, linking Python to CUDA 10 installation

2022-03-10-janeliainference-workstationANNA-linkingPythonWithCUDA10.md

There's some kind of problem linking python to the existing CUDA 10 installation (CUDA 10.0 lives at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\)

(base) C:\Users\CryoEM>conda activate CNNectome

(CNNectome) C:\Users\CryoEM>ipython
Python 3.6.12 |Anaconda, Inc.| (default, Sep  9 2020, 00:29:25) [MSC v.1916 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import tensorflow as tf
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\framework\dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\framework\dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\framework\dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\framework\dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\framework\dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\framework\dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])

In [2]: tf.test.gpu_device_name()
2022-03-10 15:55:42.605511: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2022-03-10 15:55:42.627295: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library nvcuda.dll
2022-03-10 15:55:43.084063: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: NVIDIA RTX A6000 major: 8 minor: 6 memoryClockRate(GHz): 1.8
pciBusID: 0000:18:00.0
2022-03-10 15:55:43.084412: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: NVIDIA RTX A6000 major: 8 minor: 6 memoryClockRate(GHz): 1.8
pciBusID: 0000:3b:00.0
2022-03-10 15:55:43.088422: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2022-03-10 15:55:43.089353: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1

In comparison, the ZeroCostDL4Mic-3dUNet env can easily find CUDA 11.6 (can't figure out where this version of CUDA lives on Windows)

(base) C:\Users\CryoEM>conda activate ZeroCostDL4Mic-3dUNet

(ZeroCostDL4Mic-3dUNet) C:\Users\CryoEM>ipython
Python 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:22:46) [MSC v.1916 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.1.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import tensorflow as tf

In [2]: tf.test.gpu_device_name()
2022-03-10 15:56:20.877162: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-03-10 15:56:25.162106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /device:GPU:0 with 45892 MB memory:  -> device: 0, name: NVIDIA RTX A6000, pci bus id: 0000:18:00.0, compute capability: 8.6
2022-03-10 15:56:25.163922: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /device:GPU:1 with 45892 MB memory:  -> device: 1, name: NVIDIA RTX A6000, pci bus id: 0000:3b:00.0, compute capability: 8.6
Out[2]: '/device:GPU:0'

In [3]:

...but give it a few minutes, and it does actually find the GPU. See here:

(base) C:\Users\CryoEM>conda activate CNNectome

(CNNectome) C:\Users\CryoEM>ipython
Python 3.6.12 |Anaconda, Inc.| (default, Sep  9 2020, 00:29:25) [MSC v.1916 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import tensorflow as tf
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\framework\dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\framework\dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\framework\dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\framework\dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\framework\dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\framework\dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])

In [2]: tf.test.gpu_device_name()
2022-03-10 15:55:42.605511: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2022-03-10 15:55:42.627295: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library nvcuda.dll
2022-03-10 15:55:43.084063: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: NVIDIA RTX A6000 major: 8 minor: 6 memoryClockRate(GHz): 1.8
pciBusID: 0000:18:00.0
2022-03-10 15:55:43.084412: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: NVIDIA RTX A6000 major: 8 minor: 6 memoryClockRate(GHz): 1.8
pciBusID: 0000:3b:00.0
2022-03-10 15:55:43.088422: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2022-03-10 15:55:43.089353: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1
2022-03-10 16:01:11.931881: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-03-10 16:01:11.932292: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1
2022-03-10 16:01:11.938031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y
2022-03-10 16:01:11.938075: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N
2022-03-10 16:01:11.940104: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:0 with 45083 MB memory) -> physical GPU (device: 0, name: NVIDIA RTX A6000, pci bus id: 0000:18:00.0, compute capability: 8.6)
2022-03-10 16:01:11.942985: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:1 with 45083 MB memory) -> physical GPU (device: 1, name: NVIDIA RTX A6000, pci bus id: 0000:3b:00.0, compute capability: 8.6)
Out[2]: '/device:GPU:0'

In [3]:

In [3]: exit

(CNNectome) C:\Users\CryoEM>

The current CNNectome environment won't train. I suspect we'll need to build/recompile tensorflow version 1.14.0 ourselves, instead of installing it from pip.

Internal: Blas GEMM launch failed

(base) C:\Users\CryoEM>conda activate CNNectome

(CNNectome) C:\Users\CryoEM>conda list | grep tensor
tensorboard               1.14.0                   pypi_0    pypi
tensorflow-estimator      1.14.0                   pypi_0    pypi
tensorflow-gpu            1.14.0                   pypi_0    pypi

(CNNectome) C:\Users\CryoEM>conda list | grep cud
cudatoolkit               10.1.243             h74a9793_0
cudnn                     7.6.5                cuda10.1_0

(CNNectome) C:\Users\CryoEM>ipython
Python 3.6.12 |Anaconda, Inc.| (default, Sep  9 2020, 00:29:25) [MSC v.1916 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import tensorflow as tf
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\framework\dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\framework\dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\framework\dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\framework\dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\framework\dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\framework\dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])

In [2]: tf.test.is_gpu_available()
2022-03-10 16:26:22.235280: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2022-03-10 16:26:22.257470: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library nvcuda.dll
2022-03-10 16:26:22.698875: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: NVIDIA RTX A6000 major: 8 minor: 6 memoryClockRate(GHz): 1.8
pciBusID: 0000:18:00.0
2022-03-10 16:26:22.699086: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: NVIDIA RTX A6000 major: 8 minor: 6 memoryClockRate(GHz): 1.8
pciBusID: 0000:3b:00.0
2022-03-10 16:26:22.700538: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2022-03-10 16:26:22.700632: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1
2022-03-10 16:28:14.497092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-03-10 16:29:38.334608: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1
2022-03-10 16:29:38.338420: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y
2022-03-10 16:29:38.338485: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N
2022-03-10 16:29:38.338843: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:0 with 44808 MB memory) -> physical GPU (device: 0, name: NVIDIA RTX A6000, pci bus id: 0000:18:00.0, compute capability: 8.6)
2022-03-10 16:29:38.342738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:1 with 45083 MB memory) -> physical GPU (device: 1, name: NVIDIA RTX A6000, pci bus id: 0000:3b:00.0, compute capability: 8.6)
Out[2]: True

In [3]: tf.test.gpu_device_name()
2022-03-10 16:29:51.467645: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: NVIDIA RTX A6000 major: 8 minor: 6 memoryClockRate(GHz): 1.8
pciBusID: 0000:18:00.0
2022-03-10 16:29:51.467815: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: NVIDIA RTX A6000 major: 8 minor: 6 memoryClockRate(GHz): 1.8
pciBusID: 0000:3b:00.0
2022-03-10 16:29:51.468414: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2022-03-10 16:29:51.468944: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1
2022-03-10 16:29:51.469527: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-03-10 16:29:51.470023: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1
2022-03-10 16:29:51.470515: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y
2022-03-10 16:29:51.471012: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N
2022-03-10 16:29:51.471666: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:0 with 44808 MB memory) -> physical GPU (device: 0, name: NVIDIA RTX A6000, pci bus id: 0000:18:00.0, compute capability: 8.6)
2022-03-10 16:29:51.472065: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:1 with 45083 MB memory) -> physical GPU (device: 1, name: NVIDIA RTX A6000, pci bus id: 0000:3b:00.0, compute capability: 8.6)
Out[3]: '/device:GPU:0'

In [4]: import tensorflow as tf
   ...: mnist = tf.keras.datasets.mnist
   ...:
   ...: (x_train, y_train),(x_test, y_test) = mnist.load_data()
   ...: x_train, x_test = x_train / 255.0, x_test / 255.0
   ...:
   ...: model = tf.keras.models.Sequential([
   ...:   tf.keras.layers.Flatten(input_shape=(28, 28)),
   ...:   tf.keras.layers.Dense(128, activation='relu'),
   ...:   tf.keras.layers.Dropout(0.2),
   ...:   tf.keras.layers.Dense(10, activation='softmax')
   ...: ])
   ...:
   ...: model.compile(optimizer='adam',
   ...:               loss='sparse_categorical_crossentropy',
   ...:               metrics=['accuracy'])
   ...:
   ...: model.fit(x_train, y_train, epochs=5)
   ...: model.evaluate(x_test, y_test)
   ...:
WARNING:tensorflow:From C:\Users\CryoEM\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\ops\init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
2022-03-10 16:30:09.487046: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: NVIDIA RTX A6000 major: 8 minor: 6 memoryClockRate(GHz): 1.8
pciBusID: 0000:18:00.0
2022-03-10 16:30:09.487261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: NVIDIA RTX A6000 major: 8 minor: 6 memoryClockRate(GHz): 1.8
pciBusID: 0000:3b:00.0
2022-03-10 16:30:09.488765: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2022-03-10 16:30:09.488851: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1
2022-03-10 16:30:09.489582: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-03-10 16:30:09.489992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1
2022-03-10 16:30:09.490031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y
2022-03-10 16:30:09.490602: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N
2022-03-10 16:30:09.491233: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 44808 MB memory) -> physical GPU (device: 0, name: NVIDIA RTX A6000, pci bus id: 0000:18:00.0, compute capability: 8.6)
2022-03-10 16:30:09.491661: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 45083 MB memory) -> physical GPU (device: 1, name: NVIDIA RTX A6000, pci bus id: 0000:3b:00.0, compute capability: 8.6)
Epoch 1/5
2022-03-10 16:31:38.590648: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED
---------------------------------------------------------------------------
InternalError                             Traceback (most recent call last)
<ipython-input-4-357d0cd1907d> in <module>
     16               metrics=['accuracy'])
     17
---> 18 model.fit(x_train, y_train, epochs=5)
     19 model.evaluate(x_test, y_test)

~\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
    778           validation_steps=validation_steps,
    779           validation_freq=validation_freq,
--> 780           steps_name='steps_per_epoch')
    781
    782   def evaluate(self,

~\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in model_iteration(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps, validation_freq, mode, validation_in_fit, prepared_feed_values_from_dataset, steps_name, **kwargs)
    361
    362         # Get outputs.
--> 363         batch_outs = f(ins_batch)
    364         if not isinstance(batch_outs, list):
    365           batch_outs = [batch_outs]

~\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs)
   3290
   3291     fetched = self._callable_fn(*array_vals,
-> 3292                                 run_metadata=self.run_metadata)
   3293     self._call_fetch_callbacks(fetched[-len(self._fetches):])
   3294     output_structure = nest.pack_sequence_as(

~\.conda\envs\CNNectome\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs)
   1456         ret = tf_session.TF_SessionRunCallable(self._session._session,
   1457                                                self._handle, args,
-> 1458                                                run_metadata_ptr)
   1459         if run_metadata:
   1460           proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

InternalError: 2 root error(s) found.
  (0) Internal: Blas GEMM launch failed : a.shape=(32, 128), b.shape=(128, 10), m=32, n=10, k=128
         [[{{node dense_1/MatMul}}]]
         [[loss/mul/_59]]
  (1) Internal: Blas GEMM launch failed : a.shape=(32, 128), b.shape=(128, 10), m=32, n=10, k=128
         [[{{node dense_1/MatMul}}]]
0 successful operations.
0 derived errors ignored.

In [5]:

In [5]: exit

(CNNectome) C:\Users\CryoEM>

I tried installing tensorflow-gpu=1.14.0 from conda (using the -c anaconda channel), but found the same problem.
... We really will have to try building Tensorflow from source. Instructions are here, it's a fairly involved process: https://www.tensorflow.org/install/source_windows