numpy_avx_detection.md

Numpy Blas and AVX detection

Comment to https://twitter.com/honnibal/status/918427934700032000

Numpy with MKL

For numpy with MKL installed by conda the proposed solution doesn't appear to work,

conda create -n fd-env numpy python=3.6
python

yields,

Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:51:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import numpy.core.multiarray, subprocess
>>> sh = lambda cmd: subprocess.check_output(cmd, shell=True)
>>> linked_blas = sh(f'ldd {numpy.core.multiarray.__file__} | grep blas')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <lambda>
  File "/home/rth/.miniconda3/envs/mark-crater-env/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/home/rth/.miniconda3/envs/mark-crater-env/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'ldd /home/rth/.miniconda3/envs/mark-crater-env/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so | grep blas' returned non-zero exit status 1.

indeed,

ldd /home/rth/.miniconda3/envs/fd-env/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so   
        linux-vdso.so.1 (0x00007ffd87fd5000)
        libmkl_intel_lp64.so => /home/rth/.miniconda3/envs/fd-env/lib/python3.6/site-packages/numpy/core/../../../../libmkl_intel_lp64.so (0x00007f5c41327000)
        libmkl_intel_thread.so => /home/rth/.miniconda3/envs/fd-env/lib/python3.6/site-packages/numpy/core/../../../../libmkl_intel_thread.so (0x00007f5c3f862000)
        libmkl_core.so => /home/rth/.miniconda3/envs/fd-env/lib/python3.6/site-packages/numpy/core/../../../../libmkl_core.so (0x00007f5c3dd30000)
        libiomp5.so => /home/rth/.miniconda3/envs/fd-env/lib/python3.6/site-packages/numpy/core/../../../../libiomp5.so (0x00007f5c3d986000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5c3d72e000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f5c3d433000)
        libpython3.6m.so.1.0 => /home/rth/.miniconda3/envs/fd-env/lib/python3.6/site-packages/numpy/core/../../../../libpython3.6m.so.1.0 (0x00007f5c3cf2b000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f5c3cb95000)
        /lib64/ld-linux-x86-64.so.2 (0x0000560220100000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f5c3c991000)
        libgcc_s.so.1 => /home/rth/.miniconda3/envs/fd-env/lib/python3.6/site-packages/numpy/core/../../../.././libgcc_s.so.1 (0x00007f5c3c778000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00007f5c3c575000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f5c3c36d000)
% ldd /home/rth/.miniconda3/envs/fd-env/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so | grep blas

so none of the linked libraries contain the the term blas.

Numpy with OpenBLAS

Python 3.6.3 | packaged by conda-forge | (default, Oct  5 2017, 14:07:33)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import numpy.core.multiarray, subprocess
>>> sh = lambda cmd: subprocess.check_output(cmd, shell=True)
>>> linked_blas = sh(f'ldd {numpy.core.multiarray.__file__} | grep blas')
>>> linked_blas
b'\tlibopenblas.so.0 => /home/rth/.miniconda3/envs/test7-env/lib/python3.6/site-packages/numpy/core/../../../../libopenblas.so.0 (0x00007f91ead8c000)\n'

we detect the linked blas correctly, however there is still an issue with detecting AVX,

>>> sh(f"objdump -d {linked_blas.split()[2]} | grep vaddss | wc")
objdump: 'b/home/rth/.miniconda3/envs/test7-env/lib/python3.6/site-packages/numpy/core/../../../../libopenblas.so.0': No such file
b'      0       0       0\n'

From the CLI, this works as expected,

objdump -d /home/rth/.miniconda3/envs/test7-env/lib/python3.6/site-packages/numpy/core/../../../../libopenblas.so.0 | grep vaddss | wc   
   1887   15128  110352