Skip to content

Instantly share code, notes, and snippets.

@isuruf
Last active May 25, 2021 14:47
Show Gist options
  • Save isuruf/aaead06e2b1d801a20910bc44a9ffcdf to your computer and use it in GitHub Desktop.
Save isuruf/aaead06e2b1d801a20910bc44a9ffcdf to your computer and use it in GitHub Desktop.
Use conda-forge lapack with acelerate as blas backend
git clone https://github.com/isuruf/vecLibFort
cd vecLibFort
conda create -n numpy pytest hypothesis numpy "libblas=*=*netlib" --yes
conda activate numpy
clang -O3 -c -o vecLibFort.o vecLibFort.c
clang -shared -o libblas.3.dylib vecLibFort.o -Wl,-reexport_library,$(xcrun -show-sdk-path)/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.tbd
codesign -s - -f libblas.3.dylib
rm $CONDA_PREFIX/lib/libblas.3.dylib
rm $CONDA_PREFIX/lib/libcblas.3.dylib
cp libblas.3.dylib $CONDA_PREFIX/lib/libblas.3.dylib
cp libblas.3.dylib $CONDA_PREFIX/lib/libcblas.3.dylib
python -c "import numpy; numpy.test()"
@ogrisel
Copy link

ogrisel commented Jan 31, 2021

If you use the openblas build, tests work?

Running the numpy test on the same machine in an openblas env, I get:

10 failed, 11836 passed, 102 skipped, 1205 deselected, 19 xfailed, 3 xpassed, 24 warnings, 444 errors in 86.37s (0:01:26)

the log is even more verbose. Here is the full log:

https://gist.github.com/ogrisel/be1fd7f9d00f32f3c702a10a9886239d

The 10 failures are:

FAILED miniforge3/envs/sklearn-0241/lib/python3.9/site-packages/numpy/core/tests/test_ufunc.py::TestUfuncGenericLoops::test_unary_PyUFunc_O_O_method_full[reciprocal]
FAILED miniforge3/envs/sklearn-0241/lib/python3.9/site-packages/numpy/distutils/tests/test_system_info.py::TestSystemInfoReading::test_compile1
FAILED miniforge3/envs/sklearn-0241/lib/python3.9/site-packages/numpy/distutils/tests/test_system_info.py::TestSystemInfoReading::test_compile2
FAILED miniforge3/envs/sklearn-0241/lib/python3.9/site-packages/numpy/f2py/tests/test_compile_function.py::test_f2py_init_compile[extra_args0]
FAILED miniforge3/envs/sklearn-0241/lib/python3.9/site-packages/numpy/f2py/tests/test_compile_function.py::test_f2py_init_compile[--noopt --debug]
FAILED miniforge3/envs/sklearn-0241/lib/python3.9/site-packages/numpy/f2py/tests/test_compile_function.py::test_f2py_init_compile[]
FAILED miniforge3/envs/sklearn-0241/lib/python3.9/site-packages/numpy/f2py/tests/test_compile_function.py::test_compile_from_strings[program test_f2py\nend program test_f2py0]
FAILED miniforge3/envs/sklearn-0241/lib/python3.9/site-packages/numpy/f2py/tests/test_compile_function.py::test_compile_from_strings[program test_f2py\nend program test_f2py1]
FAILED miniforge3/envs/sklearn-0241/lib/python3.9/site-packages/numpy/ma/tests/test_core.py::TestMaskedArrayInPlaceArithmetics::test_inplace_floor_division_scalar_type
FAILED miniforge3/envs/sklearn-0241/lib/python3.9/site-packages/numpy/ma/tests/test_core.py::TestMaskedArrayInPlaceArithmetics::test_inplace_floor_division_array_type

@ogrisel
Copy link

ogrisel commented Jan 31, 2021

So the tests that only fail in the Accelerate env are:

FAILED miniforge3/envs/numpy/lib/python3.9/site-packages/numpy/f2py/tests/test_array_from_pyobj.py::TestSharedMemory::test_in_from_2casttype[LONGDOUBLE]
FAILED miniforge3/envs/numpy/lib/python3.9/site-packages/numpy/f2py/tests/test_array_from_pyobj.py::TestSharedMemory::test_f_in_from_23casttype[LONGDOUBLE]
FAILED miniforge3/envs/numpy/lib/python3.9/site-packages/numpy/f2py/tests/test_array_from_pyobj.py::TestSharedMemory::test_c_in_from_23casttype[LONGDOUBLE]

@isuruf
Copy link
Author

isuruf commented Jan 31, 2021

Running the numpy test on the same machine in an openblas env, I get:

You probably need to install compilers there too.

@ogrisel
Copy link

ogrisel commented Jan 31, 2021

I did install the compilers in that env (prior to running the numpy tests):

% conda list compilers
# packages in environment at /Users/ogrisel/miniforge3/envs/sklearn-0241:
#
# Name                    Version                   Build  Channel
compilers                 1.1.3                hce30654_0    conda-forge

@isuruf
Copy link
Author

isuruf commented Jan 31, 2021

/Users/ogrisel/miniforge3/envs/sklearn-0241/include/python3.9/Python.h:25:10: fatal error: 'stdio.h' file not found
E #include <stdio.h>

above error is because the env is not activated. (If it is activated, then $SDKROOT env variable should be set to the SDK dir)

@ogrisel
Copy link

ogrisel commented Feb 1, 2021

You are right, something was off with this env. I deactivated everything and reactivated and now I get:

=========================== short test summary info ============================
FAILED miniforge3/envs/sklearn-0241/lib/python3.9/site-packages/numpy/core/tests/test_ufunc.py::TestUfuncGenericLoops::test_unary_PyUFunc_O_O_method_full[reciprocal]
FAILED miniforge3/envs/sklearn-0241/lib/python3.9/site-packages/numpy/f2py/tests/test_array_from_pyobj.py::TestSharedMemory::test_in_from_2casttype[LONGDOUBLE]
FAILED miniforge3/envs/sklearn-0241/lib/python3.9/site-packages/numpy/f2py/tests/test_array_from_pyobj.py::TestSharedMemory::test_f_in_from_23casttype[LONGDOUBLE]
FAILED miniforge3/envs/sklearn-0241/lib/python3.9/site-packages/numpy/f2py/tests/test_array_from_pyobj.py::TestSharedMemory::test_c_in_from_23casttype[LONGDOUBLE]
FAILED miniforge3/envs/sklearn-0241/lib/python3.9/site-packages/numpy/ma/tests/test_core.py::TestMaskedArrayInPlaceArithmetics::test_inplace_floor_division_scalar_type
FAILED miniforge3/envs/sklearn-0241/lib/python3.9/site-packages/numpy/ma/tests/test_core.py::TestMaskedArrayInPlaceArithmetics::test_inplace_floor_division_array_type
6 failed, 12284 passed, 102 skipped, 1205 deselected, 19 xfailed, 3 xpassed, 24 warnings in 87.73s (0:01:27)

which are the same failures as with the Accelerate BLAS + netlib LAPACK combo so probably unrelated to BLAS and LAPACK.

@ogrisel
Copy link

ogrisel commented Feb 1, 2021

@ogrisel
Copy link

ogrisel commented Feb 1, 2021

I noticed that I had two nested conda env in the previous run (the sklearn-0241 env nested under my dev env). Maybe that was the cause of the problem...

@ogrisel
Copy link

ogrisel commented Feb 1, 2021

Or maybe I installed compilers in sklearn-0241 after the creation and activation of sklearn-0241 without reactivating it. I cannot remember.

@kirksw
Copy link

kirksw commented May 3, 2021

I get AssertionErrors from multiple important routines using this on my M1 mac. However from what I have been reading numpy and scipy blocked the accelerate backend due to numerical issues.

=============================== short test summary info ================================ FAILED core/tests/test_ufunc.py::TestUfuncGenericLoops::test_unary_PyUFunc_O_O_method_full[reciprocal] FAILED f2py/tests/test_array_from_pyobj.py::TestSharedMemory::test_in_from_2casttype[LONGDOUBLE] FAILED f2py/tests/test_array_from_pyobj.py::TestSharedMemory::test_f_in_from_23casttype[LONGDOUBLE] FAILED f2py/tests/test_array_from_pyobj.py::TestSharedMemory::test_c_in_from_23casttype[LONGDOUBLE] FAILED linalg/tests/test_linalg.py::TestSVDHermitian::test_herm_cases - AssertionErro... FAILED linalg/tests/test_linalg.py::TestPinvHermitian::test_herm_cases - AssertionErr... FAILED linalg/tests/test_linalg.py::TestEighCases::test_herm_cases - AssertionError: ... FAILED linalg/tests/test_linalg.py::TestCholesky::test_basic_property - numpy.linalg.... FAILED ma/tests/test_core.py::TestMaskedArrayInPlaceArithmetics::test_inplace_floor_division_scalar_type FAILED ma/tests/test_core.py::TestMaskedArrayInPlaceArithmetics::test_inplace_floor_division_array_type 10 failed, 12457 passed, 103 skipped, 1208 deselected, 19 xfailed, 3 xpassed, 16 warnings in 98.77s (0:01:38)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment