With a somewhat complex set of data science related dependencies (matplotlib, scikitlearn, and pytorch amongst others) I initially I came across this error:
Error #15: Initializing libiomp5.dylib, but found libiomp5.dylib already initialized OMP: Hint: This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.
The solutions I found to this problem when searching was primarily around forcing the Conda environment to use X86 (basically Rosetta2 emulation). Upon doing this I found issues with MKL as per the error below:
Intel MKL FATAL ERROR: This system does not meet the minimum requirements for use of the Intel(R) Math Kernel Library. The processor must support the Intel(R) Supplemental Streaming SIMD Extensions 3 (Intel(R) SSSE3) instructions. The processor must support the Intel(R) Streaming SIMD Extensions 4.2 (Intel(R) SSE4.2) instructions. The processor must support the Intel(R) Advanced Vector Extensions (Intel(R) AVX) instructions.
Bottom line is that while all the solutions were focused on ensuring there was a single architecture being used, it still did not resolve my issues. As more versions of the common data science libraries have been migrated to Apple silicon, it seems like more platform specific optimisations have also been added to the mix including Intel architectures. Bottom line is that I realised the best solution was not to use compatibility, rather, force the Conda installation to use M1 specific builds for everything. It has been nearly two years since M1 was introduced, so we should be safe in doing this (I hoped) - using the bash script below I was proven correct.
You can easily modify the script below to:
- use plain Conda instead of (the vastly superior) Mamba
- change where the yml file is expected to be
- change your 'forced' architecture (I can't think of other environments where this emulation problem would be an issue so I only made the script support M1)
The expectation is that you understand the format of Conda dependency YML files. See https://stackoverflow.com/questions/44742138/how-to-create-a-conda-environment-based-on-a-yaml-file for more details.