Notes about installing PyNEC in an Anaconda Virtual Environment

From_Source_Simplified.md

DOES NOT WORK!!! Windows 10 Instructions - PyNEC From Source in Anaconda 3 virtual environment with WSL2 + Ubuntu + VS2019

First of all, you need to follow the original installation instructions for PyNEC with respect to things you need to install on your computer.

This includes installing swig and C/C++ compilers. I have Visual Studio 2019 Community Edition installed, which gives me the MSVC compiler tools. I installed swig 4.0.2 and added it to my PATH.

When I tried the rest of the instructions, I couldn't get build.sh to work. Presumably I need to install make for Windows to get this to work? I think I also had trouble with the softlink. Will try again later with git bash trying to apply some of the things I learned here. However, I got it installed using the instructions below.

These modified instructions use Windows Subsystem for Linux for the make and configure steps, and assume WSL2 is installed as described here, a distro is installed (I use Ubuntu 20.04) and make is installed.

I know I also had to apt install several other things, including whatever provides libtoolize. Linux should prompt you to resolve these issues when you're going through the commands. Also, presumably this works with the original Windows Subsystem for Linux, instead of WSL2 which requires a very-up-to-date windows installation, but unfortunately I haven't tested that.

Replacing build.sh

Start by opening an Administrator Anaconda Command Prompt for the virtual environment you're installing in:

Then type:

git clone --recursive https://github.com/tmolteno/python-necpp/
cd python-necpp
cd PyNEC
mklink /d necpp_src ..\necpp_src
git submodule update --remote

Now open necpp_src/Makefile.am in Visual Studio Code. At the bottom, see where it says CRLF, click and change (at the top) to LF

then save the file. The motivation for this step is described here.

Now, return to the command prompt and type

wsl
cd necpp_src
make -f Makefile.git
./configure --without-lapack
exit

Now you should be back at the Anaconda command prompt (instead of linux). Make sure you're back in the PyNEC folder (you should be) and then type

python setup.py build

At this point, I get a bunch of errors and a linker failure!

LINK : error LNK2001: unresolved external symbol PyInit__PyNEC However, this doesn't seem to be a problem. Ignore the failure and type

pip install .

This successfully builds the wheel and installs PyNEC in the virtual environment.

Issues/Limitations

I don't know how to softlink necpp_src without being an administrator. This is not an issue on my machine, but it's inconvenient and could be a dealbreaker for those without admin access. Presumably something in the makefile or something could be changed to do this without softlinking?

The symbolic link needs to be in place for pip and python setup.py build so a Linux symlink doesn't work.

The biggest issue is cleaning up the admin-created python-necpp source directory, which I do with RMDIR /S python-necpp.

PyNEC_Install_Notes.md

PyNEC Installation Notes

I'm going to collect my notes for installing PyNEC on Windows 10.

I did this sometime in 2019 successfully, I think via cloning the repository and following these instructions. I remember hacking around some issue with the MSVC compiler on Windows, installing a different version or updating something.

However, I've forgotten how I got it to work. I'm going to take some notes this time, first trying to figure out what's going wrong with the much simpler pip install.

Machine and Software Environment

The machine is running Windows 10, Version 1909, Build 18363.959 (note date: 28 July 2020) I'm running Anaconda python and installing into a virtual environment.

Anaconda version info from conda list anaconda

# packages in environment at C:\Users\<user>\anaconda3\envs\main:
#
# Name                    Version                   Build  Channel
anaconda                  2020.02                  py37_0
anaconda-client           1.7.2                    py37_0
anaconda-navigator        1.9.12                   py37_0
anaconda-project          0.8.4                      py_0

cl.exe is Microsoft (R) C/C++ Optimizing Compiler Version 19.26.28806 for x86

where cl returns C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\bin\Hostx86\x86\cl.exe

I believe this was installed along with my installation of Visual Studio Community 2019, Version 16.6.4.

Attempted install via pip

It seems from this discussion that running pip install PyNEC should work on Windows, and I get the impression that it should do so via a binary wheel.

But it tries to build a wheel:

Collecting PyNEC
  Using cached PyNEC-1.7.3.6.tar.gz (24 kB)
Building wheels for collected packages: PyNEC
  Building wheel for PyNEC (setup.py) ... error

The traceback shows that the problem is in _msvcompiler.py. The last few lines of the traceback are

      File "c:\users\<username>\anaconda3\envs\main\lib\distutils\command\build_ext.py", line 559, in build_extension
        target_lang=language)
      File "c:\users\<username>\anaconda3\envs\main\lib\distutils\ccompiler.py", line 717, in link_shared_object
        extra_preargs, extra_postargs, build_temp, target_lang)
      File "c:\users\<username>\anaconda3\envs\main\lib\distutils\_msvccompiler.py", line 502, in link
        build_temp = os.path.dirname(objects[0])
    IndexError: list index out of range
    ------------------------------------

I added a temporary debug print to link() above in _msvccompiler.py, and this suggests that objects passed to link() is an empty list, explaining the index error. In context, starting at L497 of _msvccompiler.py:

            # The MSVC linker generates .lib and .exp files, which cannot be
            # suppressed by any linker switches. The .lib files may even be
            # needed! Make sure they are generated in the temporary build
            # directory. Since they have different names for debug and release
            # builds, they can go into the same directory.
            build_temp = os.path.dirname(objects[0])

I modified the assignment of build_temp to a hardcoded path on my machine, and it got further along:

This actually kicks it over to having a linker error, which makes sense, because pip install is building into a random-named directory like C:\Users\<username>\AppData\Local\Temp\pip-install-pdoqyynt\PyNEC\build.

Maybe I could hack this all the way through the chain.

Possibly related issues

Same sort of error with PyTorch

Attempted install from the repo

Trying to follow the installation instructions here

I did this, it didn't work, I didn't take notes. I will try it again after exhausting pip attempts.

Source build appears to be failing because build.sh is not Windows compatible. Need to look up Windows build instructions.

WSL2 to run commands from build.sh

build.sh fails because I don't have the whole toolchain available from git bash I think. So I'm trying to run the individual commands with Windows Subsystem for Linux.

One issue with make was that libtoolize crashed on make -f Makefile.git, possibly related to this

Forked Repo

Let's start over with a forked repo. Added .gitattributes per the link above.

In windows cmd

git clone --recursive https://github.com/danzimmerman/python-necpp.git
cd python-necpp
cd pynec
wsl

git submodule update --remote //fails! pressing on
DIR=`pwd`
cd ../necpp_src

make -f Makefile.git

Errors out with

libtoolize --force --copy
libtoolize: putting auxiliary files in '.'.
libtoolize: copying file './ltmain.sh'
.ibtoolize:   error: AC_CONFIG_MACRO_DIRS([m4]) conflicts with ACLOCAL_AMFLAGS=-I m4
make: *** [Makefile.git:12: all] Error 1

I verify that ltmain.sh is only <LF> line endings, so maybe it's something else.

I manually changed line endings in Makefile.am to <LF> from <CRLF>, now it succeeds.

make -f Makefile.git
exit //leave WSL!
swig -Wall -v -c++ -python PyNEC.i

Now bring up conda virtual env anaconda command window.

cd c:\code\radio\python-necpp\PyNEC
mklink necpp_src ..\necpp_src // to stop failing on something in src, apparently linux symlink can't be followed in windows
python3 setup.py build

Now this fails with

c:\code\radio\python-necpp\PyNEC\necpp_src\src\common.h(42): fatal error C1083: Cannot open include file: 'config.h': No such file or directory
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.26.28801\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2

Finally

Needs the directory switch on mklink, now python setup.py build and python setup.py install succeed:

mklink /d necpp_src ..\necpp_src

x_Full_Traceback_Original.md

Full Error, Original Attempt

Below is the full error output including traceback before the hard-coded build_temp path hack.

(main) C:\Users\<username>>pip install PyNEC
Collecting PyNEC
  Using cached PyNEC-1.7.3.6.tar.gz (24 kB)
Building wheels for collected packages: PyNEC
  Building wheel for PyNEC (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: 'c:\users\<username>\anaconda3\envs\main\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\<username>\\AppData\\Local\\Temp\\pip-install-dj_ldd6s\\PyNEC\\setup.py'"'"'; __file__='"'"'C:\\Users\\<username>\\AppData\\Local\\Temp\\pip-install-dj_ldd6s\\PyNEC\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\<username>\AppData\Local\Temp\pip-wheel-pnw0ph7v'
       cwd: C:\Users\<username>\AppData\Local\Temp\pip-install-dj_ldd6s\PyNEC\
  Complete output (62 lines):
  running bdist_wheel
  running build
  running build_py
  file PyNEC.py (for module PyNEC) not found
  file PyNEC.py (for module PyNEC) not found
  running egg_info
  writing PyNEC.egg-info\PKG-INFO
  writing dependency_links to PyNEC.egg-info\dependency_links.txt
  writing top-level names to PyNEC.egg-info\top_level.txt
  file PyNEC.py (for module PyNEC) not found
  reading manifest file 'PyNEC.egg-info\SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching 'necpp_src\src\*.h'
  warning: no files found matching 'necpp_src\config.h'
  writing manifest file 'PyNEC.egg-info\SOURCES.txt'
  running build_ext
  building '_PyNEC' extension
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "C:\Users\<username>\AppData\Local\Temp\pip-install-dj_ldd6s\PyNEC\setup.py", line 72, in <module>
      "Intended Audience :: Science/Research"]
    File "C:\Users\<username>\AppData\Roaming\Python\Python37\site-packages\setuptools\__init__.py", line 165, in setup
      return distutils.core.setup(**attrs)
    File "c:\users\<username>\anaconda3\envs\main\lib\distutils\core.py", line 148, in setup
      dist.run_commands()
    File "c:\users\<username>\anaconda3\envs\main\lib\distutils\dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "c:\users\<username>\anaconda3\envs\main\lib\distutils\dist.py", line 985, in run_command
      cmd_obj.run()
    File "c:\users\<username>\anaconda3\envs\main\lib\site-packages\wheel\bdist_wheel.py", line 223, in run
      self.run_command('build')
    File "c:\users\<username>\anaconda3\envs\main\lib\distutils\cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "c:\users\<username>\anaconda3\envs\main\lib\distutils\dist.py", line 985, in run_command
      cmd_obj.run()
    File "c:\users\<username>\anaconda3\envs\main\lib\distutils\command\build.py", line 135, in run
      self.run_command(cmd_name)
    File "c:\users\<username>\anaconda3\envs\main\lib\distutils\cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "c:\users\<username>\anaconda3\envs\main\lib\distutils\dist.py", line 985, in run_command
      cmd_obj.run()
    File "C:\Users\<username>\AppData\Roaming\Python\Python37\site-packages\setuptools\command\build_ext.py", line 87, in run
      _build_ext.run(self)
    File "c:\users\<username>\anaconda3\envs\main\lib\site-packages\Cython\Distutils\old_build_ext.py", line 186, in run
      _build_ext.build_ext.run(self)
    File "c:\users\<username>\anaconda3\envs\main\lib\distutils\command\build_ext.py", line 340, in run
      self.build_extensions()
    File "c:\users\<username>\anaconda3\envs\main\lib\site-packages\Cython\Distutils\old_build_ext.py", line 195, in build_extensions
      _build_ext.build_ext.build_extensions(self)
    File "c:\users\<username>\anaconda3\envs\main\lib\distutils\command\build_ext.py", line 449, in build_extensions
      self._build_extensions_serial()
    File "c:\users\<username>\anaconda3\envs\main\lib\distutils\command\build_ext.py", line 474, in _build_extensions_serial
      self.build_extension(ext)
    File "C:\Users\<username>\AppData\Roaming\Python\Python37\site-packages\setuptools\command\build_ext.py", line 208, in build_extension
      _build_ext.build_extension(self, ext)
    File "c:\users\<username>\anaconda3\envs\main\lib\distutils\command\build_ext.py", line 559, in build_extension
      target_lang=language)
    File "c:\users\<username>\anaconda3\envs\main\lib\distutils\ccompiler.py", line 717, in link_shared_object
      extra_preargs, extra_postargs, build_temp, target_lang)
    File "c:\users\<username>\anaconda3\envs\main\lib\distutils\_msvccompiler.py", line 502, in link
      build_temp = os.path.dirname(objects[0])
  IndexError: list index out of range
  ----------------------------------------
  ERROR: Failed building wheel for PyNEC
  Running setup.py clean for PyNEC
Failed to build PyNEC

... runs through it again with the same errors.

x_Full_Traceback_pip_Upgrade.md

Full Error, pip Upgrade

I upgraded to pip 20.2.1 and pip install pynec now throws the following error:

(main) C:\Users\<user>>pip install pynec
Collecting pynec
  Using cached PyNEC-1.7.3.6.tar.gz (24 kB)
Building wheels for collected packages: pynec
  Building wheel for pynec (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: 'c:\users\<user>\anaconda3\envs\main\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\<user>\\AppData\\Local\\Temp\\pip-install-uimjitm5\\pynec\\setup.py'"'"'; __file__='"'"'C:\\Users\\<user>\\AppData\\Local\\Temp\\pip-install-uimjitm5\\pynec\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\<user>\AppData\Local\Temp\pip-wheel-6tg186ry'
       cwd: C:\Users\<user>\AppData\Local\Temp\pip-install-uimjitm5\pynec\
  Complete output (62 lines):
  running bdist_wheel
  running build
  running build_py
  file PyNEC.py (for module PyNEC) not found
  file PyNEC.py (for module PyNEC) not found
  running egg_info
  writing PyNEC.egg-info\PKG-INFO
  writing dependency_links to PyNEC.egg-info\dependency_links.txt
  writing top-level names to PyNEC.egg-info\top_level.txt
  file PyNEC.py (for module PyNEC) not found
  reading manifest file 'PyNEC.egg-info\SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching 'necpp_src\src\*.h'
  warning: no files found matching 'necpp_src\config.h'
  writing manifest file 'PyNEC.egg-info\SOURCES.txt'
  running build_ext
  building '_PyNEC' extension
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "C:\Users\<user>\AppData\Local\Temp\pip-install-uimjitm5\pynec\setup.py", line 72, in <module>
      "Intended Audience :: Science/Research"]
    File "C:\Users\<user>\AppData\Roaming\Python\Python37\site-packages\setuptools\__init__.py", line 165, in setup
      return distutils.core.setup(**attrs)
    File "c:\users\<user>\anaconda3\envs\main\lib\distutils\core.py", line 148, in setup
      dist.run_commands()
    File "c:\users\<user>\anaconda3\envs\main\lib\distutils\dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "c:\users\<user>\anaconda3\envs\main\lib\distutils\dist.py", line 985, in run_command
      cmd_obj.run()
    File "c:\users\<user>\anaconda3\envs\main\lib\site-packages\wheel\bdist_wheel.py", line 223, in run
      self.run_command('build')
    File "c:\users\<user>\anaconda3\envs\main\lib\distutils\cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "c:\users\<user>\anaconda3\envs\main\lib\distutils\dist.py", line 985, in run_command
      cmd_obj.run()
    File "c:\users\<user>\anaconda3\envs\main\lib\distutils\command\build.py", line 135, in run
      self.run_command(cmd_name)
    File "c:\users\<user>\anaconda3\envs\main\lib\distutils\cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "c:\users\<user>\anaconda3\envs\main\lib\distutils\dist.py", line 985, in run_command
      cmd_obj.run()
    File "C:\Users\<user>\AppData\Roaming\Python\Python37\site-packages\setuptools\command\build_ext.py", line 87, in run
      _build_ext.run(self)
    File "c:\users\<user>\anaconda3\envs\main\lib\site-packages\Cython\Distutils\old_build_ext.py", line 186, in run
      _build_ext.build_ext.run(self)
    File "c:\users\<user>\anaconda3\envs\main\lib\distutils\command\build_ext.py", line 340, in run
      self.build_extensions()
    File "c:\users\<user>\anaconda3\envs\main\lib\site-packages\Cython\Distutils\old_build_ext.py", line 195, in build_extensions
      _build_ext.build_ext.build_extensions(self)
    File "c:\users\<user>\anaconda3\envs\main\lib\distutils\command\build_ext.py", line 449, in build_extensions
      self._build_extensions_serial()
    File "c:\users\<user>\anaconda3\envs\main\lib\distutils\command\build_ext.py", line 474, in _build_extensions_serial
      self.build_extension(ext)
    File "C:\Users\<user>\AppData\Roaming\Python\Python37\site-packages\setuptools\command\build_ext.py", line 208, in build_extension
      _build_ext.build_extension(self, ext)
    File "c:\users\<user>\anaconda3\envs\main\lib\distutils\command\build_ext.py", line 559, in build_extension
      target_lang=language)
    File "c:\users\<user>\anaconda3\envs\main\lib\distutils\ccompiler.py", line 717, in link_shared_object
      extra_preargs, extra_postargs, build_temp, target_lang)
    File "c:\users\<user>\anaconda3\envs\main\lib\distutils\_msvccompiler.py", line 502, in link
      build_temp = os.path.dirname(objects[0])
  IndexError: list index out of range
  ----------------------------------------
  ERROR: Failed building wheel for pynec
  Running setup.py clean for pynec
Failed to build pynec
DEPRECATION: Could not build wheels for pynec which do not use PEP 517. pip will fall back to legacy 'setup.py install' for these. pip 21.0 will remove support for this functionality. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.