If one wants to import from a python module or a package that is stored or is being developed locally to other local module to have no import problems and not to have to play manually with PATH or PYTHONPATH or deal with absolute/relative import issues one should install it. These are my notes on how to do it successfully on Windows using conda environment.
To import local python module to a sibling or a totally different (unrelated) directory.
There are two modules: module_a
and module_b
. Both modules are stored locally, but in different, unrelated directories. One wants to import a some_function()
from module_a
to module_b
. The drama takes place on Windows operating system, also a conda environment is to be used.
In short: one needs to install module_a locally, in development (editable) mode: pip install -e
. To do so:
- Create a virtual environment, for sanity reasons, install
pip
in it (important!) and activate it.
In my case it's a conda environment.
Important note for conda environments:conda
andpip
can play together but with a caution. One should remember that allpip
installs should be after all theconda
packages installs are made. If one has toconda install
in an environment that has somepip install
s already in it it is better to (1) remove whole conda environment and (2) re-create it from the scratch, including this newconda
package, and then do the neccessarypip
installs.
Documentation onconda
andpip
: https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#pip-in-env
Also see in: https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-pkgs.html:
It is best to install all packages at once, so that all of the dependencies are installed at the same time.(...)
To gain the benefits of conda integration, be sure to install pip inside the currently active conda environment and then install packages with that instance of pip. The command conda list shows packages installed this way, with a label showing that they were installed with pip. - Prepare a proper directory structure for
module_a
.
I used a src-layout: https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#src-layout. - To be able to install the
module_a
as a package, create suitable__init__.py
andpyproject.toml
and/orsetup.cfg
,setup.py
files.
Package name for this use case would bemodulea
. - EDIT: Before taking this step check the UPDATE below this numbered list - alternative is possible.
Important for Windows: Run Powershell (preferably Anaconda Powershell in case of usingconda
) as administrator.
If you don't instead of installing in conda environmentpip
could rather install the package somewhere else, f.e. in base conda environment or even in your global python interpreter. You will know that such event took place, when afterpip install -e
you get an info:Defaulting to user installation because normal site-packages is not writeable
. This message indicates thatpip
could not write to conda environment directory. - In Powershell: go to the directory where those previously (pt.3.) created package config files are stored.
- Check if you are using
pip
instance installed in your virtual environment. It's crucial for conda environment (see: https://bitsofanalytics.org/posts/pip-conda-local-dev/pip_conda_local_dev).
Commands to check it: for UNIX/Linux it would bewhich pip
, for Windows Powershell it iswhere.exe pip
. As a result you will get full path topip
.
Confirm that it is in your virtual environment's directory. If it is so, copy this full path. - In Powershell: paste that copied path and then
full\path\to\your\conda\env\pip.exe install -e .
(note: this dot at the end of command is important! and indicates that you are installing the project configured in the directory you are in (pt. 5 above)) - Successfull installation message should be similar to:
Instead ofObtaining file:///full/dir/to/package/modulea Preparing metadata (setup.py) ... done Installing collected packages: modulea Running setup.py develop for modulea Successfully installed modulea-0.1.0
modulea
there would be your package name of course. - To confirm that the package was installed in the conda environment not somewhere else, in Powershell:
conda list
--> it will list all the packages installed in the conda environment. Your package should be on that list.pip list
--> it will list all the packages installed in the environment bypip
. The package should be there too. - In the
package\src
directory there will also be new folder created, which name is:package_name.egg-info
(f.e.modulea.egg-info
in this use case).
Instead of running Powershell as administrator I managed to install succesfully, without the Defaulting to user installation because normal site-packages is not writeable
issue (see pt. 4 above), by running pip
via conda run
as follows:
Go through points 1 - 3 and 5 - 6 as stated above. Then in VSCode Powershell terminal or other Powershell (preferably Anaconda Powershell), with your conda environment activated:
conda run -n yourEnvName full\path\to\your\conda\env\pip.exe install -e .
It should behave exactly as stated in points 8 - 10.
Now, when using this virtual environment and its python's interpreter you can simply import modulea
or from modulea import some_function
to any python module easily!
Take into consideration, that it is oftes argued, that using python -m pip install package
is batter / safer than plain pip install package
or pip3 install package
. More on this:
https://snarky.ca/why-you-should-use-python-m-pip/
https://stackoverflow.com/questions/25749621/whats-the-difference-between-pip-install-and-python-m-pip-install
Windows 11
python 3.12
pip 23.3
conda 23.9
VS Code 1.85.1 (user setup)
About python imports, modules/packages and virtual environments.
https://packaging.python.org/en/latest/tutorials/installing-packages/
https://pip.pypa.io/en/stable/topics/local-project-installs/
https://setuptools.pypa.io/en/latest/userguide/quickstart.html
https://stackoverflow.com/questions/14132789/relative-imports-for-the-billionth-time
https://stackoverflow.com/questions/1896918/running-unittest-with-typical-test-directory-structure
https://stackoverflow.com/a/50193944/19716626
https://realpython.com/python-import/#create-and-install-a-local-package
https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure
https://snarky.ca/how-virtual-environments-work/
https://bitsofanalytics.org/posts/pip-conda-local-dev/pip_conda_local_dev