Python project setup and packaging, for relatively new users using pdm.

python_project_packaging.md

Python project setup and packaging for relatively new users using pdm.

Create a project folder

I use Linux with Bash Shell, so this is the general meaning of the command prompt. Please refer to it if you are confused as to which folder to run the commands from.

[username@localhost currentfolder]$ 

So, for example, I have a folder called repos for saving my projects. I will create a project folder called pypassive in repos. I like to organize the packages according to their names.

[username@localhost repos]$ mkdir pypassive 

cd into the project folder, pypassive, and create a package folder of the same name pypassive. You may choose to do things differently but this is how I am used to doing things.

New Project

I will add a README.md file and create two additional directories for tests and examples. Follow the pdm documentation pdm - New Project. Also place a readme and a license file in the project directory.

pypassive <-- Project directory root
├── examples
├── LICENSE.md
├── pypassive  <-- Package directory root
├── README.md
└── tests

where, the top level pypassive directory is the project directory.

I generally like everything in my project directory virtual environment. Create a virtural environment folder.The 3.11 is the Python version I want to use pdm - Working with virtual environments

[username@localhost pypassive]$ pdm venv create 3.11

The project folder will now have the virtual environment folder.

pypassive
├── examples
├── LICENSE.md
├── pypassive
├── README.md
├── tests
└── .venv

Now we need to tell pdm to use this virtual environment.

[username@localhost pypassive]$ pdm use -f .venv

Using Python interpreter: /home/username/data/work/repos/pypassive/.venv/bin/python (3.11) 

The message indicates that pdm will use the virtual environment. The above line will be written to .pdm-python.

pypassive
├── examples
├── LICENSE.md
├── pypassive
├── README.md
├── tests
├── .pdm-python
└── .venv

Now to initialize the project and create the pyproject.toml file.

[username@localhost pypassive]$ pdm init

When presented with a choice of python interpreters choose the one in the virtual environment. Then complete the project setup questions.

Dependencies can then be added, e.g. numpy and the packages will be installed in the virtural environmental.

[username@localhost pypassive]$ pdm add numpy scipy

For development only dependencies checkout pdm- Dev only dependencies.

pdm add -dG test pytest
pdm add -dG lint black flake8
pdm add -dG docs sphinx sphinx-rtd-theme

Build the project without publishing.

pdm build

Install the project so that you can keep developing. This can be done without building, I think install will build and install.

In case you get an error saying module cannot be found then check projectname.pth file.

pypassive/.venv/lib/python3.11/site-packages/pypassive.pth

The file may contain the wrong path

/home/username/data/work/repos/pypassive/src

Change that to

/home/username/data/work/repos/pypassive

Package directory

Now add your code inside the package directory pypassive. Also, add __init__.py and __about__.py. I wrote my code inside soil.py.

./  <-- This is the package directory root.
├── __about__.py
├── __init__.py
└── soil.py

Inside the soil.py I have a class called SoilLayer. So to be able to import the SoilLayer class from the soil.py modulue as from pypassive import SoilLayer, put the following import in __init__.py file.

from .soil import SoilLayer

Without the above import in the __init__.py file we would have to import the SoilLayer class as from pypassive.soil import SoilLayer

Documentation

These notes are for creating the documentation using the docstrings with Sphinx and publish those in Read the docs. This is based on docstrings so please document your code accordingly.

1. Install sphinx and sphinx-rtd-theme, in the same environment in which you are building your package.

[username@localhost pypassive]$ pdm add sphinx
[username@localhost pypassive]$ pdm add sphinx-rtd-theme

pdm should install these in the virtual environment.

2. Create docs directory in your package directory to hold the documentation files.

[username@localhost pypassive]$ mkdir docs

Your project tree will look as follows:

pypassive
├── docs
├── examples
├── LICENSE.md
├── pdm.lock
├── __pycache__
├── pypassive
├── pyproject.toml
├── README.md
└── tests

3. Activate the virtural environment.

[username@localhost pypassive]$eval $(pdm venv activate)
(pypassive-3.11) [username@localhost pypassive]$ 

4. Run sphinx-quickstart in the docs folder. Provide the information and when asked about creating a separate folder for build type y.

(pypassive-3.11) [username@localhost pypassive]$ cd docs
(pypassive-3.11) [username@localhost docs]$ sphinx-quickstart

> Separate source and build directories (y/n) [n]: y

The project name will occur in several places in the built documentation.
> Project name: pypassive
> Author name(s): Your name
> Project release []: 0.0.1

If the documents are to be written in a language other than English,
you can select a language here by its language code. Sphinx will then
translate text that it generates into that language.

For a list of supported codes, see
https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-language.
> Project language [en]: 

Creating file /home/username/data/work/repos/pypassive/docs/source/conf.py.
Creating file /home/username/data/work/repos/pypassive/docs/source/index.rst.
Creating file /home/username/data/work/repos/pypassive/docs/Makefile.
Creating file /home/username/data/work/repos/pypassive/docs/make.bat.

Finished: An initial directory structure has been created.

The docs folder will have the following files and folders:

docs
├── build
├── make.bat
├── Makefile
└── source

Within the source folder there will be two files conf.py and index.rst.

source
├── conf.py
├── index.rst
├── _static
└── _templates

5. Edit the conf.py file.

   • a. Change the html_theme to the one of your liking, html_theme = 'sphinx_rtd_theme'.

   • b. For autocode generation from docstrings in the code, specifiy the directory of source code.

   import os
   import sys
   sys.path.insert(0, os.path.abspath('../../pypassive/'))
   

   • c. Also add some extensions

   extensions = ['sphinx.ext.todo', 'sphinx.ext.viewcode', 'sphinx.ext.autodoc']
   

6. Run the Sphinx API documentation command sphinx-apidoc to generate Sphinx source files. Run this from the docs folder and not the source folder.

(pypassive-3.11) [username@localhost docs]$ sphinx-apidoc -o source/ ../pypassive

pypassive is the package folder. Sphinx will convert the docstrings in your code in that package directory to create the documentation.

At this point your docs folder should look similar to this:

/docs
├── build
├── make.bat
├── Makefile
└── source
    ├── conf.py
    ├── index.rst
    ├── modules.rst
    ├── pypassive.rst
    ├── _static
    └── _templates

6. To build the documentation, html files in the build/ directory.

(pypassive-3.11) [username@localhost docs]$ make html

7. To view the documentation in your browser

(pypassive-3.11) [username@localhost docs]$ firefox ./build/html/index.html

Publishing your documentation on the ReadTheDocs

1. Since the project/dependencies are managed by pdm, we need to create a requirements.txt file. Run this command in the project root directory.

(pypassive-3.11) [username@localhost pypassive]$ pdm export -o requirements.txt --pyproject --prod
``

2. Create a account in `ReadTheDocs` and log into `ReadTheDocs` and click on `Import Project`. Fill out the fields. For the repository URL use `https://gitlab.com/geosharma/pypassive.git`. This can be got from `CLONE -> Clone with HTPPS` URL.

3. Then in `Projects -> Admin -> Integration`, click on `Add Integration` and choose `GitLab incoming webhook`.

4. Then copy the URL starting with `readthedocs.org/api/v2/webhook/...../......` and paste it into GitLab Project `Settings -> Webhooks`

5. Also copy the `Secret` token and paste it in `Gitlab` project `Webhooks -> Secret token` field.


## References:
 - [Read the docs](https://readthedocs.org/)
 - [Sphinx-RTD-Tutorial](https://sphinx-rtd-tutorial.readthedocs.io/en/latest/index.html)
 - [A idiot's guide to Python documentation with Sphinx and ReadTheDocs](https://samnicholls.net/2016/06/15/how-to-sphinx-readthedocs/)
 - [Read the Docs VSC Integration](https://docs.readthedocs.io/en/stable/integrations.html#github)
 - [Deplyoing a Sphinx project online](https://www.sphinx-doc.org/en/master/tutorial/deploying.html)
 -[Making Readthedocs for a Python package](https://pennyhow.github.io/blog/making-readthedocs/)
 -[An introduction to Sphinx and Read the Docs for Technical Writers](https://www.ericholscher.com/blog/2016/jul/1/sphinx-and-rtd-for-writers/)
 - [Python Packaging](https://sinoroc.gitlab.io/kb/python/packaging.html)