Rez-Windows-Development-Workflow.md

Introduction

• This guide is for developers or rez enthusiasts wondering how to get started creating a development environment for working on or debugging rez.
• While the steps are applicable to Linux, this guide was written while walking through the steps on a Windows virtual machine, so it focuses on Windows. It uses bash so that it more closesly resembles what would be seen on Linux.

- Throughout this process I use a virtual machine that has been configured using a [setup script](https://gist.github.com/Jawabiscuit/6298e2c4bebe5ddde694df9d18d9451b) but it is not necessary. It might be worthwhile looking at since it provides some information about the testing environment.

• This guide highlights a workaround for creating an editable install using pip. Pip is normally used this way to install a Python package that is either stored locally instead of on Pypi and is robust enough that it and can easily clone and install remote repo. There are various reasons why a developer would choose to do this, but for the sake of this guide the desire is to have a working copy of rez with source files that can be inspected for any possible issues while using it Windows.

Prep Virtual Machine for rez Installation

• Python 2 & 3 are installed but machine is configured (manually for now) so that Python 3 appears first in *PATH variables. This is the version that is used to install rez. At the time of writing I consider Python 3 with rez experimental but my limited experience has been stable. You may choose either depending on relevant external factors.

• This guide uses gitbash. Powershell may be necessary given gitbash is still a new plugin for rez at the time of writing.

Install virtualenv

• I prefer installing virtualenv and not using the builtin venv at the time of writing. This will probably change as I transition to using Python 3 full time.

python -m pip install --user virtualenv

• Make sure virtualenv executables are on the PATH

export PATH=$APPDATA/Python/Python310/Scripts:$PATH

• cygpath is an option for variable expansion using posix path delimiters, if you choose to try to only use posix paths:

export PATH="`cygpath -u "$APPDATA"`/Python/Python310/Scripts":$PATH

Create a virtual env

• Create a virtual env called venv or rez_venv if you are creating multiple or just like naming it that so that its easier to remember what the virtual env is used for.

virtualenv venv

# or

virtualenv rez_venv

Activate the virtual env, and keep it activated for either of the editable install process methods that follow.

. ~/src/venv/Scripts/activate

# or 

. ~/src/rez_venv/Scripts/activate

Pip Editable Install: Method 1

• Disclosure: I've encountered warning installing rez this first way. It should work but install.py actually is what rez uses to pip install the package so an edit is needed there until it is "fixed".
• To my knowledge, editing install.py is how you can circumnavigate this warning. See also: Pip-based rez installation detected warning with installation script for more background info.
• Skip to [[2022-07-13-rez-devel-workflow-getting-started#Pip Editable Install Method 2]] if you want to create an editable install using this second method

Warning: Pip-based rez installation detected

• Normally, creating an editable install for a python egg hosted remotely is achieved by running a command like this
  • python -m pip install -e "git+https://github.com/AcademySoftwareFoundation/rez.git#egg=rez"
  • This installs rez directly from the official repository
  • read more about eggs and wheels here

A personal fork of rez was cloned while this guide was written. This changes the location source files will live on disk in relation to the virtual env directory. This information is imporant to know in later steps, in Method 2, where it will be necessary to make a tiny change to the source code.

If installing from a remote repo, the source will be installed inside the virtual env directory `venv` or `rez_venv` depending on how you chose to name it. If installing from a local directory, as discussed next, then obviously the source is outside the virtual env directory and is linked through .egg-info metadata.

• If creating an editable install from a downloaded zip or tar.gz or cloned repo
  • If zipped, unzip the file to the source directory
  • run pip install, giving it the local directory
  • python -m pip install -e path/to/src/rez-repo

# chdir to project
cd

# activate virtualenv
. venv/Scripts/activate

# editable install `-e`
python -m pip install -e "git+https://github.com/AcademySoftwareFoundation/rez.git#egg=rez"`

# or
python -m pip install -e path/to/local/rez

• Other helpful pip flags, especially for installing using python27

--no-input --disable-pip-version-check

Developers sometimes choose to bootstrap these steps by creating a their own install script. I chose to do it this way to have a more reliable "recipe" for getting started quickly. Coincidentally, the idea is similar to installing any other rez package.

- Running `rez` with the virtual env activated executes `venv/Scripts/rez.exe`
    - Without activating, `rez` is executed using PATH which is more fully illustrated using the 2nd method
- A `rez.egg-info` directory is installed in the `src` directory
    - ``.egg-info`` format: a file or directory placed *adjacent* to the
   project's code and resources, that directly contains the project's
   metadata. -- https://svn.python.org/projects/sandbox/trunk/setuptools/doc/formats.txt

Pip Editable Install: Method 2

• This involves editing the pip install command found in install.py of the rez source files.

- It may be easiest to use https://vscode.dev if you are installing on a bare-bones virtual machine with no IDE software installed like I am.

• Clone the official rez repo somewhere
  • I chose to download the zip edit the master with intentions of creating a branch later if necessary and the opportunity presents itself for me to focus on creating a proper fix.
  • You may choose to clone the repo from the internet
  • Either way, make the edits in a local copy of rez

- The following information for this workaround is inspired by this [Slack discussion](https://academysoftwarefdn.slack.com/archives/C0321B828FM/p1650655464411449)

• Edit install.py (link shows an example commit)
  • Insert "-e, " into the line that reads run_command([py_executable, "m", "pip", "install", "."]) so that it looks like run_command([py_executable, "m", "pip", "install", "-e", "."])

• Important: make sure to activate venv or rez_venv depending on what you chose to name you virtual env
• Install and note what happens
  • install.py is running pip using "-e" that was inserted in the previous step

python install.py -v /c/rez

What is a "production install" directory and where is it? On Windows, rez exe binaries call python and "production install" refers to where that binary directory is located. There's a hidden file marking the directory if an `ls -la` is invoked in the directory.

I'm taking an educated guess that it's done this way because, unlike Unix-based OSs, legacy Windows by default cannot execute python files, or hardly any other file besides exes.

- Since I'm using a virtual machine, I save a snapshot after the editable install is finished installing.

• Add production install rez paths to the environment. I usually echo these lines into a bashrc file for quick testing purposes.
  • This is an example of some setup I would put in a rez install sh, ps, or batch script

export PATH=/c/rez/Scripts/rez:$PATH
export PYTHONPATH=/c/rez/Lib/site-packages:/c/rez/Scripts

Test Edits to rez Source Files

• Production install rez directory structure on the system drive. This directory exists in a separate location on the filesystem from the source but is linked back to the source.

• If the source was cloned from a remote repo, add venv or rez_venv directory to editor workspace

• If the source was located on the local filesystem at the time of installation, add that directory to the editor workspace instead.

• Insert a debug print message in _main.py (link shows an example edit)

• Observe the print output using the rez cli

  • Open a new terminal, if not done so already
  • The path to rez executable is now pointing to the production install instead of the virtual env, due to the PATH variable that was discussed earlier.
  • Test the print message
  • Also observe there is no "pip-based rez install warning"
    • Conversely, the warning will appear if rez is executed in a terminal if the rez virtual env is activated

rez config platform_map

• Rez is now installed to a production install location as an "editable install", so you can now edit the source .py files and test out the effects!

Reference

• pip editable installs

• setuptools development

• how do editable pip installs work? anthony explains

• Stackoverflow - Installing Python packages from local file system to virtualenv with pip

• Stackoverflow - What is a Python Egg?