A simple way to setup a python virtual environment without having to install anything (NO CONDA) - Linux and Windows

python.sh

#!/bin/bash
# ================================================================================================================================================================================================================================
# a simple script to create a virtual environment for python, install some basic packages and run a python script
# I known anaconda does the same, but I prefer to have a self-contained python environment in a folder without
# the need to install anything in the system.
# This script automatically downloads a self-contained python distribution in windows, so it can be used in any 
# computer without any installation.
# 
# In windows, use python.cmd to run this script. python.cmd will automatically download MSYS2 with bash and
# all dependencies to properly run this script.
# https://gist.github.com/hradec/370dc46326256ed64ff58c7f59446765#file-python-cmd
#
# For AMD, install rocm (apt install -y rocm* ; apt install -y hipcc) and use --check-rocm to check if it's working
# 
# TODO: env vars for versions of python and cuda.
# ================================================================================================================================================================================================================================

# Windows always use standalone python distribution
# Force to use standalone python distribution in linux
# comment the LINUX_STANDALONE_URL= line to use system python in linux


#➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖
# user configurable variables to setup versions of python, cuda and pytorch
#➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖
# python 3.10
# ================================================================================================================
LINUX_STANDALONE_URL='https://github.com/indygreg/python-build-standalone/releases/download/20241016/cpython-3.10.15+20241016-x86_64-unknown-linux-gnu-pgo+lto-full.tar.zst'
WINDS_STANDALONE_URL='https://github.com/winpython/winpython/releases/download/6.1.20230527/Winpython64-3.10.11.1dot.exe'

# these versions work with python 3.10 in windows
# torch 2.1.0 / cuda 12.1.0
# version 2.5.1/12.4 is broken on python 3.10 in windows

# ================================================================================================================
# python 3.11
# ================================================================================================================
LINUX_STANDALONE_URL='https://github.com/indygreg/python-build-standalone/releases/download/20241016/cpython-3.11.10+20241016-x86_64-unknown-linux-gnu-pgo+lto-full.tar.zst'
WINDS_STANDALONE_URL='https://github.com/winpython/winpython/releases/download/7.1.20240203final/Winpython64-3.11.8.0dot.exe'


# torch and cuda versions for windows
WINDS_TORCH_VERSION=2.1.0
WINDS_CUDA_VERSION=12.1.0

# torch and cuda versions for linux
LINUX_TORCH_VERSION=2.5.1
LINUX_CUDA_VERSION=12.4.0

#➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖
# end of user configurable variables
#➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖

# ================================================================================================================
# procedurally generated variables based on user configurable variables and the current system
# ================================================================================================================

# check if we are in windows( $CYGWIN = 1 ) or linux( $CYGWIN = 0 )
export CYGWIN="0"
export OSNAME="linux"
[ "$(grep CYGWIN /proc/version)" != "" ] && export CYGWIN="1"
[ "$(grep MSYS_NT /proc/version)" != "" ] && export CYGWIN="1"
[ "$CYGWIN" == "1" ] && export OSNAME="win32"

# set different versions for windows and linux
if [ "$CYGWIN" == "1" ] ; then
    TORCH_VERSION=$WINDS_TORCH_VERSION
    CUDA_VERSION=$WINDS_CUDA_VERSION
else
    TORCH_VERSION=$LINUX_TORCH_VERSION
    CUDA_VERSION=$LINUX_CUDA_VERSION
fi
# get the first two numbers of the cuda version for the pytorch url installation
CUDA_VERSION_URL=$(echo $CUDA_VERSION | awk -F'.' '{print $1$2}')


# get the script folder
SRC=$(dirname "$(readlink -f $0)")
if [ "$CYGWIN" == "1" ] ; then
	SRC="$(cygpath -m -a $BASH_SOURCE | sed 's#/[^/]*$##')"
fi
SRC=$(echo "$SRC" |  sed 's/ /\ /g')

# get the current folder
export PWD=$(pwd -P | sed 's/ /\ /g')
export __PWD=$(pwd -P | sed 's/ /\ /g')

# check if cuda folder exists where this script is
# if so, set CUDA_HOME to that folder
if [ -e "$SRC"/cuda/$CUDA_VERSION ] ; then
	export CUDA_HOME="$SRC"/cuda/$CUDA_VERSION/$OSNAME/
	export CUDA_TOOLKIT_ROOT_DIR="$CUDA_HOME"
fi
# detect the version of cuda pytorch was installed with, just in case!
if [ "$CYGWIN" == "1" ] ; then
    export __CUDA_VERSION=$(ls -dt "$SRC"/cuda/$(grep 'cuda:' "$SRC"/WP*/python*/Lib/site-packages/torch/version.py  | awk -F'=' '{print $(NF)}' | sed -e "s/'//g" -e "s/ //g" | tr -d '\r')* | head -1)
else
    export __CUDA_VERSION=$(ls -dt "$SRC"/cuda/$(grep 'cuda:' "$SRC"/LP*/install/lib/python*/site-packages/torch/version.py  | awk -F'=' '{print $(NF)}' | sed -e "s/'//g" -e "s/ //g" | tr -d '\r')* | head -1)
fi
# if the cuda version exists, set the environment variables
if [ "$__CUDA_VERSION" != "" ] ; then
    export CUDA_VERSION=$(basename $__CUDA_VERSION)
    export CUDA_HOME="$SRC/cuda/$CUDA_VERSION/$OSNAME/"
    export CUDA_TOOLKIT_ROOT_DIR="$CUDA_HOME"
fi

# change the home folder to the script folder so everything stays
# self-contained in the script folder
export USERPROFILE="$SRC/home/"
export HOME="$SRC/home/"
mkdir -p $HOME

# check if nvidia or amd rocm is installed in the system
export CUDA_NVIDIA_INSTALL=0
if [ "$(which nvidia-smi 2>/dev/null)" != "" ] ; then
    export CUDA_NVIDIA_INSTALL=1
fi
export CUDA_AMD_INSTALL=0
if [ "$(which rocm-smi 2>/dev/null)" != "" ] ; then
    export CUDA_AMD_INSTALL=1
fi

# display help!!
if [ "$1" == "-h" ] ; then
    echo "Usage: python.sh [--check-cuda] [--check-rocm] [--bash] [--cmd] [ --force-cuda-cpu-install ] [ --force-cuda-nvidia-install ] [ --force-cuda-amd-install ] | <python standard arguments>"
    exit 0
fi

export FORCE_CUDA_INSTALL=0
if [ "$1" == "--force-cuda-cpu-install" ] ; then
    shift
    export FORCE_CUDA_INSTALL=1
    export CUDA_NVIDIA_INSTALL=0
    export CUDA_AMD_INSTALL=0
fi
if [ "$1" == "--force-cuda-nvidia-install" ] ; then
    shift
    export FORCE_CUDA_INSTALL=1
    export CUDA_NVIDIA_INSTALL=1
fi
if [ "$1" == "--force-cuda-amd-install" ] ; then
    shift
    export FORCE_CUDA_INSTALL=1
    export CUDA_AMD_INSTALL=1
fi

# reset python environment so we don't inherit anything from the system
unset PYTHONPATH
unset PYTHONHOME
unset PYTHONDONTWRITEBYTECODE

# pip is a flag to trigger default pip package installation
export pip=0
# the path separator for linux or windows
export sep=":"

# system python executable by default
virtualenv_python="/usr/bin/python3"

# ================================================================================================================
# in windows, always download a self contained full python distribution and store in the script folder
# ================================================================================================================
if [ "$CYGWIN" == "1" ] ; then
    export sep=";"
    # download winpython in windows, so python is self-contained in the folder
    if [ "$(ls -d "$SRC"/WP*)" == "" ] ; then
        cd "$SRC"
        curl -L -k "$WINDS_STANDALONE_URL" -o winpython.7z
        7z x winpython.7z && rm -f winpython.7z
        cd "$PWD"
        export pip=1
    fi
    wpython_folder=$(ls -drt "$SRC"/WP* | tail -1)
    wpython_folder2=$(ls -drt "$wpython_folder"/python* | tail -1)
    virtualenv_python="$wpython_folder2/python"
    # we need ";:' befora adding $PATH to PATH, or else it wont convert the first
    # path in PATH to windows format. The conversion relies on regex ':\/.\/' (ex :/c/ => ;C:)
    # to identify if a path should be converted to windows format.
    # we also convert python folder and cuda_home to windows format, to prevent then to be 
    # converted to UNC (//IP ADDRESS//SHARE NAME) format in case they are already in windows format

    export PATH=";;$(cygpath -a -w $(readlink -f $wpython_folder2));;$(cygpath -a -w $(readlink -f $wpython_folder2/Scripts));;$(cygpath -a -w $CUDA_HOME/bin/);:$PATH"
    export PATH=";;$(cygpath -w $(readlink -f $wpython_folder2));;$(cygpath  -w $(readlink -f $wpython_folder2/Scripts));;$(cygpath  -w $CUDA_HOME/bin/);:$PATH"

# ================================================================================================================
# in linux, download a self-contained python distribution if LINUX_STANDALONE_URL is set
# ================================================================================================================
elif [ "$LINUX_STANDALONE_URL" != "" ] ; then
    VERSION=$(echo "$LINUX_STANDALONE_URL" | awk -F 'cpython-' '{print $2}' | awk -F'+' '{print $1}')
    # download a self contained python distribution in linux, if none is already downloaded  
    if [ "$(ls -d "$SRC"/LPy64-$VERSION)" == "" ] ; then
        # curl -L -k 'https://github.com/25077667/standalone-python/releases/download/release-2024-04-29/release-3.12-x86_64.tar.gz' | tar -xz -f -
        cd "$SRC"
        curl -L -k "$LINUX_STANDALONE_URL" | tar --zstd -x -f -
        # use the same name convention as in windows
        mv "$SRC"/python "$SRC"/LPy64-$VERSION
        cd "$PWD"
        export pip=1
    fi
    wpython_folder=$(ls -drt "$SRC"/LP* | tail -1)
    wpython_folder2="$wpython_folder/install"
    virtualenv_python="$wpython_folder2/bin/python"

    export PATH="$wpython_folder2$sep$CUDA_HOME/bin/$sep$PATH"

# ================================================================================================================
# not windows/not LINUX_STANDALONE_URL, use virtualenv to create a custom self contained environment (not tested)
# ================================================================================================================
else
    # get python version and hostname
    pv=$("$virtualenv_python" -c 'import sys,socket;print(f"{sys.version.split()[0]}-{socket.gethostname()}")' | head -1)

    # create virtual environment folder name
    venv="$SRC"/env.$pv
    if [ "$CYGWIN" == "1" ] ; then
        pv=$("$virtualenv_python" -c 'import sys,socket;print(f"{sys.version.split()[0]}-win")' | head -1)
        venv="$SRC"/env.$pv
        venv="$(cygpath -a -m "$venv")"
    fi

    # initialize the virtual environment, if none is already created
    if [ ! -e "$SRC"/env.$pv/bin/activate ] || [ "$REINSTALL" != "" ] ; then
        cd "$SRC"
        if [ "$CYGWIN" == "0" ] ; then
            "$virtualenv_python" -m ensurepip
        fi
        "$virtualenv_python" -m pip install virtualenv
        "$virtualenv_python" -m virtualenv -p python  "$venv"
        [ $? -ne 0 ] && exit 1

        if [ "$CYGWIN" == "1" ] ; then
            ln -s Scripts "$venv"/bin
            ln -s python "$venv"/bin/python3.exe
        else
            [ ! -e "$venv"/bin/python ] && ln -s "$venv"/bin/python3 "$venv"/bin/python
        fi

        source "$SRC"/env.$pv/bin/activate

        cd "$PWD"

        export pip=1
    else
        source "$SRC"/env.$pv/bin/activate
    fi

    export LD_LIBRARY_PATH="$SRC"/libs/$sep$LD_LIBRARY_PATH
    export PYTHONPATH=$PYTHONPATH$sep"$SRC"/src/
    export PYTHONPATH="$SRC"$sep$PYTHONPATH
    export PYTHONPATH="$__PWD"/python$sep$PYTHONPATH

    export virtualenv_python="$SRC/env.$pv/bin/python"

fi

# ================================================================================================================
# install some default packages - it looks for requirements.txt file in the script folder, in the parent 
# folder (../) or in the ../python folder. If none is found, it installs some basic packages, including pytorch
# ================================================================================================================
if [ "$pip" == "1" ] ; then

    # this ensures pip works correctly in virtualenv for some reason (also fixes certificate problems)
    curl -k -sS https://bootstrap.pypa.io/get-pip.py | "$virtualenv_python"

    # update certificates so pip works correctly
    "$virtualenv_python" -m pip install --upgrade certifi

    # upgrade pip
    "$virtualenv_python"  -m pip install --upgrade pip

    if [ -e "$SRC"/requirements.txt ] ; then
        "$virtualenv_python" -m pip install -r "$SRC"/requirements.txt
    elif [ -e "$SRC"/../requirements.txt ] ; then
        "$virtualenv_python" -m pip install -r "$SRC"/../requirements.txt
    elif [ -e "$SRC"/../python/requirements.txt ] ; then
        "$virtualenv_python" -m pip install -r "$SRC"/../python/requirements.txt
    else
        # install some basic packages
        "$virtualenv_python" -m pip install PySide6
        "$virtualenv_python" -m pip install numpy
        "$virtualenv_python" -m pip install pandas
        "$virtualenv_python" -m pip install opencv-python
        "$virtualenv_python" -m pip install pyinstaller
        "$virtualenv_python" -m pip install scipy
        "$virtualenv_python" -m pip install matplotlib
       
        export FORCE_CUDA_INSTALL=1 
    fi
fi

# ================================================================================================================
# pytorch installation
# ================================================================================================================
if [ "$FORCE_CUDA_INSTALL" == "1" ] ; then
    "$virtualenv_python" -m pip uninstall -y torch torchvision torchaudio

    # if amd rocm in the system
    if [ "$CUDA_AMD_INSTALL" == "1" ] ; then
        # install pytorch with rocm support (rocm 6.2 = rocm62), according with the website table: https://pytorch.org/
        # $SRC/env.$pv/bin/python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.2
        "$virtualenv_python" -m pip install torch==1.13.1+rocm5.2 torchvision==0.14.1+rocm5.2 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/rocm5.2

        # install support for AMD APUs to allocate system memory as GPU memory
        cd "$SRC"/env.$pv/
        git clone https://github.com/pomoke/torch-apu-helper.git
        cd torch-apu-helper
        hipcc ./gttalloc.c -o alloc.so -shared -fPIC
        install_path=$($SRC/env.$pv/bin/python -c 'import torch,os;print(os.path.dirname(torch.__file__))')
        cp ./alloc.so "$install_path"/

    # if nvidia in the system
    elif [ "$CUDA_NVIDIA_INSTALL" == "1" ] ; then
        # install pytorch with cuda support (cuda 12.1 = cu121), according with the website table: https://pytorch.org/
        "$virtualenv_python" -m pip install torch==$TORCH_VERSION torchvision torchaudio --index-url https://download.pytorch.org/whl/cu$CUDA_VERSION_URL
    else
        "$virtualenv_python" -m pip install torch torchvision torchaudio
    fi

    # install pytorch with intel gpu support (https://pytorch.org/docs/stable/notes/get_start_xpu.html)
    # "$virtualenv_python" -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/xpu
fi

# ================================================================================================================
# all environment is set, so from here we can run stuff
# ================================================================================================================
cd $__PWD

if [ "$1" == "--bash" ] ; then
    shift
    echo $PATH
    if [ "$CYGWIN" == "1" ] ; then
        "$SRC/msys64/usr/bin/bash" --noprofile --norc --verbose "$@"
    else
        bash --noprofile --norc --verbose "$@"
    fi
    exit 0
elif [ "$1" == "--cmd" ] ; then
    shift
    echo $PATH
    cmd "$@"
    exit 0
elif [ "$1" == "--check-cuda" ] ; then
    "$virtualenv_python" -c 'import torch;print(f"torch path:{torch.__file__}");print(f"torch version: {torch.version.__version__}");print(f"torch cuda version: {torch.version.cuda}");print(torch.cuda.get_device_properties(0).total_memory)'
elif [ "$1" == "--check-rocm" ] ; then
    "$virtualenv_python" -c 'import torch,os;new_alloc = torch.cuda.memory.CUDAPluggableAllocator(os.path.dirname(torch.__file__)+"/alloc.so","gtt_alloc","gtt_free");torch.cuda.memory.change_current_allocator(new_alloc);print(f"torch path:{torch.__file__}");print(f"torch version: {torch.version.__version__}");print(f"torch rocm version: {torch.version.hip}");print(torch.cuda.get_device_properties(0).total_memory)'
else
    "$virtualenv_python" "$@"
fi

Run python.sh -h for a quick help.

    --check_cuda                - to make sure pytorch cuda NVIDIA is working correctly on your system. 
    --check_rocm                - to make sure pytorch cuda/rocm (AMD) is working correctly on your system. 
    --bash                      - run bash inside the environment used to run python (windows and linux).
    --cmd                       - run windows CMD inside the environment used to run python.
    --force-cuda-cpu-install    - install python cuda CPU only, no gpu acceleration.
    --force-cuda-nvidia-install - install python cuda for NVIDIA gpus.
    --force-cuda-amd-install    - install python cuda/rocm for AMD gpus.

The versions of python, cuda and pytorch are setup at the top of the script. Change it accordingly to your needs.

To run in windows, use the python.cmd wrapper found here:
https://gist.github.com/hradec/370dc46326256ed64ff58c7f59446765#file-python-cmd

In Linux it downloads python standalone now, so we don't need to have python installed in the local distro, or if the local python is not the version we want.

You can force it to use the default Linux python by commenting out the LINUX_STANDALONE_URL line at the top.

The script will check the installed pytorch cuda version, and set the CUDA_VERSION and CUDA_HOME env vars accordingly.
TODO: download and extract cuda sdk automatically.