Debugging pyo3 virtualenv issues

pyo3-venv-debugging.md

I'm running Windows 10. I am using Git-bash, but the same issues happen when run from Powershell.

I have tried Python 3.9 and 3.7, fresh installs.

pyo3 Gitter discussion: https://gitter.im/PyO3/Lobby?at=6143ba0b7cd57813a8cbc355

Minimal example

use pyo3::prelude::*;
use std::process::Command;

fn main() -> PyResult<()> {
    // Module 'serial' (pip install pyserial) is installed in the virtualenv but NOT the global python

    // This works
    let cmd = Command::new("python")
        .args(&["-c", "import serial; print(serial)"])
        .output()
        .unwrap();
    println!("{}", String::from_utf8_lossy(&cmd.stdout));

    // This fails
    Python::with_gil(|py| {
        py.import("serial")?;
        Ok(())
    }
}

Error: Error: PyErr { type: <class 'ModuleNotFoundError'>, value: ModuleNotFoundError("No module named 'serial'"), traceback: None }

ffi::Py_SetPythonHome

Trying PyO3/pyo3#1554 (comment)

use pyo3::prelude::*;
use std::process::Command;

pub fn initialize_python() -> PyResult<()> {
    // Due to https://github.com/ContinuumIO/anaconda-issues/issues/11439,
    // we first need to set PYTHONHOME. To do so, we will look for whatever
    // directory on PATH currently has python.exe.
    let python_exe = which::which("python").unwrap();
    let python_home = python_exe.parent().unwrap();

    // The Python C API uses null-terminated UTF-16 strings, so we need to
    // encode the path into that format here.
    // We could use the Windows FFI modules provided in the standard library,
    // but we want this to work cross-platform, so we do things more manually.
    let mut python_home = python_home
        .to_str()
        .unwrap()
        .encode_utf16()
        .collect::<Vec<u16>>();
    python_home.push(0);
    unsafe {
        pyo3::ffi::Py_SetPythonHome(python_home.as_ptr());
    }

    // Once we've set the configuration we need, we can go on and manually
    // initialize PyO3.
    pyo3::prepare_freethreaded_python();

    Ok(())
}

fn main() -> PyResult<()> {
    // Module 'serial' (pip install pyserial) is installed in the virtualenv but NOT the global python

    // Sanity check that the python in my path is correct
    {
        let cmd = Command::new("python")
            .args(&[
                "-c",
                "import sys; print(sys.version); import serial; print(serial)",
            ])
            .output()
            .unwrap();
        println!("{}", String::from_utf8_lossy(&cmd.stdout));
    }

    initialize_python()?;

    Python::with_gil(|py| {
        let sys = py.import("sys")?;

        println!("{}", sys.getattr("version")?);

        // This fails
        py.import("serial")?;

        Ok(())
    })
}

Output:

     Running `target\debug\pyo3-venv-test.exe`
3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)]
<module 'serial' from 'C:\\Users\\kscot\\Projects\\temp\\pyo3-venv-test\\.env\\lib\\site-packages\\serial\\__init__.py'>       

Python path configuration:
  PYTHONHOME = 'C:\Users\kscot\Projects\temp\pyo3-venv-test\.env\Scripts'
  PYTHONPATH = (not set)
  program name = 'python'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = 'C:\\Users\\kscot\\Projects\\temp\\pyo3-venv-test\\target\\debug\\pyo3-venv-test.exe'
  sys.base_prefix = 'C:\\Users\\kscot\\Projects\\temp\\pyo3-venv-test\\.env\\Scripts'
  sys.base_exec_prefix = 'C:\\Users\\kscot\\Projects\\temp\\pyo3-venv-test\\.env\\Scripts'
  sys.platlibdir = 'lib'
  sys.executable = 'C:\\Users\\kscot\\Projects\\temp\\pyo3-venv-test\\target\\debug\\pyo3-venv-test.exe'
  sys.prefix = 'C:\\Users\\kscot\\Projects\\temp\\pyo3-venv-test\\.env\\Scripts'
  sys.exec_prefix = 'C:\\Users\\kscot\\Projects\\temp\\pyo3-venv-test\\.env\\Scripts'
  sys.path = [
    'C:\\Users\\kscot\\AppData\\Local\\Programs\\Python\\Python39\\python39.zip',
    'C:\\Users\\kscot\\Projects\\temp\\pyo3-venv-test\\.env\\Scripts\\DLLs',
    'C:\\Users\\kscot\\Projects\\temp\\pyo3-venv-test\\.env\\Scripts\\lib',
    'C:\\Users\\kscot\\Projects\\temp\\pyo3-venv-test\\target\\debug',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x000058e0 (most recent call first):
<no Python frame>
error: process didn't exit successfully: `target\debug\pyo3-venv-test.exe` (exit code: 1)

PYO3_PYTHON

Setting PYO3_PYTHON as suggested: PyO3/pyo3#1554 (comment)

$ PYO3_PYTHON=python cargo run

It does not seem to have any effect with either code snippets above.