Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Debugging pyo3 virtualenv issues

I'm running Windows 10. I am using Git-bash, but the same issues happen when run from Powershell.

I have tried Python 3.9 and 3.7, fresh installs.

pyo3 Gitter discussion: https://gitter.im/PyO3/Lobby?at=6143ba0b7cd57813a8cbc355

Minimal example

use pyo3::prelude::*;
use std::process::Command;

fn main() -> PyResult<()> {
    // Module 'serial' (pip install pyserial) is installed in the virtualenv but NOT the global python

    // This works
    let cmd = Command::new("python")
        .args(&["-c", "import serial; print(serial)"])
        .output()
        .unwrap();
    println!("{}", String::from_utf8_lossy(&cmd.stdout));

    // This fails
    Python::with_gil(|py| {
        py.import("serial")?;
        Ok(())
    }
}

Error: Error: PyErr { type: <class 'ModuleNotFoundError'>, value: ModuleNotFoundError("No module named 'serial'"), traceback: None }

ffi::Py_SetPythonHome

Trying https://github.com/PyO3/pyo3/issues/1554#issuecomment-850702544

use pyo3::prelude::*;
use std::process::Command;

pub fn initialize_python() -> PyResult<()> {
    // Due to https://github.com/ContinuumIO/anaconda-issues/issues/11439,
    // we first need to set PYTHONHOME. To do so, we will look for whatever
    // directory on PATH currently has python.exe.
    let python_exe = which::which("python").unwrap();
    let python_home = python_exe.parent().unwrap();

    // The Python C API uses null-terminated UTF-16 strings, so we need to
    // encode the path into that format here.
    // We could use the Windows FFI modules provided in the standard library,
    // but we want this to work cross-platform, so we do things more manually.
    let mut python_home = python_home
        .to_str()
        .unwrap()
        .encode_utf16()
        .collect::<Vec<u16>>();
    python_home.push(0);
    unsafe {
        pyo3::ffi::Py_SetPythonHome(python_home.as_ptr());
    }

    // Once we've set the configuration we need, we can go on and manually
    // initialize PyO3.
    pyo3::prepare_freethreaded_python();

    Ok(())
}

fn main() -> PyResult<()> {
    // Module 'serial' (pip install pyserial) is installed in the virtualenv but NOT the global python

    // Sanity check that the python in my path is correct
    {
        let cmd = Command::new("python")
            .args(&[
                "-c",
                "import sys; print(sys.version); import serial; print(serial)",
            ])
            .output()
            .unwrap();
        println!("{}", String::from_utf8_lossy(&cmd.stdout));
    }

    initialize_python()?;

    Python::with_gil(|py| {
        let sys = py.import("sys")?;

        println!("{}", sys.getattr("version")?);

        // This fails
        py.import("serial")?;

        Ok(())
    })
}

Output:

     Running `target\debug\pyo3-venv-test.exe`
3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)]
<module 'serial' from 'C:\\Users\\kscot\\Projects\\temp\\pyo3-venv-test\\.env\\lib\\site-packages\\serial\\__init__.py'>       

Python path configuration:
  PYTHONHOME = 'C:\Users\kscot\Projects\temp\pyo3-venv-test\.env\Scripts'
  PYTHONPATH = (not set)
  program name = 'python'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = 'C:\\Users\\kscot\\Projects\\temp\\pyo3-venv-test\\target\\debug\\pyo3-venv-test.exe'
  sys.base_prefix = 'C:\\Users\\kscot\\Projects\\temp\\pyo3-venv-test\\.env\\Scripts'
  sys.base_exec_prefix = 'C:\\Users\\kscot\\Projects\\temp\\pyo3-venv-test\\.env\\Scripts'
  sys.platlibdir = 'lib'
  sys.executable = 'C:\\Users\\kscot\\Projects\\temp\\pyo3-venv-test\\target\\debug\\pyo3-venv-test.exe'
  sys.prefix = 'C:\\Users\\kscot\\Projects\\temp\\pyo3-venv-test\\.env\\Scripts'
  sys.exec_prefix = 'C:\\Users\\kscot\\Projects\\temp\\pyo3-venv-test\\.env\\Scripts'
  sys.path = [
    'C:\\Users\\kscot\\AppData\\Local\\Programs\\Python\\Python39\\python39.zip',
    'C:\\Users\\kscot\\Projects\\temp\\pyo3-venv-test\\.env\\Scripts\\DLLs',
    'C:\\Users\\kscot\\Projects\\temp\\pyo3-venv-test\\.env\\Scripts\\lib',
    'C:\\Users\\kscot\\Projects\\temp\\pyo3-venv-test\\target\\debug',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x000058e0 (most recent call first):
<no Python frame>
error: process didn't exit successfully: `target\debug\pyo3-venv-test.exe` (exit code: 1)

PYO3_PYTHON

Setting PYO3_PYTHON as suggested: https://github.com/PyO3/pyo3/issues/1554#issuecomment-919875755

$ PYO3_PYTHON=python cargo run

It does not seem to have any effect with either code snippets above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment