Piping Unicode to Python From Powershell

Readme.Utf8 in Python.md

Using utf8 in Python

Env vars control unspecified encodings

• locale.GetPrerredEncoding()
• Env var PYTHONIOENCODING

manually forcing utf8 without Env Vars

python 3.7+ using Reconfigure

reconfigure uses python 3.7, but is more robust https://docs.python.org/3/library/io.html#io.TextIOWrapper.reconfigure

sys.stdin.reconfigure(encoding="utf8", errors="strict") # or ignore or replace

Python before 3.7

Before reconfigure, you may have to change the stream before it is used. If you've already printed, it may not change. reconfigure works either way

input_stream = io.TextIOWrapper(sys.stdin.buffer, encoding='utf-8')

Piping or invoking from powershell

$Env:PYTHONIOENCODING = 'utf-8:strict'

python read_pipe.py test

ls | python read_pipe.py test

read_pipe.py

import os, sys


def GetEncoding() -> None:
    """set your"""
    print(
        f"""
        stdin:  {sys.stdin.encoding}
        stdout: {sys.stdout.encoding}
        stderr: {sys.stderr.encoding}
    """
    )


def ForceUtf8(UseReconfigure: bool = True) -> None:
    """
    reconfigure uses python 3.7, but is more robust
        https://docs.python.org/3/library/io.html#io.TextIOWrapper.reconfigure
    """
    if UseReconfigure:
        sys.stdin.reconfigure(encoding="utf8", errors="replace")
    else:
        raise NotImplementedError("...")


def main() -> None:

    if sys.argv[1] == "forceutf8":
        ForceUtf8()

    GetEncoding()

    if sys.argv[1] == "test":
        return

    for line in sys.stdin.readlines():
        print(f"line: {line}", end="")

    print("done")


if __name__ == "__main__":
    main()

run_test Pipe Utf8 to Python.ps1

function test3([string]$mode) {
    switch($mode) {
        'test' {
            $pyArgs = 'test'
        }
        'forceutf8' {
            $pyArgs = 'forceutf8'
        }
        default {
            $pyArgs = ''
        }
    }
    $pyArgs

    'running python alone'
    '> python .\read_pipe.py $pyArgs'

    python .\read_pipe.py $pyArgs

    'piping from powershell'
    '> "1" | python .\read_pipe.py $pyArgs'
    '1' | python .\read_pipe.py $pyArgs

    'piping (python to python) from inside powershell'

    '> python -c "print(1)" | python .\read_pipe.py $pyArgs'
    python -c 'print(1)' | python .\read_pipe.py $pyArgs
}


& {
    'test', 'forceutf8', '' | ForEach-Object {
        $mode = $_
        "`n" * 4
        write-host -ForegroundColor magenta $mode

        $env:PYTHONIOENCODING = "utf-8:replace"
        'Env:PythonIOEncoding = ', $env:PYTHONIOENCODING -join ''

        test3 $mode

        "`n" * 2

        $env:PYTHONIOENCODING = $null
        'Env:PythonIOEncoding = ', $env:PYTHONIOENCODING -join ''
        test3 $mode
    }

}

write_pipe.ps1

$monkey = "🐒"
$monkey