hartzell/configparser-is-broken.md

## configparser-is-broken.md

      
    Raw
  

              configparser-is-broken.md
            
          
    Importing configparser

The backported configparser is a P.I.T.A..
I tripped across this when trying to use py-jupyter-notebook with my
Spack PR that adds
dependencies via site.addsitedir().  It fails immediately.  That package
appears to work if you use "normal" Spack and PYTHONPATH, but it turns
out that the only bit that uses configparser is jupyter-nbconvert and
that does fail.
TL;DR is that it fails quickly in PR #8436 because configparser's .pth file
is processed and breaks the backports namespace for every other package that
uses it.  In normal Spack w/ PYTHONPATH configparser's .pth file is never
processed so the other backport packages work but when the code actually gets
around to trying to use configparser if fails.
You can't import configparser via PYTHONPATH

In a standard Spack, with lmod modules...
Simply importing doesn't work

Adding the directory to PYTHONPATH and trying to import it doesn't
work:
(alice)[15:27:28]spack-plain>>module purge
(alice)[15:27:40]spack-plain>>module load py-configparser
(alice)[15:27:47]spack-plain>>python
Python 2.7.15 (default, Jun 19 2018, 12:32:41)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import configparser
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/hartzell/tmp/spack-plain/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-configparser-3.5.0-hcll7pv4hkl46st6vzswyrcedpq6r7vt/lib/python2.7/site-packages/configparser.py", line 12, in <module>
    from backports.configparser import (
ImportError: No module named backports.configparser
>>> from backports import configparser
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named backports
>>>

pylint falls back to six.moves => works with PYTHONPATH

pylint works in a Spack environment that's tying things together with
PYTHONPATH  because when import configparser fails it just imports
from six.moves (which aliases it to the old-school ConfigParser)
Here's the relevant bit from pylint:
 try:
     import configparser
 except ImportError:
     from six.moves import configparser
Here's the relevant bit from
six.moves.
For the record:

ConfigParser is the Python 2 thing.
configparser is the Python 3 thing, in addition to the name change
the functionality was improved (in 3.2?).
the configparser package (which should be
"backports-configparser"...) makes the updated version available to
Python 2.

configparser works when its dir is added via site.addsitedir()

site.addsitedir()
provides an alternative to using PYTHONPATH to add directories to
sys.path.  Its documentation says:

Add a directory to sys.path and process its .pth files. Typically
used in sitecustomize or usercustomize (see above).

configparser installs a .pth file.
.../opt/spack/.../py-configparser-3.5.0-.../lib/python2.7/site-packages/configparser-3.5.0-py2.7-nspkg.pth

If configparser's directory is added to sys.path via
site.addsitedir() then it can be imported.
Here's an example of loading it:
(alice)[15:49:42]spack-plain>>module purge
(alice)[15:49:44]spack-plain>>module load py-configparser
(alice)[15:49:48]spack-plain>>printenv PYTHONPATH
/Users/hartzell/tmp/spack-plain/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-configparser-3.5.0-hcll7pv4hkl46st6vzswyrcedpq6r7vt/lib/python2.7/site-packages
(alice)[15:49:49]spack-plain>>module purge
(alice)[15:49:51]spack-plain>>python
Python 2.7.15 (default, May  1 2018, 16:44:08)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import site
>>> site.addsitedir("/Users/hartzell/tmp/spack-plain/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-configparser-3.5.0-hcll7pv4hkl46st6vzswyrcedpq6r7vt/lib/python2.7/site-packages")
>>> import configparser
>>>

You can't use addsitedir() reliably

I have a Spack PR that
writes a package's dependencies to a file and provides a
sitecustomize.py that reads that file when a script starts up and
loads the dirs via site.addsitedir().  It allows e.g. flake8 to
work without any environment variables.
Sadly, other packages don't work.  E.g. py-jupyter-notebook.
It turns out that there are two different ways to create namespace
packages in Python2 (plus a different one in Python3): pkgutil-style
and pkg_resources-style.
Details
here
You cannot mix packages that use the different techniques.


There's a backports package that does nothing other than claim
the namespace.  It's pkgutil-style.


Many (most?) "backports" packages,
e.g. (backports-shutil-get-terminal-size) are pkgutil-style.


configparser is pkg_resources-style.


SIGH.
In an addsitedir() system...

Here are some demos, using the system in my PR.
If you're not familiar with the PR, scripts figure out the directory
that they started in and look for a file named
../.spack/python-sitedirs.  If it exists, they pass each line in it
to site.addsitedir.
When configparser's directory has been added via addsitedir, the
backports modules appears to be "built-in":
(alice)[16:13:13]bin>>cat ../.spack/python-sitedirs
/Users/hartzell/tmp/spack-rpaths/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-configparser-3.5.0-hcll7pv4hkl46st6vzswyrcedpq6r7vt
/Users/hartzell/tmp/spack-rpaths/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-backports-shutil-get-terminal-size-1.0.0-smyswe3gg4khpsinkiflyrhz3ra4zs5o
(alice)[16:17:20]bin>>cat foo
#!/bin/bash /Users/hartzell/tmp/spack-rpaths/bin/sbang
#!/Users/hartzell/tmp/spack-rpaths/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/python-2.7.15-fueghvuvuid6cw43s2eqovgyauoc73ii/bin/python2.7

import backports

print(backports)
(alice)[16:17:20]bin>>$(pwd)/foo
<module 'backports' (built-in)>

But if you don't add the configparser dir, then the backports
module points to the directory that contains it.
(alice)[16:13:13]bin>>cat ../.spack/python-sitedirs
/Users/hartzell/tmp/spack-rpaths/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-backports-shutil-get-terminal-size-1.0.0-smyswe3gg4khpsinkiflyrhz3ra4zs5o
(alice)[16:15:44]bin>>$(pwd)/foo
<module 'backports' from '/Users/hartzell/tmp/spack-rpaths/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-backports-shutil-get-terminal-size-1.0.0-smyswe3gg4khpsinkiflyrhz3ra4zs5o/lib/python2.7/site-packages/backports/__init__.pyc'>

Here's how that plays out with something that's actually trying to
import a real piece of code.
(alice)[16:13:13]bin>>cat ../.spack/python-sitedirs
/Users/hartzell/tmp/spack-rpaths/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-configparser-3.5.0-hcll7pv4hkl46st6vzswyrcedpq6r7vt
/Users/hartzell/tmp/spack-rpaths/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-backports-shutil-get-terminal-size-1.0.0-smyswe3gg4khpsinkiflyrhz3ra4zs5o
(alice)[16:13:22]bin>>$(pwd)/demo
Traceback (most recent call last):
  File "/Users/hartzell/tmp/spack-rpaths/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-jupyter-notebook-4.2.3-l4hlvjv7ce64xantfcnfiga5jnr546wr/bin/demo", line 5, in <module>
    from backports.shutil_get_terminal_size import get_terminal_size as _get_terminal_size
ImportError: No module named shutil_get_terminal_size
(alice)[16:13:27]bin>># remove configparser line from ../.spack/python-sitedirs
(alice)[16:13:39]bin>>cat ../.spack/python-sitedirs
/Users/hartzell/tmp/spack-rpaths/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-backports-shutil-get-terminal-size-1.0.0-smyswe3gg4khpsinkiflyrhz3ra4zs5o
(alice)[16:13:43]bin>>$(pwd)/demo
(alice)[16:13:45]bin>>