Skip to content

Instantly share code, notes, and snippets.

@jwiggins
Created March 16, 2018 10:15
Show Gist options
  • Save jwiggins/809204413ce350c6c4a68359434e5f68 to your computer and use it in GitHub Desktop.
Save jwiggins/809204413ce350c6c4a68359434e5f68 to your computer and use it in GitHub Desktop.
Windows PyTables & PyInstaller Failure

Freezing a very simple script fails with PyTables 3.3.0 and PyInstaller 3.3.1 on Windows 10.

To create a test environment using the EDM tool:

C:\Enthought\frozen-tables>edm install pytables pywin32 -e frozen_tables
C:\Enthought\frozen-tables>edm run -e frozen_tables -- pip install pyinstaller

Some debugging info:

C:\Enthought\frozen-tables>edm run -e frozen_tables -- python --version
Python 2.7.13 -- Enthought, Inc. (x86_64)

C:\Enthought\frozen-tables>edm run -e frozen_tables -- pip list
altgraph (0.15)
appdirs (1.4.3)
dis3 (0.1.2)
distribute-remove (1.0.0)
future (0.16.0)
macholib (1.9)
MKL (2017.0.3)
numexpr (2.6.2)
numpy (1.13.3)
packaging (16.8)
pefile (2017.11.5)
pip (9.0.1)s
PyInstaller (3.3.1)
pyparsing (2.2.0)
pytables (3.3.0)
pywin32 (220)
setuptools (38.2.5)
six (1.10.0)

Create a freeze.py script which imports PyTables and calls the print_versions function. It's output:

C:\Enthought\frozen-tables>edm run -e frozen_tables -- python freeze.py
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
PyTables version:    3.3.0
HDF5 version:        1.8.16
NumPy version:       1.13.3
Numexpr version:     2.6.2 (using VML/MKL 2017.0.3)
Zlib version:        1.2.8 (in Python interpreter)
Blosc version:       1.11.1 (2016-09-03)
Blosc compressors:   blosclz (1.0.5), lz4 (1.7.2), lz4hc (1.7.2), snappy (1.1.1), zlib (1.2.8), zstd (1.0.0)
Blosc filters:       shuffle, bitshuffle
Python version:      2.7.13 |Enthought, Inc. (x86_64)| (default, Mar  2 2017, 16:05:12) [MSC v.1500 64 bit (AMD64)]
Platform:            Windows-10-10.0.16299
Byte-ordering:       little
Detected cores:      4
Default encoding:    ascii
Default FS encoding: mbcs
Default locale:      (en_US, cp1252)

Then freeze the script with --onedir or --onefile. Both versions fail, each for a slightly different reason.

The --onefile version fails for reasons which I have not been able to figure out. Ostensibly, it's failing to load the hdf5.dll library. The file is present, and if I insert a breakpoint into the PyInstaller ctypes.cdll machinery, I see that the actual error from LoadLibrary is: WindowsError: [Error 126] The specified module could not be found. I find that error confusing, to say the least. Calling os.path.exists on the same path returns True. While the script is blocked in the Python debugger, I made a copy of the hdf5.dll library within and I see nothing wrong when I examine it using Dependency Walker.

The --onedir version fails because the PyInstaller ctypes.cdll machinery raises an exception when a library is not found, rather than returning None as is expected by code running against an unpatched ctypes. Namely, the checks for lzo2.dll and libbz2.dll will fail because the files are not present (causing an exception in the frozen script). Examining the affected code: https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L76-L86 we see that PyTables is OK with getting a None back from LoadLibrary, so the files don't actually have to be present.

The problem can be resolved by skipping the LoadLibrary sanity check in PyTables, as suggested in PyTables/PyTables#177.

import tables
tables.print_versions()
C:\Enthought\frozen-tables>dist\freeze\freeze.exe
Traceback (most recent call last):
File "freeze.py", line 1, in <module>
import tables
File "C:\Enthought\pythons\envs\frozen_tables\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 396, in load_module
exec(bytecode, module.__dict__)
File "site-packages\tables\__init__.py", line 82, in <module>
File "site-packages\tables\__init__.py", line 53, in _load_library
File "ctypes\__init__.py", line 440, in LoadLibrary
File "site-packages\PyInstaller\loader\pyiboot01_bootstrap.py", line 151, in __init__
__main__.PyInstallerImportError: Failed to load dynlib/dll 'C:\\ENTHOU~1\\FROZEN~1\\dist\\freeze\\tables\\lzo2.dll'. Most probably this dynlib/dll was not found when the application was frozen.
[2248] Failed to execute script freeze
C:\Enthought\frozen-tables>edm run -e frozen_tables -- pyi-archive_viewer dist\freeze.exe
pos, length, uncompressed, iscompressed, type, name
[(0, 193, 262, 1, 'm', u'struct'),
(193, 1154, 2676, 1, 'm', u'pyimod01_os_path'),
(1347, 4350, 12327, 1, 'm', u'pyimod02_archive'),
(5697, 7381, 22957, 1, 'm', u'pyimod03_importers'),
(13078, 1838, 5263, 1, 's', u'pyiboot01_bootstrap'),
(14916, 230, 313, 1, 's', u'pyi_rth_pkgres'),
(15146, 446, 706, 1, 's', u'pyi_rth_win32comgenpy'),
(15592, 1104, 2282, 1, 's', u'pyi_rth_multiprocessing'),
(16696, 115, 147, 1, 's', u'freeze'),
(16811, 546, 1052, 1, 'b', u'Microsoft.VC90.CRT.manifest'),
(17357, 575, 1141, 1, 'b', u'Microsoft.VC90.MFC.manifest'),
(17932, 572469, 1284608, 1, 'b', u'_bsddb.pyd'),
(590401, 45910, 113152, 1, 'b', u'_ctypes.pyd'),
(636311, 733864, 1583104, 1, 'b', u'_hashlib.pyd'),
(1370175, 13463, 32256, 1, 'b', u'_multiprocessing.pyd'),
(1383638, 20401, 47616, 1, 'b', u'_socket.pyd'),
(1404039, 1007213, 2255872, 1, 'b', u'_ssl.pyd'),
(2411252, 2940, 7168, 1, 'b', u'_win32sysloader.pyd'),
(2414192, 38964, 80896, 1, 'b', u'bz2.pyd'),
(2453156, 465, 1014, 1, 'b', u'freeze.exe.manifest'),
(2453621, 896691, 2319872, 1, 'b', u'hdf5.dll'),
(3350312, 2363725, 5079040, 1, 'b', u'mfc90.dll'),
(5714037, 2366712, 5097472, 1, 'b', u'mfc90u.dll'),
(8080749, 25103, 67072, 1, 'b', u'mfcm90.dll'),
(8105852, 25084, 67072, 1, 'b', u'mfcm90u.dll'),
(8130936, 9340343, 25369360, 1, 'b', u'mk2_core.dll'),
(17471279, 8470864, 24506640, 1, 'b', u'mk2_intel_thread.dll'),
(25942143, 539467, 1339648, 1, 'b', u'mk2iomp5md.dll'),
(26481610, 71004, 245760, 1, 'b', u'msvcm90.dll'),
(26552614, 219630, 854168, 1, 'b', u'msvcp90.dll'),
(26772244, 299667, 642200, 1, 'b', u'msvcr90.dll'),
(27071911, 81431, 354816, 1, 'b', u'numexpr.interpreter.pyd'),
(27153342, 492602, 1590272, 1, 'b', u'numpy.core.multiarray.pyd'),
(27645944, 207440, 828416, 1, 'b', u'numpy.core.umath.pyd'),
(27853384, 32859, 67072, 1, 'b', u'numpy.fft.fftpack_lite.pyd'),
(27886243, 58734, 176640, 1, 'b', u'numpy.linalg._umath_linalg.pyd'),
(27944977, 13573, 36864, 1, 'b', u'numpy.linalg.lapack_lite.pyd'),
(27958550, 223238, 701440, 1, 'b', u'numpy.random.mtrand.pyd'),
(28181788, 64522, 164352, 1, 'b', u'pyexpat.pyd'),
(28246310, 1271509, 3034112, 1, 'b', u'python27.dll'),
(29517819, 178928, 547328, 1, 'b', u'pythoncom27.dll'),
(29696747, 48931, 136704, 1, 'b', u'pywintypes27.dll'),
(29745678, 5087, 10752, 1, 'b', u'select.pyd'),
(29750765, 33801, 74240, 1, 'b', u'tables._comp_bzip2.pyd'),
(29784566, 7315, 15360, 1, 'b', u'tables._comp_lzo.pyd'),
(29791881, 342550, 896512, 1, 'b', u'tables.hdf5extension.pyd'),
(30134431, 41395, 122368, 1, 'b', u'tables.indexesextension.pyd'),
(30175826, 21743, 50176, 1, 'b', u'tables.linkextension.pyd'),
(30197569, 37435, 90624, 1, 'b', u'tables.lrucacheextension.pyd'),
(30235004, 314545, 815616, 1, 'b', u'tables.tableextension.pyd'),
(30549549, 314039, 821248, 1, 'b', u'tables.utilsextension.pyd'),
(30863588, 257934, 689664, 1, 'b', u'unicodedata.pyd'),
(31121522, 48725, 129536, 1, 'b', u'win32api.pyd'),
(31170247, 153591, 492544, 1, 'b', u'win32com.shell.shell.pyd'),
(31323838, 11661, 28672, 1, 'b', u'win32pdh.pyd'),
(31335499, 11066, 26624, 1, 'b', u'win32pipe.pyd'),
(31346565, 6707, 16896, 1, 'b', u'win32trace.pyd'),
(31353272, 305049, 1035264, 1, 'b', u'win32ui.pyd'),
(31658321, 11749, 28672, 1, 'b', u'win32wnet.pyd'),
(31670070, 41868, 80384, 1, 'b', u'zlib1.dll'),
(31711938,
0,
0,
0,
'o',
u'pyi-windows-manifest-filename freeze.exe.manifest'),
(31711938, 6819, 22612, 1, 'x', u'Include\\pyconfig.h'),
(31718757, 3663621, 3663621, 0, 'z', u'out00-PYZ.pyz')]
C:\Enthought\frozen-tables>dist\freeze.exe
[4576] PyInstaller Bootloader 3.x
[4576] LOADER: executable is C:\Enthought\frozen-tables\dist\freeze.exe
[4576] LOADER: homepath is C:\Enthought\frozen-tables\dist
[4576] LOADER: _MEIPASS2 is NULL
[4576] LOADER: archivename is C:\Enthought\frozen-tables\dist\freeze.exe
[4576] LOADER: Extracting binaries
[4576] LOADER: Executing self as child
[4576] LOADER: set _MEIPASS2 to C:\Users\JWIGGINS\AppData\Local\Temp\12\_MEI45762
[4576] LOADER: Setting up to run child
[4576] LOADER: Creating child process
[4576] LOADER: Waiting for child process to finish...
[12020] PyInstaller Bootloader 3.x
[12020] LOADER: executable is C:\Enthought\frozen-tables\dist\freeze.exe
[12020] LOADER: homepath is C:\Enthought\frozen-tables\dist
[12020] LOADER: _MEIPASS2 is C:\Users\JWIGGINS\AppData\Local\Temp\12\_MEI45762
[12020] LOADER: archivename is C:\Enthought\frozen-tables\dist\freeze.exe
[12020] LOADER: SetDllDirectory(C:\Users\JWIGGINS\AppData\Local\Temp\12\_MEI45762)
[12020] LOADER: Already in the child - running user's code.
[12020] LOADER: manifestpath: C:\Users\JWIGGINS\AppData\Local\Temp\12\_MEI45762\freeze.exe.manifest
[12020] LOADER: Activation context created
[12020] LOADER: Activation context activated
[12020] LOADER: Python library: C:\Users\JWIGGINS\AppData\Local\Temp\12\_MEI45762\python27.dll
[12020] LOADER: Loaded functions from Python library.
[12020] LOADER: Manipulating environment (sys.path, sys.prefix)
[12020] LOADER: sys.prefix is C:\Users\JWIGGINS\AppData\Local\Temp\12\_MEI45~1
[12020] LOADER: Setting runtime options
[12020] LOADER: Bootloader option: pyi-windows-manifest-filename freeze.exe.manifest
[12020] LOADER: Initializing python
[12020] LOADER: Overriding Python's sys.path
[12020] LOADER: Post-init sys.path is C:\Users\JWIGGINS\AppData\Local\Temp\12\_MEI45762
[12020] LOADER: Setting sys.argv
[12020] LOADER: setting sys._MEIPASS
[12020] LOADER: importing modules from CArchive
[12020] LOADER: extracted struct
[12020] LOADER: callfunction returned...
[12020] LOADER: extracted pyimod01_os_path
[12020] LOADER: callfunction returned...
[12020] LOADER: extracted pyimod02_archive
[12020] LOADER: callfunction returned...
[12020] LOADER: extracted pyimod03_importers
[12020] LOADER: callfunction returned...
[12020] LOADER: Installing PYZ archive with Python modules.
[12020] LOADER: PYZ archive: out00-PYZ.pyz
[12020] LOADER: Running pyiboot01_bootstrap.py
[12020] LOADER: Running pyi_rth_pkgres.py
[12020] LOADER: Running pyi_rth_win32comgenpy.py
[12020] LOADER: Running pyi_rth_multiprocessing.py
[12020] LOADER: Running freeze.py
Traceback (most recent call last):
File "freeze.py", line 1, in <module>
import tables
File "C:\Enthought\pythons\envs\frozen_tables\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 396, in load_module
exec(bytecode, module.__dict__)
File "site-packages\tables\__init__.py", line 69, in <module>
File "site-packages\tables\__init__.py", line 53, in _load_library
File "ctypes\__init__.py", line 440, in LoadLibrary
File "site-packages\PyInstaller\loader\pyiboot01_bootstrap.py", line 151, in __init__
__main__.PyInstallerImportError: Failed to load dynlib/dll 'C:\\Users\\JWIGGINS\\AppData\\Local\\Temp\\12\\_MEI45~1\\hdf5.dll'. Most probably this dynlib/dll was not found when the application was frozen.
[12020] Failed to execute script freeze
[12020] LOADER: OK.
[12020] LOADER: Cleaning up Python interpreter.
[4576] LOADER: Back to parent (RC: -1)
[4576] LOADER: Doing cleanup
[4576] LOADER: Freeing archive status for C:\Enthought\frozen-tables\dist\freeze.exe
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment