CPython dev sprint 2017: Startup speed: lazy loading for modules, supported by AST static analysis

gistfile1.txt

See:
    https://public.etherpad-mozilla.org/p/cpython-dev-sprint-2017
    https://github.com/warsaw/lazyimport
    https://mail.python.org/pipermail/python-ideas/2017-September/046957.html

Introduce a lazy module import process that modules can opt-in to.
The opt-in would either be with a __future__ statement or the
compiler would statically analyze the module and determine if it is
safe.  E.g. if the module has no module level statements besides
imports.

.pyc files get some other bits of information:

A) whether the module has opted for lazy import (LAZY_SAFE)

B) the modules imported by the module (i.e. top-level imports,
IMPORT_LIST)

Make __import__ understand this data and do lazy loading for modules
that want it.  Sub-modules that have import side-effects will still
execute as normal and the side effects will happen when the parent
module is imported..

This would consist of a recursive process, something like:

def load_module(name):
    if not LAZY_SAFE(name):
        import as usual
    else:
        create lazy version of module 'name'
        for subname in IMPORT_LIST(name):
            load_module(subname)
            
An additional idea from Barry W, if a module wants lazy loading but
wants to do some init when the module is "woken up", define a
__init__ top-level function.  Python would call that function when
attributes of the module are first actually used.

My plan was to implement this with a Python __import__
implementation. I would unmarshal the .pyc, compute LAZY_SAFE and
IMPORT_LIST at import time.  So, not gaining a lot of speedup.  It
would prove if the idea works in terms of not causing application
crashes, etc.  I could try running it with bigger apps and see how
many modules are flagged for lazy loading.