nascheme

/* run using:

 $ sudo dtrace -q -s module_import_trace.d -c './python.exe -c True'
 
*/
self int indent;
self int traceit;

python$target:::module-exec-start
{

nascheme

!/usr/bin/env python3
#
# Pretty print module import dtrace output.  First column is elapsed time in ms.  Second column is % of total time.
import sys
import fileinput

def main():
    first = None
    lines = []
    for line in fileinput.input():

nascheme

import sys
import ast
import importlib.util

def safe_assign(value):
    safe = {ast.Num, ast.NameConstant, ast.Str}
    return type(value) in safe

class Transformer(ast.NodeTransformer):
    def __init__(self, *args, **kwargs):

nascheme

See:
    https://public.etherpad-mozilla.org/p/cpython-dev-sprint-2017
    https://github.com/warsaw/lazyimport
    https://mail.python.org/pipermail/python-ideas/2017-September/046957.html

Introduce a lazy module import process that modules can opt-in to.
The opt-in would either be with a __future__ statement or the
compiler would statically analyze the module and determine if it is
safe.  E.g. if the module has no module level statements besides
imports.

nascheme

See:
  https://public.etherpad-mozilla.org/p/cpython-dev-sprint-2017
  https://github.com/warsaw/lazyimport/blob/master/lazy_compile.py
  https://github.com/warsaw/lazyimport/blob/master/lazy_helper.py

This idea is based on a comment from Larry Hastings.  PHP got a good
speedup by not creating all functions defined in the source.  Python could
doso mething similar for classes and functions.  Perhaps without not too
much backwards compatibility problems.  This would be a huge win for
startup and memory usage of command-line tools that use large libraries

nascheme

See:
  https://public.etherpad-mozilla.org/p/cpython-dev-sprint-2017
  https://mail.python.org/pipermail/python-dev/2017-September/149307.html

Motivation
----------

Python objects that participate in cyclic GC (things like lists, dicts,
sets but not strings, ints and floats) have extra memory overhead.  I
think it is possible to mostly eliminate this overhead.  Also, while

nascheme

See:
  https://public.etherpad-mozilla.org/p/cpython-dev-sprint-2017
  
- background: https://lwn.net/Articles/730915/

- profiling of import

- DTrace probes for module import start/done
    https://github.com/nascheme/cpython/tree/dtrace-module-import
    https://gist.github.com/nascheme/c1cece36a3369926ee93cecc3d024179

nascheme

probe process("./python").mark("import__find__load__start") {
	modname = user_string($arg1);
	printf("%s import start %s\n",
			thread_indent(1), modname);
}

probe process("./python").mark("import__find__load__done") {
	modname = user_string($arg1);
	printf("%s import done %s\n",
			thread_indent(1), modname);

nascheme

     0 python(6402): => <module> in <frozen importlib._bootstrap>:25
    11 python(6402):  => _DeadlockError in <frozen importlib._bootstrap>:48
    14 python(6402):  <= _DeadlockError in <frozen importlib._bootstrap>:49
    43 python(6402):  => _ModuleLock in <frozen importlib._bootstrap>:52
    46 python(6402):  <= _ModuleLock in <frozen importlib._bootstrap>:116
    64 python(6402):  => _DummyModuleLock in <frozen importlib._bootstrap>:120
    66 python(6402):  <= _DummyModuleLock in <frozen importlib._bootstrap>:137
    80 python(6402):  => _ModuleLockManager in <frozen importlib._bootstrap>:141
    82 python(6402):  <= _ModuleLockManager in <frozen importlib._bootstrap>:151
    97 python(6402):  => _installed_safely in <frozen importlib._bootstrap>:305

nascheme

py-base
=======

Performance version: 0.6.1
Report on Linux-4.9.0-4-amd64-x86_64-with-debian-buster-sid
Number of logical CPUs: 4
Start date: 2017-12-02 17:30:30.244152
End date: 2017-12-02 17:37:46.889962

py-unwind