gistfile1.txt

Submitted by
Matthias Bussonnier
Description
In the recent release of IPython 7.0 added the ability to have "top level" async code, so that code which usually does not work in Pure Python work out of the box:

>>>  await asyncio.sleep(1)
>>>
What can you do with this feature ? How can this not be a syntax error as it is in pure Python ? How can it do what user expect, regardless of whether you use asyncio, trio, or curio ?

Quick prototyping is one of the strength of Python, and an async REPL allows fast exploration of the asynchronous programing paradigm. At first glance it is astonishing that is is not available (yet) in the classic Python REPL.

The implementation of such a feature is tricky – but highly interesting if you want to know about the internals of Python – and underline that there is a tension in the CPython API between Python as an application language (write ahead of time and run), vs Python as an exploratory language (Think data-science and interacting with a REPL and a dataset).

We'll quickly see how we implemented the above:

>>> code = "await asyncio.sleep(1)"
>>> module = ast.parse(code)        # should be a syntax error on 3.6
>>> bytecode = compile(module, ...) # should also be a syntax error
>>> await exec(bytecode, ...)       # exec does not return a coroutine... 
(hint: don't do that in production), and wonder how (C)Python could be improved to support these use case.

We will use the above journey in async as a thread (pun intended) to describe what we've learned implementing this, how CPython could learn from this (wether they decide to implement such a feature), as well as the various places where core Python could be improved, and where the tensions are to suit the needs of classical applications developments as well as exploratory programming.

Audience
Intermediate Python programmers and Users interested in Future of Python, as well as CPython core dev. 

While this talks will touch some fairly advanced topics (manipulating AST, and executing Bytecode), I'll mostly focus on high level view a to make it accessible. 

It is also important to make intermediate users of what is possible with Async/Await and how the language could evolve as they are the vast majority of Python programers and highly influence how the language it taught and the directions the language could take. 

I also hope this will trigger some question and discussion with some of the CPython core dev (already got some online exchanges with Yuri Selivanov and Nathaniel Smith)
 
Outline
- Hello / Who am I/intro (2min)
- Quick Outline (recurse here :-) )

- Talk Thread : Python in the REPL vs Python in a File
   - describe there is tension between both, and that we'll refer to it many time.

- What is top level async ? 
   - Why is it not a CPython feature ? Does it always make sens ?(need and
     eventloop, make no sens at module level). 
   - Why can it work in a REPL ? 

   - Gif demo CPython vs IPython

Working around CPython limitation

   - module = ast.parse(code)
      - Tension between REPL and Module. 
        API assume the chuck of code you parse are for a module:
          - AST optimisation, "filename"...
      - need to do some AST and string munging to get it to parse. 
          - description of these transforms. 
          - some choices due to future issue with compile/exec.
      - Suggestion of Core Python modification

    - codeobj = compile(module, ...)
       - Also a syntax Error. 
          - Tension between REPL and Module (Inherited from ast.parse)
       - Workaround: 
          - Modify Bytecode (don't create new locals)
          - Slightly modify execution semantics (?)
       - Suggestion for a better API. 

    - exec(codeobj) 
       - Usually blocking though you would like it to not be.
       - tension between, REPL and Module. No "multiline statements"
       - Make it emit a coroutine object and run that in the loop.
       - Suggestion for API modifications.

    - You know have the BASE fo an async REPL

The rest of the problems.
    - Drawbacks of these transforms 
       - return, break, yield... now valid top level. 
          - more ast munging and checking. 
    - Screw up line numbers and tracebacks.
    - Multiline REPL.
       - Detect when the user is "done" typing requires knowing whether the code
         is "valid". `codeop` does not support top level async. 
    - External tools get confused.

Recap

How can you help ?
  - break things in IPython
  - See discussion https://bugs.python.org/issue34616

Thanks