tcl_async_demo.py

from tkinter import *
import asyncio
from functools import wraps
import websockets

def runloop(func):
    '''
    This decorator converts a coroutine into a function which, when called,
    runs the underlying coroutine to completion in the asyncio event loop.
    '''
    func = asyncio.coroutine(func)
    @wraps(func)
    def wrapper(*args, **kwargs):
        return asyncio.get_event_loop().run_until_complete(func(*args, **kwargs))
    return wrapper

@asyncio.coroutine
def run_tk(root, interval=0.05):
    '''
    Run a tkinter app in an asyncio event loop.
    '''
    try:
        while True:
            root.update()
            yield from asyncio.sleep(interval)
    except TclError as e:
        if "application has been destroyed" not in e.args[0]:
            raise

@asyncio.coroutine
def listen_websocket(url):
    '''
    Connect to a websocket url, then print messages received on the connection
    until closed by the server.
    '''
    ws = yield from websockets.connect(url)
    while True:
        msg = yield from ws.recv()
        if msg is None:
            break
        print(msg)

@runloop
def main():
    root = Tk()
    entry = Entry(root)
    entry.grid()
    
    def spawn_ws_listener():
        return asyncio.async(listen_websocket(entry.get()))

    Button(root, text='Print', command=spawn_ws_listener).grid()
    
    yield from run_tk(root)

if __name__ == "__main__":
    main()

Hmm. Perhaps Python3.7 is doing something differently. I don't have 3.6 installed on any machines at this point to test for a difference. Like I said, I ran nameoftherose's script because I didn't want to spend too much time adapting the code on this page. I've also run other examples of this technique using dummy tasks and my own implementation in a websocket server/client scenario and see the same rise in memory use as it runs.

I found that I do have python3.6.1 installed on another Mac. So, I ran the same nameoftherose code there. I find the same increasing memory use. Interestingly, 3.6.1 starts off with double the memory usage of 3.7.3! But that's neither here nor there. Mind you, I'm not clicking on the button. I'm just starting the app and then ending by closing the tk window. So, the socket stuff doesn't get run.

Hmm, that's very intriguing. I decided to run nameoftherose code, exactly as written, on my system to see what happend.


It's possible that the memory consumption baseline of 3.6 vs 3.7 is down to some of the under-the-hood optimizations made to asyncio specifically and the data model generally.

What's weird is... my graphs don't show that linearly increasing behavior at all. I start right around 60 and never really go above it. I can't explain the weird sudden step-changes at all. Maybe the scheduler realizes that the event loop isn't really doing all that much and pages unused stuff out of memory? Would that even be opaque to mprof? Maybe some reference to some bootstrapping object finally gets gc'd, maybe also because of the loop doing basically nothing? Is the consumption you experience caused by MacOS somehow? This really raises more questions than answers...

I'd be inclined to maybe write to the python mailing list, if for nothing else than to satisfy curiosity.

Oooh OOOOH, what happens if you profile the same tkinter code, but without the asyncio stuff? What happens if it just spins on mainloop?

Except for the weird discontinuities in the previous graphs, I'd be hard pressed to distinguish the sync from the async runs.

So, I tried this on Windows 10 with Python 3.7.3 AND Linux Mint 19.1 with Python 3.7.3 and neither of them appear to have memory issues. It looks like this is isolated to MacOS. And High Sierra (10.13.6) is the only version that I've tried this with. I haven't tried removing the async portion yet. Can you post your code showing how you set that up?

The only changes necessary are at the very end of the listing.

Original

async def main():
    root = Tk()
    entry = Entry(root)
    entry.grid()
    
    def spawn_ws_listener():
       addr=entry.get().split(':')
       print('spawn',addr)
       return asyncio.ensure_future( tclient( addr[0],int(addr[1]) ) )

    Button(root, text='Connect', command=spawn_ws_listener).grid()
    
    await run_tk(root)

if __name__ == "__main__":
    asyncio.get_event_loop().run_until_complete(main())

Synchronous Equivalent

def main():
    root = Tk()
    entry = Entry(root)
    entry.grid()
    
    # NOTE: Just don't click the button, as it won't work.
    #       Makes no difference for testing, because we weren't clicking the button anyway.
    def spawn_ws_listener():
       addr=entry.get().split(':')
       print('spawn',addr)
       return asyncio.ensure_future( tclient( addr[0],int(addr[1]) ) )

    Button(root, text='Connect', command=spawn_ws_listener).grid()
    
    root.mainloop()

if __name__ == "__main__":
    main()

What happens if you run it on MacOS for longer? Try profiling for ~1h and see if it ever reaches equilibrium. Maybe it just takes longer to get to a steady state? The extra memory usage compared to Windows is pretty egregious in any case, but I'd be curious to see how bad the problem could get for a long-running app...

Hmm. With those modifications you never call run_tk() and therefore don't ever call root.update(). Instead what you're showing is just a standard tk app with root.mainloop() handling the refreshing. How about this code?

from tkinter import *
import asyncio
import time

def run_tk(root, interval=0.05):
    try:
        while True:
            root.update()
            time.sleep(interval)
    except TclError as e:
        if "application has been destroyed" not in e.args[0]:
            raise

async def tclient(addr,port):
    print('tclient',addr,port)
    try:
        sock ,_= await asyncio.open_connection(host=addr,port=port)
       #print(sock)
       #f=sock.as_stream()
        while True:
               #data = yield from f.readline()
                data = await sock.readline()
                if not data:break
                data=data.decode()
                print(data,end='\n' if data[-1]=='\r' else'')
    except:
        pass


def main():
    root = Tk()
    entry = Entry(root)
    entry.grid()

    def spawn_ws_listener():
       addr=entry.get().split(':')
       print('spawn',addr)
       return asyncio.ensure_future( tclient( addr[0],int(addr[1]) ) )

    Button(root, text='Connect', command=spawn_ws_listener).grid()

    run_tk(root)

if __name__ == "__main__":
    main()

I ran it for about 2 hrs on Macos with Python 3.7.3 and here's the Mprof Plot result. Some interesting memory releases but overall it still seems to be a constantly increasing bag of goodies.

Right. I just wanted to see how tkinter runs "as intended" in the MacOS environment.

Yep, that's a goofy-looking unfortunate graph, haha.

I think I've figured it out. This page at tkdocs tells of bad things that happen with the elderly version of Tcl/Tk included with MacOs.
https://tkdocs.com/tutorial/install.html
They say to not use this version (typically 8.5) but instead to install the latest 8.6 (8.6.9 now). But, if you install Python with homebrew, there is currently no easy way to point it to a newly installed version of Tk. At this point in my research, the only way to get the updated Tcl/Tk to work with Python3 is to install Python without homebrew. Anyway, I've run the tk_sync_demo.py code (that I posted previously) on a Mac with Python 3.7.3 AND Tk version 8.6. Memory runs flatline. So, it looks like my intention of simplifying cross platform support by using a built-in library for the GUI (instead of PyQt5) is not as flawless as I had hoped.

Aww man, that's a bummer. Glad you found the culprit, though!