debugging a flask/elastic beanstalk app with gdb

flask-segfault-gdb-debug.md

debugging a flask app on elastic beanstalk with gdb

this gist contains several iterations of attempting to attach gdb to a flask/wsgi app running on elastic beanstalk. assumption: ssh access (requires setting a key in the config, and then get the IP address from EB console or EC2 instance list), and sudo.

tl;dr: simple working solution is to

• write out the pid in /tmp/xpid

• sleep in a while loop and wait until that file disappears to continue

• meanwhile, visit the site, which will trigger the pid write: this is the pid of the forked handler which is handling this specific request. run sudo gdb

  • (gdb) attach <pid>
  • optionally set a useful breakpoint
  • continue

• now remove /tmp/xpid

• the app will continue, and any crashes will happen with gdb attached

the complication is that httpd forks a child per request (by default). answers on stackoverflow suggest changing httpd config so that there is only 1 PID target, but I wanted to leave the config alone.

other things attempted (in comments):

• spawn gdbserver target the running pid. this didn't work, I think gdbserver sits and waits for a connection, then exits when the process itself does. I did not see a way to set a breakpoint w/out connecting from the gdb client side.
• conditionless while loop, with a signal handler for SIGHUP to signal exit. I think this initially didn't work because there is no yield point for the signal to be checked; then I tried signal.pause, but that didn't work either -- the process just hung despite signalling (inside gdb w/ signal SIGHUP)

gistfile1.txt

# from Flask/Elastic Beanstalk tutorial https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create-deploy-python-flask.html

def getinfo():
    text = ''
    pid = os.getpid()
    text += f"<br>{pid}<br>"
    #pmap = subprocess.check_output(['pmap', str(pid)]).decode('UTF-8')
    #text += f"<details><summary>pmap output</summary><pre>{pmap}</pre></details>"

    #pmap = subprocess.check_output(['pmap', str(pid)]).decode('UTF-8')
    #text += f"<details><summary>pmap output</summary><pre>{pmap}</pre></details>"

    return text

#cc = False
#def setOnSig(n, frame):
#    global cc
#    cc = True

#signal.signal(signal.SIGHUP, setOnSig)

def index():
    v_txt = ''

    import os, time #, subprocess as sp
    pargs = ['--attach', ':45657', str(os.getpid())]
    v_txt += str(pargs)
    v_txt += '<br>'

    try:
      #sp.Popen(pargs,
      #       stdin=None, stdout=None, stderr=None, close_fds=True, shell=False)
      #os.spawnl(os.P_NOWAIT, '/usr/bin/gdbserver', pargs)
      open("/tmp/xpid", 'w').write(str(os.getpid()))
      while os.path.isfile("/tmp/xpid"):
        time.sleep(1)
    except Exception as exc:
      v_txt += str(exc)
      v_txt += '<br>'

    try:
      print("hello")
      import tiledb
      v_txt += 1 #"<br><br>ver: {}<br><br>".format(tiledb.version.version)
    except Exception as exc:
      v_txt += str(exc)

    import sys
    v_txt += str(sys.path)
    v_txt += "<br><br>"
    #import pkgutil
    #p = pkgutil.get_loader("tiledb")
    import imp
    v_txt += str(imp.find_module('tiledb'))
    info = getinfo()
    return header_text + say_hello() + v_txt + info + instructions + footer_text