# Minimal reproducer for dd-trace-py Issue #1414

README.md

Minimal reproducer for dd-trace-py Issue #1414

This is a minimal reproduction of DataDog/dd-trace-py#1414.

Running gunicorn with automatic profiling enabled causes timeouts when importing certain libraries. In this case I am importing the mgrs library.

To reproduce:

1. Run make run-docker
2. Run curl localhost:8000 from another window
3. Observe the request hangs

The request hangs regardless of whether a datadog agent is running or not.

If you run make run-gunicorn instead it works. Likewise if you modify the code to import mgrs before enabling profiling it also works. Disabling gevent will also fix the problem.

Stack trace if you CTRL-C while it's hanging:

[2020-12-03 21:59:19 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2020-12-03 21:59:19 +0000] [1] [DEBUG] Arbiter booted
[2020-12-03 21:59:19 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
[2020-12-03 21:59:19 +0000] [1] [INFO] Using worker: gevent
[2020-12-03 21:59:19 +0000] [7] [INFO] Booting worker with pid: 7
[2020-12-03 21:59:20 +0000] [1] [DEBUG] 1 workers
- DATADOG TRACER DIAGNOSTIC - Agent not reachable. Exception raised: [Errno 99] Cannot assign requested address
^C[2020-12-03 21:59:26 +0000] [1] [INFO] Handling signal: int
[2020-12-03 21:59:26 +0000] [7] [INFO] Worker exiting (pid: 7)
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/gevent/monkey.py", line 867, in _shutdown
    sleep()
  File "/usr/local/lib/python3.8/site-packages/gevent/hub.py", line 156, in sleep
    waiter.get()
  File "src/gevent/_waiter.py", line 140, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_waiter.py", line 151, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
  File "src/gevent/greenlet.py", line 854, in gevent._gevent_cgreenlet.Greenlet.run
  File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 196, in handle_quit
    sys.exit(0)
SystemExit: 0
2020-12-03T21:59:26Z <greenlet.greenlet object at 0x7f9b5a2eb1a0> failed with SystemExit

The part of the MGRS code that appears to cause the issue is this:

import ctypes
from ctypes.util import find_library
ctypes.CDLL(find_library('c')).free

I can reproduce this issue be replacing import mgrs with the above code.

Summary of results:

Docker container	Gevent	mgrs after profiling	Result
No	Any	Any	Works
Yes	Any	No	Works
Yes	Yes	Yes	Timeout
Yes	No	Yes	Works

Dockerfile

FROM python:3.8.6

WORKDIR /code

COPY requirements.txt requirements.txt
RUN python3 -m pip install -r requirements.txt 

COPY wsgi.py /code

CMD ["python3", "-m", "gunicorn.app.wsgiapp", "wsgi:app", \
    "--worker-class", "gevent", \
    "--bind", "0.0.0.0:8000", \
    "--log-level", "debug" \
]

Makefile

docker-build:
	docker build -t dd-profile-test .

run-gunicorn:
	python3 -m gunicorn.app.wsgiapp wsgi:app \
		--worker-class gevent \
		--bind 0.0.0.0:8000 \
		--log-level debug

run-docker: docker-build
	docker run -it --rm \
		--name dd-profile-test \
		-p 8000:8000 \
		dd-profile-test

requirements.txt

ddtrace==0.44.0
flask==1.1.2
gevent==20.9.0
gunicorn==20.0.4
mgrs==1.4.0

wsgi.py

from flask import Flask

# Importing mgrs before profiling works
# import mgrs
# import ddtrace.profiling.auto

# Importing mgrs after profiling request to hang
import ddtrace.profiling.auto
import mgrs

# Alternatively this also causes requests to hang
# import ddtrace.profiling.auto
# import ctypes
# from ctypes.util import find_library
# ctypes.CDLL(find_library('c')).free


app = Flask(__name__)


@app.route("/")
def hello():
    return "Hello"


if __name__ == "__main__":
    app.run()