Skip to content

Instantly share code, notes, and snippets.

@zzzeek
Created June 11, 2019 20:00
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save zzzeek/91fd34c4a9245cfbd703b685c1f1d5c0 to your computer and use it in GitHub Desktop.
Save zzzeek/91fd34c4a9245cfbd703b685c1f1d5c0 to your computer and use it in GitHub Desktop.
PyMySQL + eventlet authenticaiton timeout
import random
import time as blocking_time
import eventlet
import pymysql
# monkeypatch Python network libraries so that they are non blocking. all
# Python code that uses the pure Python network IO functions will be non
# blocking, where all IO waiting will defer back to eventlet where it will
# pass off work to another greenlet.
eventlet.monkey_patch(socket=True)
def connect():
"""Connect to a local MySQL database.
No hostname is given, so this uses a UNIX socket. Any kind of socket
will do.
"""
return pymysql.connect()
def do_work(greenlet, cpu_blocking):
while True:
print(
"greenlet %s CPU blocking for %.4f sec" % (greenlet, cpu_blocking)
)
blocking_time.sleep(cpu_blocking)
conn = connect()
print("greenlet %s connected to the database" % greenlet)
def retreive_connection_timeout():
"""Get the configured connect_timeout from the server.
This is usually ten seconds. If it's much higher, then this script
will take a lot longer to illustrate the effect.
"""
conn = connect()
cursor = conn.cursor()
cursor.execute("show variables like 'connect_timeout'")
connect_timeout = int(cursor.fetchone()[1])
cursor.close()
conn.close()
return connect_timeout
connect_timeout = retreive_connection_timeout()
pool = eventlet.GreenPool()
print(
"Creating ten continuous greenlets with a tiny amount of CPU work in "
"between database connections"
)
for i in range(10):
pool.spawn_n(do_work, i, random.choice([0.001, 0.002, 0.005]))
eventlet.sleep(1)
print(
"Now adding a CPU expensive worker that will wait "
"longer than %d seconds to finish its CPU work" % connect_timeout
)
pool.spawn_n(do_work, i, connect_timeout + 2)
pool.waitall()
@zzzeek
Copy link
Author

zzzeek commented Jun 11, 2019

The script creates ten greenlets, then an 11th one that is going to hog the CPU longer than MySQL's connect_timeout. The connection attempts begin to fail as MySQL is rejecting the connections that have waited longer than ten seconds to respond to a security challenge. The stack trace generated by PyMySQL is very distinctive and is a 100% match for this exact situation. You can see the "2006 MySQL server has gone away", the fact that it occurred right in pymysql _process_auth(), and the greenio monkeypatching of PyMySQL which prevents it from working correctly.

$ python test3.py 
Creating ten continuous greenlets with a tiny amount of CPU work in between database connections
greenlet 0 CPU blocking for 0.0010 sec
greenlet 1 CPU blocking for 0.0050 sec
greenlet 2 CPU blocking for 0.0010 sec
greenlet 3 CPU blocking for 0.0050 sec
greenlet 4 CPU blocking for 0.0020 sec
greenlet 5 CPU blocking for 0.0020 sec
greenlet 6 CPU blocking for 0.0020 sec
greenlet 7 CPU blocking for 0.0020 sec
greenlet 8 CPU blocking for 0.0050 sec
greenlet 9 CPU blocking for 0.0050 sec
greenlet 5 connected to the database
greenlet 5 CPU blocking for 0.0020 sec
greenlet 0 connected to the database
greenlet 0 CPU blocking for 0.0010 sec
greenlet 8 connected to the database
greenlet 8 CPU blocking for 0.0050 sec
greenlet 9 connected to the database
greenlet 9 CPU blocking for 0.0050 sec
greenlet 7 connected to the database
greenlet 7 CPU blocking for 0.0020 sec
greenlet 1 connected to the database

.
.  a few hundred more of these
.

greenlet 6 connected to the database
greenlet 6 CPU blocking for 0.0020 sec
greenlet 2 connected to the database
greenlet 2 CPU blocking for 0.0010 sec
greenlet 7 connected to the database
greenlet 7 CPU blocking for 0.0020 sec
greenlet 4 connected to the database
greenlet 4 CPU blocking for 0.0020 sec
greenlet 0 connected to the database
greenlet 0 CPU blocking for 0.0010 sec
greenlet 1 connected to the database
greenlet 1 CPU blocking for 0.0050 sec
Now adding a CPU expensive worker that will wait longer than 10 seconds to finish its CPU work
greenlet 9 CPU blocking for 12.0000 sec
Traceback (most recent call last):
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 708, in _write_bytes
    self._sock.sendall(data)
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenio/base.py", line 403, in sendall
    tail = self.send(data, flags)
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenio/base.py", line 397, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenio/base.py", line 384, in _send_loop
    return send_method(data, *args)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl
    func(*args, **kwargs)
  File "test3.py", line 31, in do_work
    conn = connect()
  File "test3.py", line 22, in connect
    return pymysql.connect()
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/__init__.py", line 94, in Connect
    return Connection(*args, **kwargs)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 327, in __init__
    self.connect()
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 598, in connect
    self._request_authentication()
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 862, in _request_authentication
    auth_packet = self._process_auth(plugin_name, auth_packet)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 932, in _process_auth
    self.write_packet(data)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 644, in write_packet
    self._write_bytes(data)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 713, in _write_bytes
    "MySQL server has gone away (%r)" % (e,))
pymysql.err.OperationalError: (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")
greenlet 5 connected to the database
greenlet 5 CPU blocking for 0.0020 sec
greenlet 9 connected to the database
greenlet 9 CPU blocking for 0.0050 sec
greenlet 8 connected to the database
greenlet 8 CPU blocking for 0.0050 sec
greenlet 3 connected to the database
greenlet 3 CPU blocking for 0.0050 sec
Traceback (most recent call last):
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 708, in _write_bytes
    self._sock.sendall(data)
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenio/base.py", line 403, in sendall
    tail = self.send(data, flags)
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenio/base.py", line 397, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenio/base.py", line 384, in _send_loop
    return send_method(data, *args)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl
    func(*args, **kwargs)
  File "test3.py", line 31, in do_work
    conn = connect()
  File "test3.py", line 22, in connect
    return pymysql.connect()
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/__init__.py", line 94, in Connect
    return Connection(*args, **kwargs)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 327, in __init__
    self.connect()
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 598, in connect
    self._request_authentication()
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 862, in _request_authentication
    auth_packet = self._process_auth(plugin_name, auth_packet)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 932, in _process_auth
    self.write_packet(data)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 644, in write_packet
    self._write_bytes(data)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 713, in _write_bytes
    "MySQL server has gone away (%r)" % (e,))
pymysql.err.OperationalError: (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")
Traceback (most recent call last):
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 708, in _write_bytes
    self._sock.sendall(data)
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenio/base.py", line 403, in sendall
    tail = self.send(data, flags)
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenio/base.py", line 397, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenio/base.py", line 384, in _send_loop
    return send_method(data, *args)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl
    func(*args, **kwargs)
  File "test3.py", line 31, in do_work
    conn = connect()
  File "test3.py", line 22, in connect
    return pymysql.connect()
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/__init__.py", line 94, in Connect
    return Connection(*args, **kwargs)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 327, in __init__
    self.connect()
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 598, in connect
    self._request_authentication()
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 862, in _request_authentication
    auth_packet = self._process_auth(plugin_name, auth_packet)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 932, in _process_auth
    self.write_packet(data)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 644, in write_packet
    self._write_bytes(data)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 713, in _write_bytes
    "MySQL server has gone away (%r)" % (e,))
pymysql.err.OperationalError: (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")
greenlet 2 connected to the database
greenlet 2 CPU blocking for 0.0010 sec
greenlet 8 connected to the database
greenlet 8 CPU blocking for 0.0050 sec
greenlet 6 connected to the database
greenlet 6 CPU blocking for 0.0020 sec
greenlet 7 connected to the database
greenlet 7 CPU blocking for 0.0020 sec
greenlet 8 connected to the database
greenlet 8 CPU blocking for 0.0050 sec
greenlet 7 connected to the database
greenlet 7 CPU blocking for 0.0020 sec
greenlet 5 connected to the database
greenlet 5 CPU blocking for 0.0020 sec
greenlet 9 connected to the database
greenlet 9 CPU blocking for 12.0000 sec
Traceback (most recent call last):
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 708, in _write_bytes
    self._sock.sendall(data)
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenio/base.py", line 403, in sendall
    tail = self.send(data, flags)
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenio/base.py", line 397, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenio/base.py", line 384, in _send_loop
    return send_method(data, *args)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl
    func(*args, **kwargs)
  File "test3.py", line 31, in do_work
    conn = connect()
  File "test3.py", line 22, in connect
    return pymysql.connect()
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/__init__.py", line 94, in Connect
    return Connection(*args, **kwargs)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 327, in __init__
    self.connect()
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 598, in connect
    self._request_authentication()
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 862, in _request_authentication
    auth_packet = self._process_auth(plugin_name, auth_packet)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 932, in _process_auth
    self.write_packet(data)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 644, in write_packet
    self._write_bytes(data)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 713, in _write_bytes
    "MySQL server has gone away (%r)" % (e,))
pymysql.err.OperationalError: (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")
greenlet 3 connected to the database
greenlet 3 CPU blocking for 0.0050 sec
Traceback (most recent call last):
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 708, in _write_bytes
    self._sock.sendall(data)
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenio/base.py", line 403, in sendall
    tail = self.send(data, flags)
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenio/base.py", line 397, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenio/base.py", line 384, in _send_loop
    return send_method(data, *args)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl
    func(*args, **kwargs)
  File "test3.py", line 31, in do_work
    conn = connect()
  File "test3.py", line 22, in connect
    return pymysql.connect()
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/__init__.py", line 94, in Connect
    return Connection(*args, **kwargs)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 327, in __init__
    self.connect()
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 598, in connect
    self._request_authentication()
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 862, in _request_authentication
    auth_packet = self._process_auth(plugin_name, auth_packet)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 932, in _process_auth
    self.write_packet(data)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 644, in write_packet
    self._write_bytes(data)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 713, in _write_bytes
    "MySQL server has gone away (%r)" % (e,))
pymysql.err.OperationalError: (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")
greenlet 9 connected to the database
greenlet 9 CPU blocking for 0.0050 sec
Traceback (most recent call last):
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 708, in _write_bytes
    self._sock.sendall(data)
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenio/base.py", line 403, in sendall
    tail = self.send(data, flags)
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenio/base.py", line 397, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenio/base.py", line 384, in _send_loop
    return send_method(data, *args)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/classic/.venv3/lib/python3.7/site-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl
    func(*args, **kwargs)
  File "test3.py", line 31, in do_work
    conn = connect()
  File "test3.py", line 22, in connect
    return pymysql.connect()
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/__init__.py", line 94, in Connect
    return Connection(*args, **kwargs)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 327, in __init__
    self.connect()
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 598, in connect
    self._request_authentication()
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 862, in _request_authentication
    auth_packet = self._process_auth(plugin_name, auth_packet)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 932, in _process_auth
    self.write_packet(data)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 644, in write_packet
    self._write_bytes(data)
  File "/home/classic/.venv3/lib/python3.7/site-packages/pymysql/connections.py", line 713, in _write_bytes
    "MySQL server has gone away (%r)" % (e,))
pymysql.err.OperationalError: (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment