airflow MySQL

gistfile1.txt

[2016-08-08 23:13:55,315] {models.py:1286} ERROR - (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: u'DELETE FROM xcom WHERE xcom.`key` = %s AND xcom.execution_date = %s AND xcom.task_id = %s AND xcom.dag_id = %s'] [parameters: (u'return_value', datetime.datetime(2016, 8, 6, 13, 0), 'process_installs', 'mobile_hourly')]
Traceback (most recent call last):
  File "/tmp/venv/lib/python2.7/site-packages/airflow/models.py", line 1249, in run
    self.xcom_push(key=XCOM_RETURN_KEY, value=result)
  File "/tmp/venv/lib/python2.7/site-packages/airflow/models.py", line 1463, in xcom_push
    execution_date=execution_date or self.execution_date)
  File "/tmp/venv/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in wrapper
    result = func(*args, **kwargs)
  File "/tmp/venv/lib/python2.7/site-packages/airflow/models.py", line 3197, in set
    cls.dag_id == dag_id).delete()
  File "/tmp/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3073, in delete
    delete_op.exec_()
  File "/tmp/venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 1127, in exec_
    self._do_exec()
  File "/tmp/venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 1311, in _do_exec
    mapper=self.mapper)
  File "/tmp/venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1034, in execute
    bind, close_with_result=True).execute(clause, params or {})
  File "/tmp/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/tmp/venv/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/tmp/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/tmp/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/tmp/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/tmp/venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/tmp/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/tmp/venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
  File "/tmp/venv/lib/python2.7/site-packages/MySQLdb/cursors.py", line 226, in execute
    self.errorhandler(self, exc, value)
  File "/tmp/venv/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorvalue
OperationalError: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: u'DELETE FROM xcom WHERE xcom.`key` = %s AND xcom.execution_date = %s AND xcom.task_id = %s AND xcom.dag_id = %s'] [parameters: (u'return_value', datetime.datetime(2016, 8, 6, 13, 0), 'process_installs', 'mobile_hourly')]
[2016-08-08 23:13:55,319] {models.py:1306} INFO - Marking task as FAILED.

Does this still happen?

In version 1.8.2, I experimented with adding session.close() or session.rollback() just before or after TaskInstance.execute() calls execute() on the underlying DAG task. If I artificially hack up Airflow to leave the SQLAlchemy session active, I confirm that rollback() or close() causes an extra database connection to be created, which is what we'd want. But I can't figure out how this would happen in normal usage. It looks like clear_xcom_data(), the last apparent database interaction before running the task, properly commits its transaction, which should also release the connection.