deadlock error log

gistfile1.txt

# Date: 06/13/23
# PR #16209
# api tests

2023-06-13T19:14:00.1624839Z [SQL: UPDATE job SET update_time=%(update_time)s, state=%(state)s, job_runner_name=%(job_runner_name)s, handler=%(handler)s WHERE job.id = %(job_id)s]
2023-06-13T19:14:00.1626020Z [parameters: {'update_time': datetime.datetime(2023, 6, 13, 18, 17, 46, 763299), 'state': <JobState.QUEUED: 'queued'>, 'job_runner_name': 'celery', 'handler': 'fv-az1234-686', 'job_id': 4}]
2023-06-13T19:14:00.1626864Z (Background on this error at: https://sqlalche.me/e/14/e3q8) (Background on this error at: https://sqlalche.me/e/14/7s2a)
2023-06-13T19:14:00.1628035Z celery.worker.request DEBUG 2023-06-13 18:17:48,871 [pN:main,p:2329,tN:Thread-2] Task accepted: galaxy.celery.tasks.setup_fetch_data[a554b187-f219-412b-a9da-2d73aa5dda92] pid:2329
2023-06-13T19:14:00.1629163Z galaxy.celery WARNING 2023-06-13 18:17:48,872 [pN:main,p:2329,tN:Thread-2] Celery task execution failed for setup_fetch_data setup_fetch_data (0.599 ms)
2023-06-13T19:14:00.1633944Z celery.app.trace ERROR 2023-06-13 18:17:48,876 [pN:main,p:2329,tN:Thread-2] Task galaxy.celery.tasks.setup_fetch_data[a554b187-f219-412b-a9da-2d73aa5dda92] raised unexpected: PendingRollbackError('This Session\'s transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Sessi
2023-06-13T19:14:00.1636116Z Traceback (most recent call last):
2023-06-13T19:14:00.1637067Z   File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.7/site-packages/celery/app/trace.py", line 451, in trace_task
2023-06-13T19:14:00.1637612Z     R = retval = fun(*args, **kwargs)
2023-06-13T19:14:00.1638153Z   File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/celery/__init__.py", line 163, in wrapper
2023-06-13T19:14:00.1638634Z     rval = app.magic_partial(func)(*args, **kwds)
2023-06-13T19:14:00.1643566Z   File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.7/site-packages/lagom/wrapping.py", line 28, in _bound_func
2023-06-13T19:14:00.1644555Z     return inner_func(*bound_args, **bound_kwargs)
2023-06-13T19:14:00.1645426Z   File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.7/site-packages/lagom/wrapping.py", line 45, in _error_handling_func
2023-06-13T19:14:00.1645941Z     return func(*args, **kwargs)
2023-06-13T19:14:00.1646561Z   File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/celery/tasks.py", line 201, in setup_fetch_data
2023-06-13T19:14:00.1647332Z     job = sa_session.query(model.Job).get(job_id)
2023-06-13T19:14:00.1647724Z   File "<string>", line 2, in get
2023-06-13T19:14:00.1648503Z   File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.7/site-packages/sqlalchemy/util/deprecations.py", line 468, in warned
2023-06-13T19:14:00.1648992Z     return fn(*args, **kwargs)
2023-06-13T19:14:00.1649731Z   File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 947, in get
2023-06-13T19:14:00.1650289Z     return self._get_impl(ident, loading.load_on_pk_identity)
2023-06-13T19:14:00.1651080Z   File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 959, in _get_impl
2023-06-13T19:14:00.1651604Z     execution_options=self._execution_options,
2023-06-13T19:14:00.1652380Z   File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2979, in _get_impl
2023-06-13T19:14:00.1652873Z     load_options=load_options,
2023-06-13T19:14:00.1653679Z   File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.7/site-packages/sqlalchemy/orm/loading.py", line 534, in load_on_pk_identity
2023-06-13T19:14:00.1654184Z     bind_arguments=bind_arguments,
2023-06-13T19:14:00.1654927Z   File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1716, in execute
2023-06-13T19:14:00.1655441Z     conn = self._connection_for_bind(bind)
2023-06-13T19:14:00.1656247Z   File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1556, in _connection_for_bind
2023-06-13T19:14:00.1656727Z     engine, execution_options
2023-06-13T19:14:00.1657531Z   File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 724, in _connection_for_bind
2023-06-13T19:14:00.1658002Z     self._assert_active()
2023-06-13T19:14:00.1658771Z   File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 611, in _assert_active
2023-06-13T19:14:00.1659213Z     code="7s2a",
2023-06-13T19:14:00.1660595Z sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (psycopg2.errors.DeadlockDetected) deadlock detected
2023-06-13T19:14:00.1661535Z DETAIL:  Process 98 waits for ShareLock on transaction 1194; blocked by process 95.
2023-06-13T19:14:00.1662144Z Process 95 waits for ShareLock on transaction 1193; blocked by process 98.
2023-06-13T19:14:00.1662608Z HINT:  See server log for query details.
2023-06-13T19:14:00.1663077Z CONTEXT:  while updating tuple (0,19) in relation "job"