deadlock error log

gistfile1.txt

# Date: 06/13/23
# PR #16209
# api tests

2023-06-13T19:36:43.8337105Z galaxy.workflow.scheduling_manager DEBUG 2023-06-13 18:41:56,943 [pN:main,p:2288,tN:WorkflowRequestMonitor.monitor_thread] Attempting to schedule workflow invocation [42]
2023-06-13T19:36:43.8338459Z galaxy.workflow.run DEBUG 2023-06-13 18:41:56,981 [pN:main,p:2288,tN:WorkflowRequestMonitor.monitor_thread] Marking step 459 outputs of invocation 42 delayed (workflow paused at this step waiting for review)
2023-06-13T19:36:43.8339675Z galaxy.workflow.run DEBUG 2023-06-13 18:41:56,983 [pN:main,p:2288,tN:WorkflowRequestMonitor.monitor_thread] Marking step 460 outputs of invocation 42 delayed (dependent step [459] delayed, so this step must be delayed)
2023-06-13T19:36:43.8340702Z galaxy.workflow.scheduling_manager DEBUG 2023-06-13 18:41:56,988 [pN:main,p:2288,tN:WorkflowRequestMonitor.monitor_thread] Workflow invocation [42] scheduled
2023-06-13T19:36:43.8341815Z galaxy.workflow.scheduling_manager DEBUG 2023-06-13 18:41:57,196 [pN:main,p:2288,tN:WorkflowRequestMonitor.monitor_thread] Attempting to schedule workflow invocation [42]
2023-06-13T19:36:43.8342514Z INFO:     127.0.0.1:60104 - "GET /api/histories/53a596df707bd31d HTTP/1.1" 200 OK
2023-06-13T19:36:43.8343531Z galaxy.workflow.run DEBUG 2023-06-13 18:41:57,265 [pN:main,p:2288,tN:WorkflowRequestMonitor.monitor_thread] Marking step 459 outputs of invocation 42 delayed (workflow paused at this step waiting for review)
2023-06-13T19:36:43.8344701Z galaxy.workflow.run DEBUG 2023-06-13 18:41:57,267 [pN:main,p:2288,tN:WorkflowRequestMonitor.monitor_thread] Marking step 460 outputs of invocation 42 delayed (dependent step [459] delayed, so this step must be delayed)
2023-06-13T19:36:43.8345733Z galaxy.workflow.scheduling_manager DEBUG 2023-06-13 18:41:57,272 [pN:main,p:2288,tN:WorkflowRequestMonitor.monitor_thread] Workflow invocation [42] scheduled
2023-06-13T19:36:43.8346589Z galaxy.jobs.handler DEBUG 2023-06-13 18:41:57,347 [pN:main,p:2288,tN:JobHandlerQueue.monitor_thread] Grabbed Job(s): 494 
2023-06-13T19:36:43.8347382Z galaxy.jobs.handler ERROR 2023-06-13 18:41:57,354 [pN:main,p:2288,tN:JobHandlerQueue.monitor_thread] Exception in monitor_step
2023-06-13T19:36:43.8347821Z Traceback (most recent call last):
2023-06-13T19:36:43.8348294Z   File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/jobs/handler.py", line 370, in __monitor
2023-06-13T19:36:43.8353106Z     self.__monitor_step()
2023-06-13T19:36:43.8353649Z   File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/jobs/handler.py", line 388, in __monitor_step
2023-06-13T19:36:43.8354348Z     self.__handle_waiting_jobs()
2023-06-13T19:36:43.8354836Z   File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/jobs/handler.py", line 465, in __handle_waiting_jobs
2023-06-13T19:36:43.8355326Z     .filter(ranked.c.rank <= self.app.job_config.handler_ready_window_size)
2023-06-13T19:36:43.8356099Z   File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2773, in all 
2023-06-13T19:36:43.8356490Z     return self._iter().all()
2023-06-13T19:36:43.8357116Z   File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2919, in _iter
2023-06-13T19:36:43.8357592Z     execution_options={"_sa_orm_load_options": self.load_options},
2023-06-13T19:36:43.8358263Z   File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1716, in execute
2023-06-13T19:36:43.8358693Z     conn = self._connection_for_bind(bind)
2023-06-13T19:36:43.8359382Z   File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1556, in _connection_for_bind
2023-06-13T19:36:43.8359779Z     engine, execution_options
2023-06-13T19:36:43.8360443Z   File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 724, in _connection_for_bind
2023-06-13T19:36:43.8360980Z     self._assert_active()
2023-06-13T19:36:43.8361622Z   File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 611, in _assert_active
2023-06-13T19:36:43.8361994Z     code="7s2a",
2023-06-13T19:36:43.8363193Z sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (psycopg2.errors.DeadlockDetected) deadlock detected
2023-06-13T19:36:43.8364033Z DETAIL:  Process 318 waits for ShareLock on transaction 8181; blocked by process 321.
2023-06-13T19:36:43.8364539Z Process 321 waits for ShareLock on transaction 8180; blocked by process 318.
2023-06-13T19:36:43.8364942Z HINT:  See server log for query details.
2023-06-13T19:36:43.8365339Z CONTEXT:  while updating tuple (30,69) in relation "job"
2023-06-13T19:36:43.8365540Z 
2023-06-13T19:36:43.8366152Z [SQL: UPDATE job SET update_time=%(update_time)s, state=%(state)s, job_runner_name=%(job_runner_name)s, destination_id=%(destination_id)s, destination_params=%(destination_params)s WHERE job.id = %(job_id)s]
2023-06-13T19:36:43.8367470Z [parameters: {'update_time': datetime.datetime(2023, 6, 13, 18, 41, 51, 116490), 'state': <JobState.QUEUED: 'queued'>, 'job_runner_name': 'local', 'destination_id': 'local', 'destination_params': <psycopg2.extensions.Binary object at 0x7f92e7782870>, 'job_id': 492}]
2023-06-13T19:36:43.8368303Z (Background on this error at: https://sqlalche.me/e/14/e3q8) (Background on this error at: https://sqlalche.me/e/14/7s2a)
2023-06-13T19:36:43.8369288Z galaxy.workflow.scheduling_manager DEBUG 2023-06-13 18:41:57,478 [pN:main,p:2288,tN:WorkflowRequestMonitor.monitor_thread] Attempting to schedule workflow invocation [42]
2023-06-13T19:36:43.8370428Z galaxy.workflow.run DEBUG 2023-06-13 18:41:57,538 [pN:main,p:2288,tN:WorkflowRequestMonitor.monitor_thread] Marking step 459 outputs of invocation 42 delayed (workflow paused at this step waiting for review)
2023-06-13T19:36:43.8371639Z galaxy.workflow.run DEBUG 2023-06-13 18:41:57,542 [pN:main,p:2288,tN:WorkflowRequestMonitor.monitor_thread] Marking step 460 outputs of invocation 42 delayed (dependent step [459] delayed, so this step must be delayed)
2023-06-13T19:36:43.8372647Z galaxy.workflow.scheduling_manager DEBUG 2023-06-13 18:41:57,549 [pN:main,p:2288,tN:WorkflowRequestMonitor.monitor_thread] Workflow invocation [42] scheduled
2023-06-13T19:36:43.8373321Z INFO:     127.0.0.1:60106 - "GET /api/histories/53a596df707bd31d HTTP/1.1" 200 OK