me:
Is it possible to send a signal to a worker/process?
I realize the platform sends SIGTERM
and SIGKILL
when restarting a dyno, but I need to send a USR1
to one of my workers to tell it to stop picking up new jobs. Normally this is achieved via kill -USR1 <pid>
, but on the Heroku platform not only do we not know know the pid, we also don't run one-off commands on the same dyno.
Caio (heroku support):
We had this feature experimentally at some point but it was never productized. I recommend you find other ways to signal your processes, like setting a database flag.
me:
Thanks for the quick reply.
It is unfortunate that feature has not been further investigated and productized. The suggested workaround is far from ideal, and not very realistic.
The whole idea of a USR1
signal is to signal the current process to take some action - in this case, stop taking action. But a new process, which will be spun up after the app is restarted, will have no knowledge of that signal and will get on with doing it's thing - working jobs. Using a database flag introduces a whole host of complexity around managing that state across app restarts and needing to customize (read: hack or monkey patch) existing tools to be aware of that flag.
Caio (heroku support):
That's very reasonable feedback. I'm routing this to our platform engineers.
Hi Steven- in general terms, there's a number of different ways to accomplish this goal:
I understand it would be convenient in your use case to send a signal to your library of choice to cause #2 to occur. Obviously that's not currently possible on Heroku. However, since workers must handle #3 every 24 hours during dyno cycling, we don't think this is a particularly viable solution.
In cases of large software changes or migrations, where data in the queue is tied to the software implementation handling that data, #1 may actually be the most viable. For some applications, you would enact this by scaling your web workers to zero and put your site into maintenance mode.
More flexibility by queuing libraries could also help. For instance, why is a signal the only way to enact this change in Sidekiq's processing? Perhaps there should be another way to enact these sorts of maintenance modes.