Issue with using flock with Magento's cron

README.md

Overview

This applies to Magento 2.3+.

If you have your crontab configured to run the bin/magento cron:run command using flock, you should either:

• Remove flock, since it's technically not necessary as of Magento 2.3.2
• Pass the -o command to flock, as this will prevent the indefinitely running bin/magento queue:consumers:start processes spawned by bin/magento cron:run from holding a lock when they shouldn't be.

Before:

* * * * * flock -n /var/www/html/var/locks/cron.lock /usr/bin/php /var/www/html/bin/magento cron:run | grep -v "Ran jobs by schedule" >> /var/www/html/var/log/magento.cron.log

After:

* * * * * flock -n -o /var/www/html/var/locks/cron.lock /usr/bin/php /var/www/html/bin/magento cron:run | grep -v "Ran jobs by schedule" >> /var/www/html/var/log/magento.cron.log

Details

First, some context: When the bin/magento cron:run process runs, one of the things it does is create a number of "message queues":

Processes started by cron consume the specified number of messages and then terminate.

If you use flock to run bin/magento cron:run, then the child processes that are created by cron:run will hold a lock on the lock file. For example:

# Run cron in the background (to simulate crontab)
$ flock -n /var/www/html/var/locks/cron.lock /usr/bin/php /var/www/html/bin/magento cron:run &

# Find all processes locking the lock file
$ lsof /var/www/html/var/locks/cron.lock
COMMAND  PID     USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
php7.2  3623 www-data    3r   REG  259,1        0 6916891 /var/www/html/var/locks/cron-run.lock
php7.2  3630 www-data    3r   REG  259,1        0 6916891 /var/www/html/var/locks/cron-run.lock

# Lookup details on processes belonging to the PIDs above:
www-data  3623  7.5  1.2 774504 412228 ?       S    19:21   0:03 /usr/bin/php7.2 /var/www/html/bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1
www-data  3630  1.1  0.4 492452 129080 ?       S    19:21   0:00 /usr/bin/php7.2 /var/www/html/bin/magento queue:consumers:start product_action_attribute.update

The problem is that the bin/magento cron:run command will exit and release its lock, but the magento queue:consumers:start processes will run indefinitely until "consume the specified number of messages".

From man flock:

       -o, --close
              Close the file descriptor on which the lock is held before executing command.  This is useful if command spawns a child process which should not be holding the lock.

By passing the -o flag to flock, we ensure that the magento queue:consumers:start commands don't hold a lock on the lock file.