Usually, we first create a object of a class extending WP_Background_Processing class. Then we create a queue, save queue & dispatch queue.
Calling dispatch method does two things
1. Schedule Healthcheck cron to run every 5 minutes. @see `schedule_cron_healthcheck`
2. Call Ajax request that processes queue.
Ajax Request's callback is maybe_handle. @see WP_Background_Processing::maybe_handle();
This maybe_handle terminates itself if any batch of respective background class is already in process.
So essentially, only one batch is allowed to execute at any point.
The cron callback is handle_cron_healthcheck. This will also behave very similar to maybe_handle.
At the bottom of handle_cron_healthcheck and maybe_handle, there is a call to handle method if no batch is in process.
Here is how this handle method works (Real Magic)
1. Set 1 minute transient in db to indicate that batch processing is in progress. (aka locking - see `lock_process`)
2. This `handle` method expects to complete processing in 20 seconds (see `time_exceeded` method)
2. It creates variable to hold `start time`.
3. Get 1 batch from db.
4. Create a loop that will run `task` method on each item present in the batch.
5. After running `task` method on an item, it checks if 20 seconds are passed from `start time` or if memory consumption is exceeded. If any of these criteria is violated, then
i) it will update current batch in the db by removing items which are already processed.
ii) it will remove transient set in step 1 (aka unlocking - see `unlock_process`).
iii) calls `dispatch` method again.
6. If time and memory does not exceed till the batch is processed completely, then it will remove current batch from db, pull next batch and repeat from step 4.
Wondering why Cron is necessary in this whole scenario if handle method is calling ajax at end (read dispatch method)? - Simple, it is used as a mechanism to initiate queue processing automatically if something fails.