[TIP] Process large amount of bulk tasks

queuing-large-amount-of-tasks

# Queuing process vs one call :

## One call :
How to finish a server. Could result by an execution time exception, not optimised for large amount of tasks and very bad for user experience

## Queuing :
The tasks will be added to a queue and treated one by one with a cron job : this save user experience and performance

# For each task generate 1 .json/.txt file in a directory

- Loop on tasks
- For each task create a .json/.txt file with all the datas required for treatment
- Name the file with Id or timestamp or whatelse useful
- In one directory, stock all the files corresponding to the tasks you want to add in queue

# Then create a script to read and treat X .json/.txt files in the directory with one call

- Get the X files you want to treat at once
- For each file, open its content, treat it (send mail, update, etc...)
- Close the file
- If sending or update is successful, remove the file

# Then create a cronjob every minute to treat the queue

- Example : every minute, read and treat 10 files, then remove them from the queue
- That's it