background-jobs-migration-guide.md

Migration guide to new Background Jobs backend

As of MISP version 2.4.151 we introduced a simpler way to handle background jobs without relying in CakeResque as this library is no longer mantained.

For the time being both background jobs backends will be supported, but we plan to phase out the CakeResque one in a near future.

Tip

For RHEL (8) users follow this guide: https://gist.github.com/Wachizungu/1497109c9e73d173953a0efe82810dd4

The new backend requires Supervisor and some extra PHP packages.

This guide is intended for Ubuntu/Debian systems

Install requirements

Run on your MISP instance the following commands.

1. Install Supervisord:

   sudo apt install supervisor -y
   

2. Install required PHP packages:

   cd /var/www/MISP/app
   sudo -u www-data composer require --with-all-dependencies supervisorphp/supervisor \
       guzzlehttp/guzzle \
       php-http/message  \
       lstrojny/fxmlrpc
   
   

3. Add the following settings at the bottom of the Supervisord conf file, usually located in:

   /etc/supervisor/supervisord.conf

   [inet_http_server]
   port=127.0.0.1:9001
   username=supervisor
   password=PWD_CHANGE_ME
   

4. Use the following configuration as a template for the services, usually located in:

   /etc/supervisor/conf.d/misp-workers.conf

   [group:misp-workers]
   programs=default,email,cache,prio,update
   
   [program:default]
   directory=/var/www/MISP
   command=/var/www/MISP/app/Console/cake start_worker default
   process_name=%(program_name)s_%(process_num)02d
   numprocs=5
   autostart=true
   autorestart=true
   redirect_stderr=false
   stderr_logfile=/var/www/MISP/app/tmp/logs/misp-workers-errors.log
   stdout_logfile=/var/www/MISP/app/tmp/logs/misp-workers.log
   directory=/var/www/MISP
   user=www-data
   
   [program:prio]
   directory=/var/www/MISP
   command=/var/www/MISP/app/Console/cake start_worker prio
   process_name=%(program_name)s_%(process_num)02d
   numprocs=5
   autostart=true
   autorestart=true
   redirect_stderr=false
   stderr_logfile=/var/www/MISP/app/tmp/logs/misp-workers-errors.log
   stdout_logfile=/var/www/MISP/app/tmp/logs/misp-workers.log
   directory=/var/www/MISP
   user=www-data
   
   [program:email]
   directory=/var/www/MISP
   command=/var/www/MISP/app/Console/cake start_worker email
   process_name=%(program_name)s_%(process_num)02d
   numprocs=5
   autostart=true
   autorestart=true
   redirect_stderr=false
   stderr_logfile=/var/www/MISP/app/tmp/logs/misp-workers-errors.log
   stdout_logfile=/var/www/MISP/app/tmp/logs/misp-workers.log
   directory=/var/www/MISP
   user=www-data
   
   [program:update]
   directory=/var/www/MISP
   command=/var/www/MISP/app/Console/cake start_worker update
   process_name=%(program_name)s_%(process_num)02d
   numprocs=1
   autostart=true
   autorestart=true
   redirect_stderr=false
   stderr_logfile=/var/www/MISP/app/tmp/logs/misp-workers-errors.log
   stdout_logfile=/var/www/MISP/app/tmp/logs/misp-workers.log
   directory=/var/www/MISP
   user=www-data
   
   [program:cache]
   directory=/var/www/MISP
   command=/var/www/MISP/app/Console/cake start_worker cache
   process_name=%(program_name)s_%(process_num)02d
   numprocs=5
   autostart=true
   autorestart=true
   redirect_stderr=false
   stderr_logfile=/var/www/MISP/app/tmp/logs/misp-workers-errors.log
   stdout_logfile=/var/www/MISP/app/tmp/logs/misp-workers.log
   user=www-data
   

5. Restart Supervisord to load the changes:

   sudo service supervisor restart
   

6. Check Supervisord workers are running:

   $ sudo supervisorctl status
   misp-workers:cache_00            RUNNING   pid 1673228, uptime 1:37:54
   misp-workers:cache_01            RUNNING   pid 1673225, uptime 1:37:54
   misp-workers:cache_02            RUNNING   pid 1673375, uptime 1:37:53
   misp-workers:cache_03            RUNNING   pid 1673398, uptime 1:37:52
   misp-workers:cache_04            RUNNING   pid 1673303, uptime 1:37:53
   misp-workers:default_00          RUNNING   pid 1673222, uptime 1:37:54
   misp-workers:default_01          RUNNING   pid 1673385, uptime 1:37:52
   misp-workers:default_02          RUNNING   pid 1673391, uptime 1:37:52
   misp-workers:default_03          RUNNING   pid 1673223, uptime 1:37:54
   misp-workers:default_04          RUNNING   pid 1673393, uptime 1:37:52
   misp-workers:email_00            RUNNING   pid 1673394, uptime 1:37:52
   misp-workers:email_01            RUNNING   pid 1673312, uptime 1:37:53
   misp-workers:email_02            RUNNING   pid 1673224, uptime 1:37:54
   misp-workers:email_03            RUNNING   pid 1673227, uptime 1:37:54
   misp-workers:email_04            RUNNING   pid 1673333, uptime 1:37:53
   misp-workers:prio_00             RUNNING   pid 1673279, uptime 1:37:54
   misp-workers:prio_01             RUNNING   pid 1673304, uptime 1:37:53
   misp-workers:prio_02             RUNNING   pid 1673305, uptime 1:37:53
   misp-workers:prio_03             RUNNING   pid 1673232, uptime 1:37:54
   misp-workers:prio_04             RUNNING   pid 1673319, uptime 1:37:53
   misp-workers:update_00           RUNNING   pid 1673327, uptime 1:37:53
   

MISP Config

1. Go to your MISP instances Server Settings & Maintenance page, and then to the new SimpleBackgroundJobs tab.

2. Update the SimpleBackgroundJobs.supervisor_password with the password you set in the Install requirements section 3.

3. Verify Redis and other settings are correct and then set SimpleBackgroundJobs.enabled to true.

4. Use MISP normally and visit Administration -> Jobs to check Jobs are running correctly. If there are any issues check the logs:

   • /var/www/MISP/app/tmp/logs/misp-workers-errors.log
   • /var/www/MISP/app/tmp/logs/misp-workers.log

Notes

Scheduled tasks (TasksController) are not supported with the new backend, however this feature is going to be deprecated, it is recommended to use cron jobs instead.

Please make supervisord only listen on localhost:

[inet_http_server]
port=127.0.0.1:9001
username=supervisor
password=PWD_CHANGE_ME

updated @RichieB2B

Just a few tips based on our "switch" to this new style on our test misp machine. Will appreciate any further ideas, thought:

1. Needed to composer update & --dependencies to perform the installation of php packages successfully:

cd /var/www/MISP/app
sudo -u www-data php composer.phar update
sudo -u www-data php composer.phar --with-all-dependencies require supervisorphp/supervisor \
    guzzlehttp/guzzle \
    php-http/message  \
    lstrojny/fxmlrpc

2. supervisorctl will not start the processes until the GUI (SimpleBackgroundJobs tab) is configured. Be careful - the "green lines" mean that it is NOT configured actually - so you need to double-click throught them...

• Once this has been setup and enabled, restart the supervisor.service and everything comes up. Tested by "Add Event" --> Published OK, correctly displayed in misp-workers.log.

Just to be noted - Worker type: supervisord_status in Workers page is displayed as red, but assuming this is normal...?

Hello @hlijan, thanks for the feedback

1. Updated the guide and will soon update: https://www.circl.lu/doc/misp/appendices/#appendix-g-simplebackgroundjobs-migration-guide (the official guide will be maintained here).
2. GUI settings issue was fixed in develop branch see: MISP/MISP#8053

Worker type: supervisord_status in Workers page is displayed as red, but assuming this is normal...?

This was fixed by MISP/MISP#8067, will be merged soon.

Thanks again for the feedback, do not hesitate to open a pull request to the guides if you find something else:
https://github.com/MISP/misp-book/tree/main/appendices
https://github.com/MISP/MISP/blob/2.4/docs/background-jobs-migration-guide.md

Hi,

I have this issue in logs after following the guide

Error: Undefined class constant 'SERIALIZER_JSON'
#0 /var/www/MISP/app/Lib/Tools/BackgroundJobsTool.php(118): BackgroundJobsTool->createRedisConnection()
#1 /var/www/MISP/app/Console/Command/StartWorkerShell.php(24): BackgroundJobsTool->__construct(Array)
#2 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/ShellDispatcher.php(221): StartWorkerShell->initialize()
#3 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/ShellDispatcher.php(66): ShellDispatcher->dispatch()
#4 /var/www/MISP/app/Console/cake.php(45): ShellDispatcher::run(Array)

What i am missing?

Hello @pietrogu ,
the constant SERIALIZER_JSON is defined in the php-redis package, maybe you are missing that. Try with:

  sudo apt-get -y install php-redis
  sudo phpenmod redis
  sudo service apache2 restart

If that doesnt work please add an issue here https://github.com/MISP/MISP/issues

Hi,

in some way i solved this, but a new error showed up:

Error: Call to a member function "blpop()" on null

Hope you can help me also on this

Would be good to have RHEL/CentOS version as well. Differences so far:

1. yum install (obvious)
2. path is /etc/supervisord.conf and /etc/supervisord.d/
3. files in /etc/supervisord.d/ should end in .ini

Actually using the worker view requires php-http/message-factory, and that does not appear to be installed?

I created a RHEL (8) version: https://gist.github.com/Wachizungu/1497109c9e73d173953a0efe82810dd4
Feel free to merge / provide feedback.

I created a RHEL (8) version: https://gist.github.com/Wachizungu/1497109c9e73d173953a0efe82810dd4 Feel free to merge / provide feedback.

Thanks @Wachizungu, I'll added a link your gist.
Could you create a PR a adding the additional steps for RHEL in the guide under the MISP repo?
https://github.com/MISP/MISP/blob/2.4/docs/background-jobs-migration-guide.md