Skip to content

Instantly share code, notes, and snippets.

@davidlesieur
Last active December 30, 2023 21:05
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save davidlesieur/e1dafd09636a4bb333ad360e4b2c5d6d to your computer and use it in GitHub Desktop.
Save davidlesieur/e1dafd09636a4bb333ad360e4b2c5d6d to your computer and use it in GitHub Desktop.
Deploying KerkoApp on Ubuntu 20.04 with nginx and gunicorn

Deploying KerkoApp 0.9 on Ubuntu 20.04 with nginx and gunicorn

⚠️ Update: The instructions below are superseded by the Kerko documentation.

These instructions will detail the steps and configurations required to get KerkoApp running on an Ubuntu 20.04 web server, using Gunicorn as the WSGI container and nginx as a HTTP proxy.

The general setup is the same as any Flask application, but some KerkoApp-specific steps are also covered here.

Install some required packages, including Python 3.8:

sudo apt install nginx python3 python3-pip python3-venv

Create the user who will run the app:

sudo groupadd --system kerkoapp
sudo useradd --gid kerkoapp --shell /bin/bash --create-home --home-dir /home/kerkoapp --groups www-data kerkoapp

Switch to that user to create a Python virtual environment and to install KerkoApp:

sudo su kerkoapp
python3 -m venv ~/venv
source ~/venv/bin/activate
git clone --branch 0.9 https://github.com/whiskyechobravo/kerkoapp.git ~/kerkoapp
cd ~/kerkoapp
pip3 install -r requirements/run.txt
pip3 install gunicorn

In that user's account, create the ~/kerkoapp/.env file. Its content should look like the following:

# Kerko variables.
SECRET_KEY=MY_SECRET_KEY
KERKO_TITLE=MY_LIBRARY_TITLE
KERKO_ZOTERO_API_KEY=MY_ZOTERO_API_KEY
KERKO_ZOTERO_LIBRARY_ID=MY_ZOTERO_LIBRARY_ID
KERKO_ZOTERO_LIBRARY_TYPE=MY_ZOTERO_LIBRARY_TYPE

# KerkoApp variables.
FLASK_APP=kerkoapp.py
FLASK_ENV=production
PROXY_FIX=True

You must set SECRET_KEY with a random string, and KERKO_ZOTERO_API_KEY, KERKO_ZOTERO_LIBRARY_ID, and KERKO_ZOTERO_LIBRARY_TYPE with appropriate values (see the KerkoApp documentation for details).

The above environment variables are the bare minimum required. You may configure other variables described in KerkoApp's documentation, but it may be a good idea to try first with a basic configuration.

Once the .env file is ready, Kerko should be able to talk to Zotero. Have Kerko retrieve your Zotero library's data by running the following command:

LOGGING_LEVEL=INFO flask kerko sync

Depending on your library's size, this process can complete within less than a minute to an hour or more. Zotero throttles API requests to prevent its server from getting overloaded, so sometimes you may see HTTP 500 error messages in KerkoApp's output; just ignore those, Kerko will resume synchronization a few minutes later.

Now that synchronization works, configure the cron task that will synchronize data from your Zotero library on a regular basis. Run the following command (always from the kerkoapp user):

crontab -e

That will launch the default nano editor. Add the following line at the very bottom, then save the file and exit the editor:

10 4 * * * cd /home/kerkoapp/kerkoapp && /home/kerkoapp/venv/bin/flask kerko sync

That will synchronize the data once a day, at 4:10am. You may specify a different time, of course.

Now exit from the kerkoapp user's shell, and go back to your usual sudoer account to finish the installation.

Configure a socket that will let Gunicorn to speak with nginx. As the superuser, create the /etc/systemd/system/kerkoapp.socket file, with the following content:

[Unit]
Description=KerkoApp socket

[Socket]
ListenStream=/run/kerkoapp.socket

[Install]
WantedBy=kerkoapp.target

Configure a service that will run Gunicorn. As the superuser, create the /etc/systemd/system/kerkoapp.service file, with the following content:

[Unit]
Description=KerkoApp daemon
Requires=kerkoapp.socket
After=network.target

[Service]
WorkingDirectory=/home/kerkoapp/kerkoapp
EnvironmentFile=/home/kerkoapp/kerkoapp/.env
ExecStart=/home/kerkoapp/venv/bin/gunicorn kerkoapp:app --name kerkoapp --user kerkoapp --group www-data --workers 4 --log-level warning --error-logfile - --access-logfile - --bind unix:/run/kerkoapp.socket
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Reload the systemd configurations, then enable and start the new service by running the following commands:

sudo systemctl daemon-reload
sudo systemctl enable kerkoapp.socket
sudo systemctl enable kerkoapp.service
sudo systemctl start kerkoapp.socket

You can check whether the socket is active with this command:

sudo systemctl status kerkoapp.socket

If it shows up as 'active (listening)', you may now verify that it triggers the service:

curl --unix-socket /run/kerkoapp.socket localhost
sudo systemctl status kerkoapp.service

The curl command should output some HTML, and the systemctl command should now show the KerkoApp daemon as 'active (running)'.

If either the socket or the service doesn't work, you might want to check for errors in the log:

sudo journalctl --unit=kerkoapp

Once all of the above works, configure nginx to pass requests to Gunicorn. As the superuser, create the /etc/nginx/sites-available/kerkoapp.conf file with the following content, replacing example.com with your actual domain name:

server {
    listen 80;
    server_name example.com;

    location / {
        proxy_redirect off;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $server_name;
        proxy_set_header X-Forwarded-Proto $scheme;

        if (!-f $request_filename) {
            proxy_pass http://unix:/run/kerkoapp.socket;
            break;
        }
    }
}

Enable the site by running the following command:

sudo ln -s /etc/nginx/sites-available/kerkoapp.conf /etc/nginx/sites-enabled/

Have nginx test the configuration:

sudo nginx -t

If the command reports the configuration test as successful, reload nginx to make the configuration changes effective:

sudo service nginx reload

You should now be able to view your KerkoApp site at http://example.com!

The nginx configuration above will serve the application using the HTTP protocol. As for any website, HTTPS is strongly recommended. That will require that you install a SSL certificate and configure nginx to use it; that exercise is left to the reader.

Note - Changing KerkoApp's configuration: many of KerkoApp's configuration variables can change the structure of Kerko's search index. If you edit the .env later, you may have to clean the index, re-synchronize the data from Zotero, and restart KerkoApp. You will have to run the following commands:

sudo -iu kerkoapp bash -c 'cd /home/kerkoapp/kerkoapp && /home/kerkoapp/venv/bin/flask kerko clean index'
sudo -iu kerkoapp bash -c 'cd /home/kerkoapp/kerkoapp && LOGGING_LEVEL=INFO /home/kerkoapp/venv/bin/flask kerko sync'
sudo systemctl stop kerkoapp.socket
sudo systemctl start kerkoapp.socket

Did these installation steps work for you? Did we miss any step? Any improvements to suggest? Let us know by commenting below!

@ohnemax
Copy link

ohnemax commented Feb 4, 2021

Thanks for the great instruction! I'm trying to setup kerkoapp in a subfolder of the nginx server. Any additional hints on how to do that?

@davidlesieur
Copy link
Author

@ohnemax, I don't expect any particular issues in changing the location.

@bucchere
Copy link

Some of the best instructions I've ever read, nice work!

@davidlesieur
Copy link
Author

@bucchere, thanks for the comment. I take it that it worked for you. Good to know!

@bucchere
Copy link

Yes, and beyond that, your way of configuring nginx using sockets is way cleaner and better than the janky way I've done it before with one web server proxying to the other on different ports, very cool, and useful for other projects, too.

@jtradel
Copy link

jtradel commented Aug 14, 2021

Very helpful for a quick rescue of a sample KerkoApp install I wanted somebody to look at--and she didn't get to it until after the built-in dev server had fallen over. One comment: If one wants a solid, but very fast to deploy and configure webserver, I'd recommend caddy. The entirety of my config file to support this is:

my.fqdn.example.com {
  reverse_proxy unix//run/kerkoapp.socket
}

which gives me HTTP on port 80 that redirects to HTTPS on port 443 with a cert from Let's Encrypt that is installed and renewed behind the scenes. There are lots of knobs to twiddle (though not quite as many as on apache or nginx), but the defaults are remarkably useful.

Thanks.

@kmafoudji
Copy link

Thank you very much for this superb work.
I was looking for something similar and luckily I found KerkoApp. By following the tutorial I was able to install version 1.0.0 on my local server, it works perfectly.
Now I would like to be able to deploy on my GoDaddy Shared Hosting
Capture d'écran 2023-12-28 000418
Would you please give me suggestions on how to host on Godaddy or other hostings.

Thanks

@davidlesieur
Copy link
Author

@kmafoudji I don't use GoDaddy's services, thus I cannot provide instructions for it. I can't say for GoDaddy, but in my experience control panel-based hosting solutions tend to be poorly adapted to hosting Python applications, unfortunately!

@davidlesieur
Copy link
Author

Please note that the instructions found on this Gist are now replaced by: https://whiskyechobravo.github.io/kerko/latest/deploying/

@kmafoudji
Copy link

@kmafoudji I don't use GoDaddy's services, thus I cannot provide instructions for it. I can't say for GoDaddy, but in my experience control panel-based hosting solutions tend to be poorly adapted to hosting Python applications, unfortunately!

Thank you for the prompt response.
I'm going to find hosting on a dedicated Ubuntu server.

@kmafoudji
Copy link

Please note that the instructions found on this Gist are now replaced by: https://whiskyechobravo.github.io/kerko/latest/deploying/

Initially I followed this guide to install version 1.1.0. Unfortunately I was stuck with a Flask Error. I also tried right now but still having challenges

Capture d’écran du 2023-12-28 20-35-33

@davidlesieur
Copy link
Author

davidlesieur commented Dec 29, 2023

@kmafoudji Looks like you didn't run the command from the directory where KerkoApp is installed (where wsgi.py can be found, as the error message indicates). But here is not the place to discuss such things; if you encounter bugs, please post to https://github.com/whiskyechobravo/kerkoapp/issues.

@kmafoudji
Copy link

@davidlesieur It's working fine. Great thanks.
Will continue on customization (user interfaces and translation into Fr). And will post my questions on the link provided.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment