MaximilianKohler/Listmonk-tutorial.md

## Listmonk-tutorial.md

      
    Raw
  

              Listmonk-tutorial.md
            
          
    Listmonk setup and usage guide

When I first set up Listmonk it was to use with Amazon SES. At the time Amazon would give you free 62,000 emails/mo if you sent them from an EC2 instance. So EC2 was the best server to use. In mid 2023 Amazon ended that, so now you can use whatever server you like, which makes things much easier. It shouldn't be too hard to convert these directions to another server host of your choice.
I used Hetzner with another build, and once my free EC2 year ended the AWS t2.micro cost me $14/mo. Hetzner has better specs and costs me $5/mo, so I added an nginx vhost and moved listmonk to the same server. Here's a $20 credit for Hetzner.
There is also the possibility to use the 1-click installers for their featured hosts: https://listmonk.app/ - listed under "Hosting providers". I'm not familiar with any of them but there are lots of new guides now: https://listmonk.app/docs/installation/#tutorials
If you're using Hetzner (other hosts do this too) you have to wait a month and then request that they unblock the SMTP ports. Or use port 587.
I am a total noob and it was a massive pain in the ass to get this setup because there wasn't a detailed guide at the time. So I created this to help others. The Listmonk docs were generally pretty poor and not informative. It's an open source project so feel free to help improve it (and this guide). I moved parts of this tutorial to the official docs.
Table of contents:

^{How-to easily create a TOC.}

Directions for EC2 micro install using docker

Docker update
To edit the two main files:

Change postgres password


Edit hostname
SSL:
Time zone:
SMTP:
Bounce processing:
Uploads:
Static directory:
Updating your server:
Restart containers:


Upgrading listmonk:
Downgrading listmonk:
Importing
Logs:
Cloudflare:
Wordpress:
PostgreSQL:

Basic usage and commands:
Backup:
Restore:


Installing listmonk without docker (via binary)
Other places to find solutions:

Directions for EC2 micro install

We'll use a combination of these two links:

https://levelup.gitconnected.com/setting-up-your-own-bulk-mail-server-using-sendy-and-aws-20d05ee01362
https://vpsfix.com/7624/install-sendy-aws-ec2-instance-ssl/

Generally the first one, but sometimes the 2nd one will go into more detail.
Putty and FileZilla will be your main tools on Windows. Portableapps.com has both of them.
DigitalOcean has some nice guides. For example, their How To Install PostgreSQL on Ubuntu 22.04, and another for docker linked below.

Create security group, as directed, except use port 9000 instead of 10000.
Create EC2 instance with latest version of Ubuntu Server. t2.micro is fine.
Set up key pair to login via SSH using Putty (don't bother with Windows 10 native option). https://vpsfix.com/3052/key-pair-login-amazon-ec2-instance/
Set up elastic IP
Add Elastic IP to your domain’s DNS records
Connect to EC2 via SSH
Update the OS. Can run dpkg-reconfigure tzdata to change timezone. And sudo apt autoremove to remove unnecessary packages.
Run these commands (first, see Docker update below):

sudo apt install postgresql docker docker-compose
mkdir listmonk && cd listmonk
sudo sh -c "$(curl -fsSL https://raw.githubusercontent.com/knadh/listmonk/master/install-prod.sh)"

Note: do not use "sudo -i". It will install things into the wrong directory. If you don't want to type "sudo" all the time, "sudo bash" might work. But for these, copy-paste is fine.

EDIT: I actually removed sudo from many lines below, as it was unnecessary and possibly caused by erroneously putting sudo in front of mkdir listmonk to start with. If you have to put sudo in front of any lines below, you probably messed something up. These things are finicky, and commands have to be so precise or you can run into so many problems. Another reason a novice like me should not be having to write this guide.
Lesson: don't use sudo unnecessarily. Especially when creating directories and files. Since it sets the permissions of them to root.
You should now be able to load listmonk in your browser at elasticIP:9000.

If it won't load (like in my case), wipe your EC2 drive and try again:

Visit your EC2 dashboard -> Instances -> Select instance -> top right; "actions" -> monitor and troubleshoot -> replace root volume.

Still doesn't load (like in my case)? Wipe the drive and try again.

Still doesn't load (like in my case)? Get a linux expert to mess with it for a few hours. Maybe try these commands:

View running containers: sudo docker container ls
Restart containers:
sudo docker-compose restart app db

sudo docker container stop listmonk_db listmonk_app
sudo docker container rm listmonk_db listmonk_app

sudo docker-compose up -d app db

Then wipe the drive and try again. Trust me, it magically works the 3rd time.
Docker update:

The docker-compose command is being depreciated:

https://www.docker.com/blog/new-docker-compose-v2-and-v1-deprecation/
"Type docker compose instead of docker-compose in your favorite terminal." https://docs.docker.com/compose/migrate/
apt install postgresql docker docker-compose may be changed to apt install postgresql docker docker-compose-plugin https://docs.docker.com/compose/install/linux/ or sudo apt install postgresql docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin https://docs.docker.com/engine/install/ubuntu/#install-docker-engine.
To upgrade, you have to uninstall the old packages first https://docs.docker.com/engine/install/ubuntu/#uninstall-old-versions
How To Install and Use Docker on Ubuntu 22.04 https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-22-04

Per the link above, and this discussion https://serverfault.com/questions/1110365/when-installing-docker-on-ubuntu-why-isnt-it-as-easy-as-apt-get-install-docker, it seems that the apt install docker docker-compose command installs an older version of docker from ubuntu. If you want to use the latest version you have to do the extra steps in the links above.
Since listmonk is updating their scripts & docs to the latest version of docker knadh/listmonk#1431 we should too.
Additionally, a link above has this tip:

Receiving errors when trying to run without root?
The docker user group exists but contains no users, which is why you’re required to use sudo to run Docker commands. Continue to Linux postinstall to allow non-privileged users to run Docker commands and for other optional configuration steps.

To edit the two main files:

nano ./docker-compose.yml
nano ./config.toml

Or use FileZilla:
Host: elastic IP
Port: blank
User: ubuntu
PW: key file
Protocol: SFTP SSH file transfer

Note that if you use FileZilla with an editor like Notepad++ you will need to make sure that newly created files are using LF line breaks instead of CRLF. If you create the file with nano it does it already.
After you edit those files (IE: changing admin password) run this restart command:
sudo docker-compose restart app db
Change postgres password:

The postgres passwords in those files are read-only. If you want to change the postgres password use this command:

psql postgresql://listmonk:CURRENT_PASSWORD@127.0.0.1:9432/listmonk

or

psql -U listmonk -h localhost -p 9432 listmonk

then

ALTER USER listmonk PASSWORD 'NEW_PASSWORD';
Edit hostname

Eg:
listmonk.mysite.com
SSL:

The below instructions use nginx and are a bit more complicated. You can alternatively use Caddy, which is easier.
https://yasoob.me/posts/setting-up-listmonk-opensource-newsletter-mailing/#setting-up-ssl
First copy the contents of the "nginx.conf" file and paste into text file. Replace (ctrl+h) all 4 instances of "example.com" with the subdomain hostname from the previous step -- IE: listmonk.mysite.com.
Go back to cmd:
mkdir -p data/nginx
cd data/nginx
nano ./nginx.conf

Paste in your edited nginx.conf. Save and exit.
Edit the "init-letsencrypt.sh" in a text file.
Go back to cmd:
cd
nano ./init-letsencrypt.sh

paste in the contents of the "init-letsencrypt.sh" text file. Save and exit.
Once it didn't paste in correctly and caused an error, so I had to edit the file with FileZilla instead.
If you've updated to the latest version of docker (see above) you'll need to replace all 7 instances of docker-compose with docker compose.
nano ./docker-compose.yml
paste in the NGINX and certbot lines.
Add this extra     restart: unless-stopped line under certbot:
  certbot:
    image: certbot/certbot
    restart: unless-stopped

Save and exit. Then run:
sudo bash ./init-letsencrypt.sh
sudo docker start certbot
sudo docker-compose stop ; sudo docker-compose up -d

It should be auto-renewing. If it doesn't, check to see if the certbot container is running:
sudo docker container ls.

You can check the logs with sudo docker logs certbot -t.
If you have trouble, here's a discussion about Certbot not restarting after rebooting the server: wmnnd/nginx-certbot#9 (comment)
Time zone:

https://listmonk.app/docs/configuration/#time-zone
Discussion: knadh/listmonk#637
SMTP:

SMTP credentials: The instructions in the first link for "Setting up Amazon SES" by creating a user with credentials won't work. You specifically have to create a new SMTP user knadh/listmonk#1053
STARTTLS port 587 vs TLS/SSL port 465 knadh/listmonk#1615
"HELO is SMTP specific. If your SMTP server does not ask for it, it should be left blank" knadh/listmonk#448
Bounce processing:

https://listmonk.app/docs/bounces/#external-webhooks
You can use Amazon's test emails to make sure everything's working: https://listmonk.app/docs/bounces/#verification
In the SES console, it doesn't seem like you have to verify every email you want to send from, in addition to the domain. It seems that I can send with anyemailiwant@myses-verifieddomain.com but I don't get email notifications without verifying the email and enabling "Email feedback forwarding".
It seems like you only have to enable "Feedback notifications" for the sender domain, rather than every email you want to send from.
Discussion: knadh/listmonk#166 (comment)
Exporting bounces: knadh/listmonk#863
Uploads:

https://listmonk.app/docs/configuration/#media-uploads
Solution: knadh/listmonk#1169 (comment)
Other discussions: knadh/listmonk#1163 (comment)
Static directory:

https://yasoob.me/posts/setting-up-listmonk-opensource-newsletter-mailing/#custom-static-files
You can do:
# cd to the listmonk directory with the config.toml file
cd listmonk
mkdir static && cd static
wget -O - https://github.com/knadh/listmonk/archive/master.tar.gz | tar xz --strip=2 "listmonk-master/static"

or combined in one command:
mkdir -p /home/ubuntu/listmonk/static ; wget -O - https://github.com/knadh/listmonk/archive/master.tar.gz | tar xz -C /home/ubuntu/listmonk/static --strip=2 "listmonk-master/static"

Then edit the docker-compose.yml file as shown in the yasoob link above.
You may have to pull the files from a different branch: knadh/listmonk#1708 (comment)
Updating your server:

sudo apt update && sudo apt upgrade -y
Restart containers:

I was using:

sudo docker-compose restart app db nginx certbot
But it seems to often not be sufficient, so I switched to:

sudo docker-compose stop ; sudo docker-compose up -d
Upgrading listmonk:

https://listmonk.app/docs/upgrade/
Or, per https://github.com/knadh/listmonk/releases/tag/v2.5.0
# cd /directory/with/docker-compose.yml
# In our case that's already our home directory, so we can skip it.

sudo docker-compose down
sudo docker-compose pull && sudo docker-compose run --rm app ./listmonk --upgrade
sudo docker-compose up -d app db nginx certbot

Downgrading listmonk:

https://listmonk.app/docs/upgrade/#downgrade
Discussion: knadh/listmonk#1457
Importing:

You can't import email duplicates (with different name) to the same list. If you import a list of 10 people with 1 duplicate, it will say "finished 10/10" and no errors.
Logs:

Amazon SES logs:

Here's a FOSS application that can be used to log the email addresses of people who received your campaign:
SES Dashboard https://sesdashboard.com/ - https://github.com/Nikeev/sesdashboard
Docker logs:

You can check certbot logs with sudo docker logs certbot -t. You can switch out certbot with the other containers.
docker logs --help for more info. https://stackoverflow.com/a/47829441/5862615
More info: knadh/listmonk#13 (comment)
Binary logs:

knadh/listmonk#1462 -- to clarify, stdout is not saved to a file by default, so if you want to save the logs to a file you'll need to use ./listmonk > log.txt.
There are various options for the service file depending on your systemd version:

https://stackoverflow.com/questions/37585758/how-to-redirect-output-of-systemd-service-to-a-file
https://unix.stackexchange.com/questions/321709/redirect-systemd-service-logs-to-file

To append, use one of these:

ExecStart=/bin/sh -c 'exec /usr/bin/listmonk [arguments] >>/etc/listmonk/listmonk1.log 2>>/etc/listmonk/listmonk2.log'

or

ExecStart=/bin/sh -c '/usr/bin/listmonk --config /etc/listmonk/config.toml --static-dir /etc/listmonk/static 2>&1 > /etc/listmonk/listmonk.log'

or

ExecStart=/bin/bash -ce "exec /usr/bin/listmonk --config /etc/listmonk/config.toml --static-dir /etc/listmonk/static >>/etc/listmonk/listmonk.log 2>&1"
On CentOS 7 I'm using the 3rd one.
Postgres logs:

https://betterstack.com/community/guides/logging/how-to-start-logging-with-postgresql/
Login with the superuser account via sudo -i -u postgres. Then SHOW data_directory;.
Cloudflare:

You can use Cloudflare's free proxy to protect your server's IP address from DDoS attacks https://gist.github.com/MaximilianKohler/3bdedd0185283ac30c1f1422f9626947#ddos
Since we already have http to https redirects you'll need to set Cloudflare's "Encryption mode" to "Full or Full (strict)" to avoid the "too many redirects" error https://developers.cloudflare.com/ssl/troubleshooting/too-many-redirects/
I wondered if there would be any issues with putting a CDN in front of an email server. IE: I wondered if people might get captchas when clicking "unsubscribe" and thus be more likely to mark emails as spam. I also wonder if it may mess with statistics such as open rates. I looked into it and learned that Cloudflare is now using a non-interactive captcha method that drastically reduces the amount of puzzles that people have to solve. But see below for info on decreasing or turning off captcha.
Captcha:


Best Practices For Captcha Challenges https://community.cloudflare.com/t/community-tip-best-practices-for-captcha-challenges/56301
Ease up on challenge frequency (challenge passage setting) https://community.cloudflare.com/t/ease-up-on-challenge-frequency/60211/2

How to turn off captcha:

Lower your security level to “Essentially off” https://community.cloudflare.com/t/removing-cloudflare-captcha/191436
Didn't work for this guy https://community.cloudflare.com/t/how-to-turn-off-captcha/305523/13
or create a firewall rule to ALLOW ALL from 0.0.0.0/0 https://community.cloudflare.com/t/how-to-stop-and-disable-stupid-hcaptcha-from-our-cloudflare-site/160194/6
Check "Browser Integrity Check" too https://community.cloudflare.com/t/how-to-turn-off-cloudflare-captcha/305251/7

If you're not proxying your domain/subdomain with cloudflare, you could manually set up listmonk to use Turnstile, which can be just one click or none. Migrating to Turnstile. You'll have to edit your HTML’s <head> element. I haven't bothered trying that yet knadh/listmonk#1617.
Wordpress:

Open source plugin for WordPress / WooCommerce users to easily send subscribers through payment or a web form to listmonk: https://github.com/post-duif/integration-listmonk-wordpress-plugin
PostgreSQL:

Basic usage and commands:

Example of some basic psql commands: https://yasoob.me/posts/setting-up-listmonk-opensource-newsletter-mailing/#importing-old-subscribers
You can copy-paste multiple lines into the CMD prompt. Postgres will execute them once a semicolon is passed.
Here's an example of how to collect and insert data into the postgres db: knadh/listmonk#1629 (comment)
Show orphans: knadh/listmonk#1562
Advanced SQL searches from listmonk's UI:
knadh/listmonk#1458 (comment)

Go to Subscribers list, click on "Advanced" button to use SQL query, and put query like this:
EXISTS(SELECT 1 FROM campaign_views WHERE campaign_views.subscriber_id=subscribers.id AND campaign_views.campaign_id=<put_id_of_campaign>)

For "unopened", you change EXISTS to NOT EXISTS knadh/listmonk#1810 (comment)
Search all "gmail" addresses in a certain list while limiting results to 200: knadh/listmonk#1912 (comment)
Querying and segmenting subscribers https://listmonk.app/docs/querying-and-segmentation/
Backup:

On the Upgrade page https://listmonk.app/docs/upgrade/ it says:

it is recommended to take a backup of the Postgres database before running the --upgrade option

but commands/method to do so aren't mentioned.
Confirmed this works:

pg_dump -U listmonk -h localhost -p 9432 listmonk > pglistmonkdb.sql - uncompressed

or:

pg_dump -U listmonk -h localhost -p 9432 listmonk -Fc > listmonk-$(date +"%Y-%m-%d---%H-%M-%S").sql - built-in compression

or:

pg_dump -U listmonk -h localhost -p 9432 listmonk | gzip > listmonk-$(date +"%Y-%m-%d--%H-%M-%S").sql.gz - compressed with gzip
Then connect to EC2 with FileZilla to download the file.
You may also want to run this command to secure the file, but I'm not sure if it's necessary: chmod 0640 /home/ubuntu/listmonk-2023*
The pro of using gzip is that you can unzip it after downloading and open in plain text with notepad++. Whereas the -Fc one is gibberish in notepad++. Though notepad++ is not the best program to use for this anyway. Let me know if there's a better program.
Alternative compression method with zstd: https://gist.github.com/MaximilianKohler/e5158fcfe6de80a9069926a67afcae11?permalink_comment_id=5169065#gistcomment-5169065
It sounds like there's a way to restore and backup directly from/to your PC and your server, but maybe someone else can clarify the exact commands. https://stackoverflow.com/questions/26378098/restore-dump-on-the-remote-machine
You might be able to install postgres on your PC https://www.youtube.com/watch?v=fAOBqgR9Yfo and run modified versions of the above and below commands (using your elastic IP instead of "localhost" and "127.0.0.1", and might need to change the ports too).
When I tried psql -h elasticIP -p 5432 -d listmonk -U listmonk -W -f pglistmonkdb.sql from my PC I got a timeout error.
Restore:

Be careful, this will completely delete/erase your existing DB. You may want to do a web search for drop schema and/or mysql commands to familiarize yourself with the commands.
psql -h 127.0.0.1 -p 9432 -U listmonk
drop schema public cascade;
create schema public;
\q
psql -h 127.0.0.1 -p 9432 -U listmonk -W listmonk < pglistmonkdb.sql
sudo docker-compose restart app db nginx certbot

If the file is gzip'd you can run gunzip filename.sql.gz which changes the size and file type of the existing file (doesn't create a new one). You can reverse it with gzip filename.sql.
Alternatively, there are other commands pg_restore and "restore from gzip" that I haven't figured out exactly yet. But they can be avoided by using the non-gzip backup option above.
Not working:

psql listmonk < pglistmonkdb.sql --clean

psql -U listmonk listmonk < pglistmonkdb.sql

psql -U listmonk -W listmonk < pglistmonkdb.sql
Installing listmonk without docker (via binary)

I did it on a CentOS7 nginx vhost because the firewall was blocking docker from running.
I also added more links with more info here: https://listmonk.app/docs/installation/#tutorials
Timezone is set via host machine. On CentOS7 it's timedatectl to check. Then choose one: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones. Then set with timedatectl set-timezone America/Los_Angeles. System logs won't use the new time zone until you reboot the server.
These commands told me I had an AMD x64 CPU:
uname -m
cat /proc/cpuinfo

So I downloaded the linux_AMD64 binary from the releases page. Uploaded (any directory, I picked "public") via FileZilla.
Grant execute permissions of the listmonk binary to "owner". 744 via FileZilla or chmod +x /path-to-listmonk/listmonk.
You have to setup listmonk as a service to keep it running: knadh/listmonk#843
This guide mostly worked for me, but I had to make some modifications. And in addition to grant all privileges on database listmonk to listmonk; I had to run ALTER DATABASE listmonk OWNER TO listmonk;. I created a simplified version of the service file with greater compatibility for older OSs.
You can pair it with a simple nginx config. In my case, port 9000 is used by php-fpm, so I had to change it (in config.toml and proxy_pass) to something else, such as 9003.
If you leave the DB port on 5432 you'll have to manually change most of the commands above that use 9432. Or you could just change your config.toml to port = 9432 before running the install command ./listmonk --install.
If you're importing an existing database you probably have to restart the service after. Basic commands:
nano /etc/systemd/system/listmonk.service
nano /etc/listmonk/config.toml
systemctl daemon-reload && systemctl restart listmonk
systemctl enable listmonk.service
systemctl start listmonk.service
systemctl status listmonk
systemctl stop listmonk

I'm using this backup.sh file.
How to stop listmonk for upgrade knadh/listmonk#1113
Other places to find solutions:

If you can't find an answer in the listmonk issues you may be able to find it on the Mautic issues https://github.com/mautic/mautic/issues or their forum.


Good luck.