There is already a guide on scaling your Mastodon server up. This is a short guide on scaling your Mastodon server down.
I.e., maybe you want to run a small instance of <100 active users, and you want to keep your cloud costs reasonable.
So you might be running everything on a single machine, with limited memory and CPU. (In my case, I was using a t3.medium instance with 2 vCPUs and 4GB of RAM.) How
do you do this?
Note that I'm not a Ruby or Sidekiq expert, and most of this stuff I figured out through trial and error.
Just like in the scaling up doc, I found PgBouncer useful to keep response times fast. It's worth using.
I set default_pool_size to 20 and max_client_conn to 1000 in /etc/pgbouncer/pgbouncer.ini. I don't think these matter much.
Just make sure you don't end up using more connections than the max_client_conn.
I did not bother to try running this. And nobody has complained so far.
Edit /etc/postgresql/10/main/postgresql.conf and use PgTune to dial things down a bit.
Otherwise, Postgres can consume a lot of memory.
I set PgTune to target 750MB, 2 CPUs, and 200 connections. In practice, I don't think Postgres really uses 750MB though – more like 500MB.
This one is the hardest. Sidekiq uses a lot of memory, and plus Mastodon has memory leaks. If you don't do anything about it, Sidekiq will slowly use more and more memory until your poor 4GB machine is swamped and crashes.
First off, I set DB_POOL and -c (concurrency) to 30, and I only used one Sidekiq process.
Next, you can add this config to your systemd file (under [Service]):
Restart=always
RuntimeMaxSec=10800
This will automatically restart the Sidekiq process every 3 hours, drastically cutting down on the total memory consumed.
I also added the same thing to the web (Puma) and streaming processes, just in case they leak memory too. (In practice, I'm pretty sure only Sidekiq and maybe Puma leak memory.)
See above about restarts and memory leaks. For this, I set WEB_CONCURRENCY to 1 and MAX_THREADS to 10.
One process only.
Use it. If you forget to enable this while installing Ruby, then your memory will increase by ~30% or so (IIRC). jemalloc is already mentioned in the Mastodon docs.
Use it. Don't store media directly on the machine. In AWS at least, you'll pay much more for EBS storage than S3 storage.
S3 has a bunch of tiering conceps like Infrequent access, but this seems pointless for Mastodon because it only kicks in for objects >128KB and most images, emojis, etc., are going to be smaller than that. I didn't bother to set this up.
Don't forget to clear remote media periodically. Otherwise you'll be hosting images and videos from across the fediverse, adding enormously to your S3 costs.
I just use a really simple script that runs pg_dump and then uses s3cmd to upload to S3. Maybe there is a more efficient way to do this, but I haven't found it.
Avoid this. I found this was really hard to scale well. Each Mastodon instance is very memory-hungry and will fight with the other ones. Especially if there is some viral video that hits both instances at the same time, so now they're both running ffmpeg at the same time.
Updating Mastodon can be very memory-intensive, especially when it comes to Webpack compilation. If you've only got 4GB of RAM to work with, then you're better off shutting down your server during the upgrade. Or at the very least, restarting all processes beforehand to reclaim some memory.
If you have limited disk space available, you can also do yarn cache clean to reclaim some space from Yarn's cache.
Even with all this work, Mastodon will go down sometimes. You will be running at ~3.5GB on average, with only a 500MB window for running other stuff (backups, ImageMagick, ffmpeg, etc.). If too many people upload videos at the same time, ffmpeg can crash the whole server.
I had a running joke:
Instead of "five 9s" of reliability, we have "five 7s."
Set up some alerts for when the server goes down, and maybe to auto-restart it if you'd like.
Or just embrace the fact that your Mastodon server will go down sometimes. Go for a walk. Call up a loved one. Find peace in the impermanence of things.