2026-10-16 Migrating to a new VPS, gonna be in read-only mode, hopefully only an hour or two. Reload this gist for updates, or drop by the chat room.
- pushcx: rsync -vaP --exclude 'public/cache/' --exclude 'bundle/' --exclude 'comment_redirects' l:lobsters/shared/ 2026-10-17_shared
- pushcx: rsync -vaP root@l:/etc/ 2026-10-17_etc
- pushcx: rsync -vaP l:.local/ 2026-10-17_local
- DNS: turn down ttl on lobste.rs and lobsters.dev
- hatchbox: save ENV vars
- hatchbox console: create cluster, assign roles
- NEW console: create reserved IP, check for spam blacklists (Spamhaus, Barracuda, SpamCop), recycle until green
- NEW console: create instance
- hatchbox initial provision
- link to this gist from app/views/layouts/application.html.erb
- set the site to read-only mode (config/application.rb)
- lobsters-deploy
- hatchbox OLD: disable cron jobs, queue
- OLD: systemctl stop postfix
- OLD: cd ~/lobsters/current
rack_attack would never finish, every non-cache hit writes to it, safe to drop, thanks chamlisfor db in primary queue cache rack_attack; do bin/sqlite3 storage/$db.sqlite3 "PRAGMA wal_checkpoint(TRUNCATE)" bin/sqlite3 storage/$db.sqlite3 "vacuum" bin/sqlite3 storage/$db.sqlite3 ".backup '/home/deploy/lobsters/shared/$db.sqlite3'" done - pushcx: rsync -vaP --exclude 'public/cache/' --exclude 'bundle/' --exclude 'comment_redirects' l:lobsters/shared/ 2026-10-17_shared
- OLD: root:
. /home/deploy/lobsters/shared/etc/restic-env ; restic backup --no-scan /etc - OLD:
. /home/deploy/lobsters/shared/etc/restic-env ; restic backup --no-scan /home/deploy/lobsters/shared/etc /home/deploy/lobsters/shared/log #{home}/.*_history - pushcx: rsync -vaP root@l:/etc/ 2026-10-17_etc
- pushcx: rsync -vaP 2026-10-17_local/ web2:.local
- NEW: reverse dns ptr
- NEW: apt update && apt upgrade && reboot
- NEW:
dpkg-reconfigure --priority=low unattended-upgrades - copy hatchbox ssh key up to web2
- web2: rsync -vaP --exclude comment_redirects --exclude 'log/*bz2' --exclude bundle deploy@lobste.rs:lobsters/shared/ lobsters/shared
- rm hatchbox ssh key from web2
- NEW: replace shared/storage/*sqlite3 with .backup versions from parent dir
- hatchbox: in new cluster, create 'lobsters' app, copy env vars from OLD
- deploy
- web2: DISABLE_DATABASE_ENVIRONMENT_CHECK=1 bundle exec rails db:reset:rack_attack
- hatchbox web2: disable server, cron jobs, queue
- web2:
ln -s /home/deploy/lobsters/current/hatchbox/root-deploy.service /etc/systemd/system ln -s /home/deploy/lobsters/current/hatchbox/root-deploy.path /etc/systemd/system systemctl daemon-reload systemctl enable --now root-deploy.service systemctl enable --now root-deploy.path ```
- deploy
- pushcx: rsync -vaP 2026-10-17_etc/dkimkeys/ root@web2:/etc/dkimkeys
- NEW: chown -R opendkim:opendkim /etc/dkimkeys
- pushcx: rsync -vaP 2026-10-17_etc/iptables/ root@web2:/etc/iptables
- NEW: chown -R root:root /etc/iptables
- hatchbox OLD: confirm cron jobs, queue still disabled
- NEW:
systemctl status anubis@lobsters opendkim postfix caddyall active. - NEW: Puma listens on 127.0.0.1:9001, Anubis on 9000 [hatchbox/lobsters-service.*.port.conf, hatchbox/lobsters.anubis.env].
- NEW:
opendkim-testkey -d lobste.rs -s mail -vvvsays key OK (same key as DNS). - NEW:
iptables -S | wc -lis still the same as OLD (the save did not erase it). - NEW:
bin/sqlite3 -versionis the gem-built version, andwhich sqlite3finds only the alias [hatchbox/bash_aliases]. - NEW:
curl -sI --resolve lobste.rs:443:NEW https://lobste.rs/gives a valid cert and a response. - DNS: change A and AAAA to 64.177.113.22 on lobste.rs and lobsters.dev
- spf, dkim
- NEW:
. /home/deploy/lobsters/shared/etc/restic-env ; restic backup --no-scan /home/deploy/lobsters/shared/etc /home/deploy/lobsters/shared/log /home/deploy/lobsters/shared/database-backups #{home}/.*_history - NEW: root:
. /home/deploy/lobsters/shared/etc/restic-env ; restic backup --no-scan /etc - pushcx: see both in 'restic snapshots'
- [-] manually try a mail tester (
echo test | mail -s test a@example.com) - hatchbox NEW: copy cron jobs from web1
- disable mailing list mode
- hatchbox enable queue
- new commit reverting the read-only and notice
- new commit updating /about
- deploy
- sign up with email host, configure postfix to relay
- migrate mockturtle
- OLD console: turn off vps
We're stuck at the moment and taking a break (19:45 Chicago time).
The new host filters outgoing port 25. I checked this before trying to set up with them and their documentation made it sound like requiring TLS would be fine. Nope.
All our email is three buckets:
- transactional (email confirmation, password resets, etc), around 30/day
- opt-out notifications of user-to-user interactions (replies, DMs), around 200/day
- opt-in mailing list mode
Mailing list mode is big: (20-50 stories per day * 95 users) + (200-500 comments per day * 58 users) = 13k - 28k emails per day. This is enough to scare VPS hosts, who will assume this level of outgoing SMTP is spam regardless of the site. Or that it's running a Wordpress install that will be popped to send spam.
So, options:
- Stay with existing host that hates life.
- Hope new host believes us and allows us to send email. This feels vanishingly unlikely, but we've asked politely.
- Take ~calvin up on his offer to rack a server; Hurricane Electric doesn't filter outbound port 25.
- Peter racks a server in the Chicagoland area that also isn't filtered. (Preferred over previous because many state laws now censor websites and reducing jurisdictional complexity reduces those risks.)
- Give up on self-hosting and use a mail service. The web is corporate, sucks to want to run your own things.
Problem with that last one is that our volume puts us into a plan that would be >= $500/month. c We could kill reply notifications and mailing list mode, but that just puts us back into the same set of decisions, though at ~30/day.
OK, after taking a break to think, eat, rest:
I don't expect to hear back about the firewall until Monday morning at best. I don't want to revert: it would be more work, and it would save nothing if the new host doesn't work out.
I'm going to disable mailing list mode, drop us down to just transactional + notification emails, then sign up for one of these email services. We'll probably operate in this degraded mode for at least a few couple days. That gives some breathing room to hear back from the new host, evaluate hosting options like calvin's generous offer, research me racking a server here, and hear back from another VPS host we have a couple ins with.
- revert dns if changed
- new commit reverting the read-only and notice
- deploy
- OLD: systemctl --user start lobsters-solid_queue
- OLD: systemctl start postfix
- add to cron: find /home/lobsters/shared/storage -type d -empty -delete
- pushcx: delete
2026-10-17_backups - 1 day later: check automatic restic snapshot appears
- OLD console: delete vps
- DNS: turn up TTL
- DNS: remove OLD from SPF
- should prod have been vacuuming the sqlites on a schedule?
Hopefully they do permit it, as that sounds like the easiest of the options. It's for an opt-in feature and it sounds like less than a 100 users receive most of the mail.