Skip to content

Instantly share code, notes, and snippets.

@pushcx
Last active October 25, 2026 21:23
Show Gist options
  • Select an option

  • Save pushcx/e59839c44403ea224f37a33c20232cbe to your computer and use it in GitHub Desktop.

Select an option

Save pushcx/e59839c44403ea224f37a33c20232cbe to your computer and use it in GitHub Desktop.

2026-10-16 Migrating to a new VPS, gonna be in read-only mode, hopefully only an hour or two. Reload this gist for updates, or drop by the chat room.

migration checklist:

  • pushcx: rsync -vaP --exclude 'public/cache/' --exclude 'bundle/' --exclude 'comment_redirects' l:lobsters/shared/ 2026-10-17_shared
  • pushcx: rsync -vaP root@l:/etc/ 2026-10-17_etc
  • pushcx: rsync -vaP l:.local/ 2026-10-17_local
  • DNS: turn down ttl on lobste.rs and lobsters.dev
  • hatchbox: save ENV vars
  • hatchbox console: create cluster, assign roles
  • NEW console: create reserved IP, check for spam blacklists (Spamhaus, Barracuda, SpamCop), recycle until green
  • NEW console: create instance
  • hatchbox initial provision
  • link to this gist from app/views/layouts/application.html.erb
  • set the site to read-only mode (config/application.rb)
  • lobsters-deploy
  • hatchbox OLD: disable cron jobs, queue
  • OLD: systemctl stop postfix
  • OLD: cd ~/lobsters/current
    for db in primary queue cache rack_attack; do
      bin/sqlite3 storage/$db.sqlite3 "PRAGMA wal_checkpoint(TRUNCATE)"
      bin/sqlite3 storage/$db.sqlite3 "vacuum"
      bin/sqlite3 storage/$db.sqlite3 ".backup '/home/deploy/lobsters/shared/$db.sqlite3'"
    done
    
    rack_attack would never finish, every non-cache hit writes to it, safe to drop, thanks chamlis
  • pushcx: rsync -vaP --exclude 'public/cache/' --exclude 'bundle/' --exclude 'comment_redirects' l:lobsters/shared/ 2026-10-17_shared
  • OLD: root: . /home/deploy/lobsters/shared/etc/restic-env ; restic backup --no-scan /etc
  • OLD: . /home/deploy/lobsters/shared/etc/restic-env ; restic backup --no-scan /home/deploy/lobsters/shared/etc /home/deploy/lobsters/shared/log #{home}/.*_history
  • pushcx: rsync -vaP root@l:/etc/ 2026-10-17_etc
  • pushcx: rsync -vaP 2026-10-17_local/ web2:.local
  • NEW: reverse dns ptr
  • NEW: apt update && apt upgrade && reboot
  • NEW: dpkg-reconfigure --priority=low unattended-upgrades
  • copy hatchbox ssh key up to web2
  • web2: rsync -vaP --exclude comment_redirects --exclude 'log/*bz2' --exclude bundle deploy@lobste.rs:lobsters/shared/ lobsters/shared
  • rm hatchbox ssh key from web2
  • NEW: replace shared/storage/*sqlite3 with .backup versions from parent dir
  • hatchbox: in new cluster, create 'lobsters' app, copy env vars from OLD
  • deploy
  • web2: DISABLE_DATABASE_ENVIRONMENT_CHECK=1 bundle exec rails db:reset:rack_attack
  • hatchbox web2: disable server, cron jobs, queue
  • web2:

ln -s /home/deploy/lobsters/current/hatchbox/root-deploy.service /etc/systemd/system ln -s /home/deploy/lobsters/current/hatchbox/root-deploy.path /etc/systemd/system systemctl daemon-reload systemctl enable --now root-deploy.service systemctl enable --now root-deploy.path ```

  • deploy
  • pushcx: rsync -vaP 2026-10-17_etc/dkimkeys/ root@web2:/etc/dkimkeys
  • NEW: chown -R opendkim:opendkim /etc/dkimkeys
  • pushcx: rsync -vaP 2026-10-17_etc/iptables/ root@web2:/etc/iptables
  • NEW: chown -R root:root /etc/iptables
  • hatchbox OLD: confirm cron jobs, queue still disabled
  • NEW: systemctl status anubis@lobsters opendkim postfix caddy all active.
  • NEW: Puma listens on 127.0.0.1:9001, Anubis on 9000 [hatchbox/lobsters-service.*.port.conf, hatchbox/lobsters.anubis.env].
  • NEW: opendkim-testkey -d lobste.rs -s mail -vvv says key OK (same key as DNS).
  • NEW: iptables -S | wc -l is still the same as OLD (the save did not erase it).
  • NEW: bin/sqlite3 -version is the gem-built version, and which sqlite3 finds only the alias [hatchbox/bash_aliases].
  • NEW: curl -sI --resolve lobste.rs:443:NEW https://lobste.rs/ gives a valid cert and a response.
  • DNS: change A and AAAA to 64.177.113.22 on lobste.rs and lobsters.dev
  • spf, dkim
  • NEW: . /home/deploy/lobsters/shared/etc/restic-env ; restic backup --no-scan /home/deploy/lobsters/shared/etc /home/deploy/lobsters/shared/log /home/deploy/lobsters/shared/database-backups #{home}/.*_history
  • NEW: root: . /home/deploy/lobsters/shared/etc/restic-env ; restic backup --no-scan /etc
  • pushcx: see both in 'restic snapshots'
  • [-] manually try a mail tester (echo test | mail -s test a@example.com)
  • hatchbox NEW: copy cron jobs from web1
  • disable mailing list mode
  • hatchbox enable queue
  • new commit reverting the read-only and notice
  • new commit updating /about
  • deploy
  • sign up with email host, configure postfix to relay
  • migrate mockturtle
  • OLD console: turn off vps

email

We're stuck at the moment and taking a break (19:45 Chicago time).

The new host filters outgoing port 25. I checked this before trying to set up with them and their documentation made it sound like requiring TLS would be fine. Nope.

All our email is three buckets:

  • transactional (email confirmation, password resets, etc), around 30/day
  • opt-out notifications of user-to-user interactions (replies, DMs), around 200/day
  • opt-in mailing list mode

Mailing list mode is big: (20-50 stories per day * 95 users) + (200-500 comments per day * 58 users) = 13k - 28k emails per day. This is enough to scare VPS hosts, who will assume this level of outgoing SMTP is spam regardless of the site. Or that it's running a Wordpress install that will be popped to send spam.

So, options:

  1. Stay with existing host that hates life.
  2. Hope new host believes us and allows us to send email. This feels vanishingly unlikely, but we've asked politely.
  3. Take ~calvin up on his offer to rack a server; Hurricane Electric doesn't filter outbound port 25.
  4. Peter racks a server in the Chicagoland area that also isn't filtered. (Preferred over previous because many state laws now censor websites and reducing jurisdictional complexity reduces those risks.)
  5. Give up on self-hosting and use a mail service. The web is corporate, sucks to want to run your own things.

Problem with that last one is that our volume puts us into a plan that would be >= $500/month. c We could kill reply notifications and mailing list mode, but that just puts us back into the same set of decisions, though at ~30/day.


OK, after taking a break to think, eat, rest:

I don't expect to hear back about the firewall until Monday morning at best. I don't want to revert: it would be more work, and it would save nothing if the new host doesn't work out.

I'm going to disable mailing list mode, drop us down to just transactional + notification emails, then sign up for one of these email services. We'll probably operate in this degraded mode for at least a few couple days. That gives some breathing room to hear back from the new host, evaluate hosting options like calvin's generous offer, research me racking a server here, and hear back from another VPS host we have a couple ins with.

revert checklist

  • revert dns if changed
  • new commit reverting the read-only and notice
  • deploy
  • OLD: systemctl --user start lobsters-solid_queue
  • OLD: systemctl start postfix

followup chores

  • add to cron: find /home/lobsters/shared/storage -type d -empty -delete
  • pushcx: delete 2026-10-17_ backups
  • 1 day later: check automatic restic snapshot appears
  • OLD console: delete vps
  • DNS: turn up TTL
  • DNS: remove OLD from SPF
  • should prod have been vacuuming the sqlites on a schedule?
@aspensmonster

Copy link
Copy Markdown

Hope new host believes us and allows us to send email. This feels vanishingly unlikely, but we've asked politely.

Hopefully they do permit it, as that sounds like the easiest of the options. It's for an opt-in feature and it sounds like less than a 100 users receive most of the mail.

@pushcx

pushcx commented Oct 25, 2026

Copy link
Copy Markdown
Author

We got an unappealable form letter that they considered it prohibited bulk email.

I have us set up with an email service run by a colleague for transactional + notification mail. We're still talking to another potential new host to see if they'd trust us to send our own email. If they say no, I have a couple more doors we can knock on to restore mailing list mode.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment