Skip to content

Instantly share code, notes, and snippets.

@blockpane
Last active August 7, 2023 18:47
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save blockpane/0cdbbe41273e66c14cba31910a9587d4 to your computer and use it in GitHub Desktop.
Save blockpane/0cdbbe41273e66c14cba31910a9587d4 to your computer and use it in GitHub Desktop.
Striped block devices on Digitalocean

LVM Setup on DO for striped block devices:

This example is specifically for my Osmosis seed node, that I run in the cloud, all my other nodes are on hardware, and I use a very different setup on those.

These instructions should work on just about any cloud provider, by striping block devices it's possible to get much higher IOPS and throughput. On Omosis in particular this is important. It doesn't cost any more to use striped volumes, so for example 3 40GB volumes cost the same as a single 120GB, but give (almost) 3x the performance.

After provisioning new block devices (do not format automatically) in the DO console and attaching them, ssh into the droplet and as root follow the directions below. I try to shoot for < 50% usage, so right now with about 120GiB needed, three 90GB volumes will do nicely.

Note that extending the volume group in the future will require adding exactly the same number and size of volumes. Alternatively you can just use the same procedure here to move the files into a new volume group and destroy the old. Either method works.

This example assumes that osmosisd is started from systemd, the user is osmo:osmo, and the home directory is /var/lib/osmosis.

# ... find new devices:
lsblk

#... match to their ID:
ls -l /dev/disk/by-id/

# ... create physical volumes:
pvcreate /dev/disk/by-id/scsi-0DO_Volume_volume-sgp1-01
pvcreate /dev/disk/by-id/scsi-0DO_Volume_volume-sgp1-02
pvcreate /dev/disk/by-id/scsi-0DO_Volume_volume-sgp1-03

vgcreate data_vg /dev/disk/by-id/scsi-0DO_Volume_volume-sgp1-03 /dev/disk/by-id/scsi-0DO_Volume_volume-sgp1-02 /dev/disk/by-id/scsi-0DO_Volume_volume-sgp1-01

# create the logical volume ...
lvcreate --type striped --stripes 3 --stripesize 128K -l 100%FREE -n data_lv data_vg

# since we are memory constrained, ext4 is the best choice
mkfs.ext4 -b 4096 /dev/data_vg/data_lv

# stop the daemon or moving files will corrupt the DB:
systemctl stop osmosisd

# temp mount to copy data
mount /dev/data_vg/data_lv /mnt

# copy the data
cd /var/lib/osmosis/
mv .osmosisd /mnt

# ... watching the graphs during the copy, each block device was writing at about 68MB/s = 204MB/s, which is very good.

# now mount it over the directory
umount /mnt
echo '/dev/data_vg/data_lv /var/lib/osmosis ext4 defaults,noatime 0 2' >> /etc/fstab
mount -a

# reset the permissions, they will be 0755 root:root
chown osmo:osmo /var/lib/osmosis
chmod 0700 /var/lib/osmosis

# all done!
systemctl start osmosisd
journalctl -fu osmosisd

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment