EC11 Shrink the DRBD LV to make room for snapshots

drbd-lv-shrink.md

Understanding the layers

1. ext4 (/var/opt/opscode/drbd/data)
2. drbd device (pc0)
3. LVM Logical Volume (/dev/opscode/data)
4. LVM Volume Group (opscode)

Resources

• http://pubmem.wordpress.com/2010/09/16/how-to-resize-lvm-logical-volumes-with-ext4-as-filesystem/
• http://www.drbd.org/users-guide/s-resizing.html

The procedure

1. Ensure you are on the primary bootstrap node and it is running all services

2. Examine our current Volume Group and Logical Volume

[root@backend1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree 
  opscode   1   1   0 wz--n- 79.99g 16.00g
[root@backend1 ~]# lvs
  LV   VG      Attr       LSize  Pool Origin Data%  Move Log Cpy%Sync Convert
  drbd opscode -wi-ao---- 63.99g                                             
[root@backend1 ~]# df -kh
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvde        12G  4.8G  6.6G  42% /
tmpfs           7.3G     0  7.3G   0% /dev/shm
/dev/drbd0       63G  260M   60G   1% /var/opt/opscode/drbd/data

3. Stop services on the secondary backend first, then on the bootstrap

[root@backend1 ~]# ssh root@backend2 "private-chef-ctl stop"
...
ok: down: redis_lb: 1272s, normally up

[root@backend1 ~]# private-chef-ctl stop
...
ok: down: redis_lb: 5s, normally up

4. Force drbd to be primary

[root@backend1 ~]# drbdadm primary pc0


[root@backend1 ~]# service drbd status
drbd driver loaded OK; device status:
version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by phil@Build64R6, 2013-10-14 15:33:06
m:res  cs         ro                 ds                 p  mounted  fstype
0:pc0  Connected  Primary/Secondary  UpToDate/UpToDate  C

5. fsck the filesystem first:

[root@backend1 ~]# e2fsck -f /dev/drbd0
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/drbd0: 5703/4194304 files (3.0% non-contiguous), 329652/16773623 blocks

6. Our resize target is 50G, so resize the filesystem to N-1 GB first:

[root@backend1 ~]# resize2fs -p /dev/drbd0 49G
resize2fs 1.41.12 (17-May-2010)
Resizing the filesystem on /dev/drbd0 to 13107200 (4k) blocks.
Begin pass 3 (max = 512)
Scanning inode table          XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
The filesystem on /dev/drbd0 is now 13107200 blocks long.

6. Reduce the Logical Volume to the resize target (50G)

[root@backend1 ~]# lvreduce -L 50G /dev/mapper/opscode-drbd 
  WARNING: Reducing active and open logical volume to 50.00 GiB
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce drbd? [y/n]: y
  Reducing logical volume drbd to 50.00 GiB
  Logical volume drbd successfully resized

7. Also, perform the same lvreduce on the secondary node:

[root@backend2 ~]# lvreduce -L 50G /dev/mapper/opscode-drbd 
  WARNING: Reducing active and open logical volume to 50.00 GiB
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce drbd? [y/n]: y
  Reducing logical volume drbd to 50.00 GiB
  Logical volume drbd successfully resized

8. Finally, reduce the drbd device size:

[root@backend1 ~]# drbdsetup resize /dev/drbd0 --size=50G

9. Check dmesg to ensure drbd is happy

[root@backend1 ~]# dmesg | tail -10
...
block drbd0: drbd_bm_resize called with capacity == 104854328
block drbd0: resync bitmap: bits=13106791 words=204794 pages=400
block drbd0: size = 50 GB (52427164 KB)
block drbd0: Writing the whole bitmap, size changed and md moved
block drbd0: Considerable difference in lower level device sizes: 134188984s vs. 104854328s
block drbd0: Requested disk size is too big (52428800 > 52427164)
block drbd0: Requested disk size is too big (52428800 > 52427164)
block drbd0: Requested disk size is too big (52428800 > 52427164)

10. Whoops, looks like we need to make the drbd volume smaller. Use the size from dmesg above, but remember to append a K

[root@backend1 ~]# drbdsetup resize /dev/drbd0 --size=52427164K

[root@backend1 ~]# dmesg | tail -10
...
block drbd0: drbd_bm_resize called with capacity == 102760448
block drbd0: resync bitmap: bits=12845056 words=200704 pages=392
block drbd0: size = 49 GB (51380224 KB)
block drbd0: Writing the whole bitmap, size changed
block drbd0: bitmap WRITE of 0 pages took 0 jiffies
block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.

11. Now, resize the ext filesystem again to match the new drbd resource size

[root@backend1 ~]# resize2fs -p /dev/drbd0 
resize2fs 1.41.12 (17-May-2010)
Resizing the filesystem on /dev/drbd0 to 12845056 (4k) blocks.
Begin pass 3 (max = 400)
Scanning inode table          XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
The filesystem on /dev/drbd0 is now 12845056 blocks long.

12. Check the filesystem again to ensure it is consistent:

[root@backend1 ~]# e2fsck -f /dev/drbd0
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/drbd0: 5703/3211264 files (3.0% non-contiguous), 267972/12845056 blocks

13. Mount the filesystem and start services:

[root@backend1 ~]# mount /dev/drbd0 /var/opt/opscode/drbd/data

[root@backend1 ~]# private-chef-ctl start keepalived
ok: run: keepalived: (pid 2849) 0s

14. Fail over to the secondary node and note that the filesystem is the correct size:

[root@backend2 ~]# df -kh
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvde        12G  4.8G  6.5G  43% /
tmpfs           7.3G     0  7.3G   0% /dev/shm
/dev/drbd0       49G  260M   46G   1% /var/opt/opscode/drbd/data

thanks!