Skip to content

Instantly share code, notes, and snippets.

@gabrieljcs
Last active March 23, 2024 13:33
Show Gist options
  • Star 24 You must be signed in to star a gist
  • Fork 8 You must be signed in to fork a gist
  • Save gabrieljcs/805c183753046dcc6131 to your computer and use it in GitHub Desktop.
Save gabrieljcs/805c183753046dcc6131 to your computer and use it in GitHub Desktop.
Instructions to create an LVM cache for root in Fedora

LVM cache in Fedora

From the man-pages: "The cache logical volume type uses a small and fast LV to improve the performance of a large and slow LV. It does this by storing the frequently used blocks on the faster LV. LVM refers to the small fast LV as a cache pool LV. The large slow LV is called the origin LV. Due to requirements from dm-cache (the kernel driver), LVM further splits the cache pool LV into two devices - the cache data LV and cache metadata LV. The cache data LV is where copies of data blocks are kept from the origin LV to increase speed. The cache metadata LV holds the accounting information that specifies where data blocks are stored (e.g. on the origin LV or on the cache data LV). Users should be familiar with these LVs if they wish to create the best and most robust cached logical volumes. All of these associated LVs must be in the same VG."

Assuming LVM is already setup in HDD (e.g. from anaconda) and SSD is untouched.

Create a physical volume for the SSD

# pvcreate /dev/sdX

e.g. pvcreate /dev/sdb, sdb being the SSD.

Extend your existing volume group to include the SSD

# vgextend VOLUME_GROUP /dev/sdX

anaconda names the volume group as fedora_HOSTNAME.

Create a cache metadata LV

# lvcreate -n meta -L YMB VOLUME_GROUP /dev/sdX

Where Y is 1000 times smaller than the cache data LV, with a minimum of 8MB. e.g. if you have a 32 GB SSD, Y = 32MB.

Create a cache data LV

# lvcreate -n cache -l Z VOLUME_GROUP /dev/sdX

Where Z is the size of your cache. Use 100%FREE to use the remaining free space in the physical volume.

Create the cache pool LV

# lvconvert --type cache-pool --cachemode writeback --poolmetadata VOLUME_GROUP/meta VOLUME_GROUP/cache

This combines the cache data and metadata into a cache pool that uses writeback mode. Ommitting defaults to writethrough, which stores data in the cache and on the origin LV.

Create the cache

# lvconvert --type cache --cachepool VOLUME_GROUP/cache VOLUME_GROUP/root

The pool is then converted into a cache for the root partition.

Rebuild initramfs

# dracut -v -f

This will rebuild the initramfs to include the cache LV.

Sources and further info:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/lvm_cache_volume_creation.html

http://man7.org/linux/man-pages/man7/lvmcache.7.html

Thanks to @ntn888 for the cachemode suggestion and @Yeshey for the 100%FREE option.

@ntn888
Copy link

ntn888 commented Mar 14, 2020

Wouldn't it be better adding the following command to use writeback mode to get better performance?

lvchange --cachemode writeback fedora_localhost-live/root

@ntn888
Copy link

ntn888 commented Mar 14, 2020

or simply edit the 'Create the cache pool LV' command to :
lvconvert --type cache-pool --cachemode writeback --poolmetadata VOLUME_GROUP/meta VOLUME_GROUP/cache

@gabrieljcs
Copy link
Author

Wouldn't it be better adding the following command to use writeback mode to get better performance?

lvchange --cachemode writeback fedora_localhost-live/root

Yes, it would. Thanks for the addition!

@ntn888
Copy link

ntn888 commented Mar 15, 2020

you're welcome

@SmallJoker
Copy link

SmallJoker commented Jun 14, 2020

update-initramfs worked for me on Ubuntu 20.04.
There's also a handy command to check the status of the cache:
lvs -a -o +devices

As a side note: I'd recommend to not have /boot inside a lvm2 volume for boot safety.
If the problem repeats on a separate /boot partition, run those command from a Live OS:

mount /dev/mapper/VOLUME_GROUP-root /mnt
mount /dev/sdXX /mnt/boot

chroot /mnt
grub-install /dev/sdX
update-initramfs -u -k A.B.C-DD-generic

@vap0rtranz
Copy link

vap0rtranz commented Nov 16, 2020

Reboot failed on CentOS 8.2 finding rootfs using dracut. Perhaps I fat fingered so full process below.

Dracut ignores several modules, including dmraid & mdadm ... but I'm not using software RAID so that should be OK, and folks typically say these "cannot be installed" messages from dracut can be ignored, no?

[justin@host3 ~]$ cat /etc/centos-release
CentOS Linux release 8.2.2004 (Core) 

[justin@host3 ~]$ sudo lvs
  LV      VG Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home    cl -wi-ao----   1.00g                                                    
  root    cl -wi-ao----   6.00g                                                    
  swap    cl -wi-ao----   1.00g                                                    
  tmp     cl -wi-ao----   1.00g                                                    
  var     cl -wi-ao---- <48.33g                                                    
  var_log cl -wi-ao----  10.00g   
                                                 
[justin@host3 ~]$ sudo vgs
  VG #PV #LV #SN Attr   VSize   VFree
  cl   1   6   0 wz--n- <67.33g    0 

[justin@host3 ~]$ sudo lsblk
NAME           MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda              8:0    0 68.3G  0 disk 
├─sda1           8:1    0    1G  0 part /boot
└─sda2           8:2    0 67.3G  0 part 
  ├─cl-root    253:0    0    6G  0 lvm  /
  ├─cl-swap    253:1    0    1G  0 lvm  [SWAP]
  ├─cl-var     253:2    0 48.3G  0 lvm  /var
  ├─cl-tmp     253:3    0    1G  0 lvm  /tmp
  ├─cl-var_log 253:4    0   10G  0 lvm  /var/log
  └─cl-home    253:5    0    1G  0 lvm  /home
sdb              8:16   0 55.9G  0 disk 
sdc              8:32   0 59.6G  0 disk 

[justin@host3 ~]$ sudo pvcreate /dev/sdc
  Physical volume "/dev/sdc" successfully created.
          
[justin@host3 ~]$ sudo vgextend cl /dev/sdc
  Volume group "cl" successfully extended
                                                  
[justin@host3 ~]$ sudo vgs
  VG #PV #LV #SN Attr   VSize    VFree  
  cl   2   6   0 wz--n- <126.92g <59.59g

[justin@host3 ~]$ sudo lvcreate -n meta -L 59MB cl /dev/sdc
  Rounding up size to full physical extent 60.00 MiB
  Logical volume "meta" created.

[justin@host3 ~]$  sudo lvcreate -n cache -l 99%PVS cl /dev/sdc
  Logical volume "cache" created.

[justin@host3 ~]$ sudo lvconvert --type cache-pool --cachemode writeback --poolmetadata cl/meta cl/cache
  WARNING: Converting cl/cache and cl/meta to cache pool's data and metadata volumes with metadata wiping.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Do you really want to convert cl/cache and cl/meta? [y/n]: y
  Converted cl/cache and cl/meta to cache pool.

[justin@host3 ~]$ sudo lvconvert --type cache --cachepool cl/cache cl/root
Do you want wipe existing metadata of cache pool cl/cache? [y/n]: y
  Logical volume cl/root is now cached.

[justin@host3 ~]$ sudo lsblk
[sudo] password for justin: 
NAME                   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                      8:0    0 68.3G  0 disk 
├─sda1                   8:1    0    1G  0 part /boot
└─sda2                   8:2    0 67.3G  0 part 
  ├─cl-swap            253:1    0    1G  0 lvm  [SWAP]
  ├─cl-var             253:2    0 48.3G  0 lvm  /var
  ├─cl-tmp             253:3    0    1G  0 lvm  /tmp
  ├─cl-var_log         253:4    0   10G  0 lvm  /var/log
  ├─cl-home            253:5    0    1G  0 lvm  /home
  └─cl-root_corig      253:8    0    6G  0 lvm  
    └─cl-root          253:0    0    6G  0 lvm  /
sdb                      8:16   0 55.9G  0 disk 
sdc                      8:32   0 59.6G  0 disk 
├─cl-cache_cpool_cdata 253:6    0   59G  0 lvm  
│ └─cl-root            253:0    0    6G  0 lvm  /
└─cl-cache_cpool_cmeta 253:7    0   60M  0 lvm  
  └─cl-root            253:0    0    6G  0 lvm  /

dracut: Executing: /usr/bin/dracut -v -f
dracut: dracut module 'modsign' will not be installed, because command 'keyctl' could not be found!
dracut: dracut module 'busybox' will not be installed, because command 'busybox' could not be found!
dracut: dracut module 'btrfs' will not be installed, because command 'btrfs' could not be found!
dracut: dracut module 'dmraid' will not be installed, because command 'dmraid' could not be found!
dracut: dracut module 'mdraid' will not be installed, because command 'mdadm' could not be found!
dracut: dracut module 'stratis' will not be installed, because command 'stratisd-init' could not be found!
dracut: dracut module 'cifs' will not be installed, because command 'mount.cifs' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsi-iname' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsiadm' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsid' could not be found!
dracut: 95nfs: Could not find any command of 'rpcbind portmap'!
dracut: dracut module 'modsign' will not be installed, because command 'keyctl' could not be found!
dracut: dracut module 'busybox' will not be installed, because command 'busybox' could not be found!
dracut: dracut module 'btrfs' will not be installed, because command 'btrfs' could not be found!
dracut: dracut module 'dmraid' will not be installed, because command 'dmraid' could not be found!
dracut: dracut module 'mdraid' will not be installed, because command 'mdadm' could not be found!
dracut: dracut module 'stratis' will not be installed, because command 'stratisd-init' could not be found!
dracut: dracut module 'cifs' will not be installed, because command 'mount.cifs' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsi-iname' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsiadm' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsid' could not be found!
dracut: 95nfs: Could not find any command of 'rpcbind portmap'!
dracut: *** Including module: bash ***
dracut: *** Including module: systemd ***
dracut: *** Including module: systemd-initrd ***
dracut: *** Including module: rngd ***
dracut: *** Including module: i18n ***
dracut: *** Including module: network-legacy ***
dracut: *** Including module: network ***
dracut: *** Including module: ifcfg ***
dracut: *** Including module: drm ***
dracut: *** Including module: plymouth ***
dracut: *** Including module: prefixdevname ***
dracut: *** Including module: dm ***
dracut: Skipping udev rule: 64-device-mapper.rules
dracut: Skipping udev rule: 60-persistent-storage-dm.rules
dracut: Skipping udev rule: 55-dm.rules
dracut: *** Including module: kernel-modules ***
dracut: *** Including module: kernel-modules-extra ***
dracut: *** Including module: kernel-network-modules ***
dracut: *** Including module: lvm ***
dracut: Skipping udev rule: 64-device-mapper.rules
dracut: Skipping udev rule: 56-lvm.rules
dracut: Skipping udev rule: 60-persistent-storage-lvm.rules
dracut: *** Including module: resume ***
dracut: *** Including module: rootfs-block ***
dracut: *** Including module: terminfo ***
dracut: *** Including module: udev-rules ***
dracut: Skipping udev rule: 91-permissions.rules
dracut: Skipping udev rule: 80-drivers-modprobe.rules
dracut: *** Including module: biosdevname ***
dracut: *** Including module: dracut-systemd ***
dracut: *** Including module: usrmount ***
dracut: *** Including module: base ***
dracut: *** Including module: fs-lib ***
dracut: *** Including module: microcode_ctl-fw_dir_override ***
dracut:   microcode_ctl module: mangling fw_dir
dracut:     microcode_ctl: reset fw_dir to "/lib/firmware/updates /lib/firmware"
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel"...
intel: model '', path ' intel-ucode/*', kvers ''
intel: blacklist ''
dracut:       microcode_ctl: intel: caveats check for kernel version "4.18.0-193.el8.x86_64" passed, adding "/usr/share/microcode_ctl/ucode_with_caveats/intel" to fw_dir variable
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-2d-07"...
intel-06-2d-07: model 'GenuineIntel 06-2d-07', path ' intel-ucode/06-2d-07', kvers ''
intel-06-2d-07: blacklist ''
Current CPU model 'GenuineIntel 06-2c-02' doesn't match configuration CPU model 'GenuineIntel 06-2d-07', skipping
dracut:     microcode_ctl: configuration "intel-06-2d-07" is ignored
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-4e-03"...
intel-06-4e-03: model 'GenuineIntel 06-4e-03', path ' intel-ucode/06-4e-03', kvers ''
intel-06-4e-03: blacklist ''
Current CPU model 'GenuineIntel 06-2c-02' doesn't match configuration CPU model 'GenuineIntel 06-4e-03', skipping
dracut:     microcode_ctl: configuration "intel-06-4e-03" is ignored
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-4f-01"...
intel-06-4f-01: model 'GenuineIntel 06-4f-01', path ' intel-ucode/06-4f-01', kvers ' 4.17.0 3.10.0-894 3.10.0-862.6.1 3.10.0-693.35.1 3.10.0-514.52.1 3.10.0-327.70.1 2.6.32-754.1.1 2.6.32-573.58.1 2.6.'
intel-06-4f-01: blacklist ''
Current CPU model 'GenuineIntel 06-2c-02' doesn't match configuration CPU model 'GenuineIntel 06-4f-01', skipping
dracut:     microcode_ctl: configuration "intel-06-4f-01" is ignored
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-55-04"...
intel-06-55-04: model 'GenuineIntel 06-55-04', path ' intel-ucode/06-55-04', kvers ''
intel-06-55-04: blacklist ''
Current CPU model 'GenuineIntel 06-2c-02' doesn't match configuration CPU model 'GenuineIntel 06-55-04', skipping
dracut:     microcode_ctl: configuration "intel-06-55-04" is ignored
dracut:     microcode_ctl: processing data directory  "/usr/share/microcode_ctl/ucode_with_caveats/intel-06-5e-03"...
intel-06-5e-03: model 'GenuineIntel 06-5e-03', path ' intel-ucode/06-5e-03', kvers ''
intel-06-5e-03: blacklist ''
Current CPU model 'GenuineIntel 06-2c-02' doesn't match configuration CPU model 'GenuineIntel 06-5e-03', skipping
dracut:     microcode_ctl: configuration "intel-06-5e-03" is ignored
dracut:     microcode_ctl: final fw_dir: "/usr/share/microcode_ctl/ucode_with_caveats/intel /lib/firmware/updates /lib/firmware"
dracut: *** Including module: shutdown ***
dracut: *** Including modules done ***
dracut: *** Installing kernel module dependencies ***
dracut: *** Installing kernel module dependencies done ***
dracut: *** Resolving executable dependencies ***
dracut: *** Resolving executable dependencies done***
dracut: *** Hardlinking files ***
dracut: *** Hardlinking files done ***
dracut: Could not find 'strip'. Not stripping the initramfs.
dracut: *** Generating early-microcode cpio image ***
dracut: *** Constructing GenuineIntel.bin ****
dracut: *** Store current command line parameters ***
dracut: *** Creating image file '/boot/initramfs-4.18.0-193.el8.x86_64.img' ***
dracut: *** Creating initramfs image file '/boot/initramfs-4.18.0-193.el8.x86_64.img' done ***

@gabrieljcs
Copy link
Author

@vap0rtranz sorry about that, it's been a while since I've done this. My guess would be to run dracut -v -f --regenerate-all and see if it works.

@Yeshey
Copy link

Yeshey commented Nov 11, 2022

When Creating the cache data LV, you can use 100%FREE instead of 99%PVS, if you have the /dev/sdX indicating the PV, it will use the remaining free space in the PV, and not fill the whole VG.
Thanks for the great guide

@gabrieljcs
Copy link
Author

@Yeshey much better, thanks!

@vpieper
Copy link

vpieper commented Jan 9, 2023

I get the following error:

Volume group "vg" has insufficient free space (286 extents): 28307 required.

This is what I have VFree which is all SSD

VG     #PV #LV #SN Attr   VSize   VFree
vg       2   1   0 wz--n-   3.75t 111.79g

Any thoughts?

Edit: I switched the lv names in this command:

# lvconvert --type cache-pool --cachemode writeback --poolmetadata VOLUME_GROUP/meta VOLUME_GROUP/cache

So meta first, then cache, NOT the other way round, that fixed it for me.

@gabrieljcs
Copy link
Author

gabrieljcs commented Jan 19, 2023

@vpieper ah, got it, thanks for the follow up!

@tuxmaster5000
Copy link

Wouldn't it be better adding the following command to use writeback mode to get better performance?

lvchange --cachemode writeback fedora_localhost-live/root

This will be dangerous, when the cache device will not be an RAID1. Its is the same risk when using caching on ZFS. You will risk your data, when the caching device will fails. So only an read cache are fail save without RAID1 on the cache device.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment