Skip to content

Instantly share code, notes, and snippets.

@deskoh

deskoh/Filesystem Performance Benchmarking.md

Last active June 2, 2019 00:43
Show Gist options
  • Save deskoh/a0595644db366ec15966039f2c269584 to your computer and use it in GitHub Desktop.
Save deskoh/a0595644db366ec15966039f2c269584 to your computer and use it in GitHub Desktop.
Download ZIP
Linux Filesystem and Disks
Raw
Filesystem Performance Benchmarking.md

Benchmarking with fio

Reference: https://docs.gitlab.com/ee/administration/operations/filesystem_benchmarking.html

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=/path/to/git-data/testfile --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75

This will create a 4GB file in /path/to/git-data/testfile. It performs 4KB reads and writes using a 75%/25% split within the file, with 64 operations running at a time. Be sure to delete the file after the test completes.

Example output:

test: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64
    fio-2.2.10
    Starting 1 process
    test: Laying out IO file(s) (1 file(s) / 1024MB)
    Jobs: 1 (f=1): [m(1)] [100.0% done] [131.4MB/44868KB/0KB /s] [33.7K/11.3K/0 iops] [eta 00m:00s]
    test: (groupid=0, jobs=1): err= 0: pid=10287: Sat Feb  2 17:40:10 2019
      read : io=784996KB, bw=133662KB/s, iops=33415, runt=  5873msec
      write: io=263580KB, bw=44880KB/s, iops=11219, runt=  5873msec
      cpu          : usr=6.56%, sys=23.11%, ctx=266267, majf=0, minf=8
      IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
         submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
         issued    : total=r=196249/w=65895/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
         latency   : target=0, window=0, percentile=100.00%, depth=64

    Run status group 0 (all jobs):
       READ: io=784996KB, aggrb=133661KB/s, minb=133661KB/s, maxb=133661KB/s, mint=5873msec, maxt=5873msec
      WRITE: io=263580KB, aggrb=44879KB/s, minb=44879KB/s, maxb=44879KB/s, mint=5873msec, maxt=5873msec

Notice the iops values in this output. In this example, the SSD performed 33,415 read operations per second and 11,219 write operations per second. A spinning disk might yield 2,000 and 700 read and write operations per second.

Raw
LVM.md

LVM

LVM Terminology and Layout

Physical Disks: A Physical Hard Disk (e.g. /dev/sda, /dev/sdb) and can be divided into one or more Physical Volumes.

Physical Volumes (PVs): Logical parition of Physical Disks. e.g. /dev/sda1, /dev/sdb1. Different PVs can be combined to form a Volume Group.

Volume Groups (VGs): A logical volume that can be formed by combining multiple PVs (and therefore can span multiple PVs and Physical Disks).

Logical Volumes (LVs): A VG can contain one or more LVs. LVs can be resized regardless of order on VGs or whether there is contiguous free space.

Logical Volume   | /dev/server/share | /dev/server/backup  |
                 |-----------------------------------------|
Volume Group     |                  server                 |
                 |-----------------------------------------|
Physical Volume  |  /dev/sdb1  |  /dev/sdc1  |  /dev/sdd1  |

Disks Commands

Display Block Devices

# lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0             2:0    1    4K  0 disk
sda             8:0    0   24G  0 disk
├─sda1          8:1    0    1G  0 part /boot
└─sda2          8:2    0   23G  0 part
  ├─rhel-root 253:0    0 20.6G  0 lvm  /
  └─rhel-swap 253:1    0  2.4G  0 lvm  [SWAP]
sdb             8:16   0    4G  0 disk
sr0            11:0    1 1024M  0 rom

Disk Space Usage

Note: The -h flag tells the df command to print byte counts in "human readable" format, i.e. 1.2G vs. 1220172. The . (period) at the end of the command indicates that we're only interested in disk usage for the device that contains the current directory.

# df -h .
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root   21G  915M   20G   5% /

View Partition Table

# fdisk -l
Disk /dev/sda: 25.8 GB, 25769803776 bytes, 50331648 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: dos
Disk identifier: 0x000d932d

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200    50331647    24116224   8e  Linux LVM

Disk /dev/mapper/rhel-root: 22.1 GB, 22112370688 bytes, 43188224 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/rhel-swap: 2579 MB, 2579496960 bytes, 5038080 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdb: 4294 MB, 4294967296 bytes, 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Create new Physical Volume (from raw disk)

The following commands create a new 2GB Physical Volume on a 4GB Physical Disk /dev/sdb.

# fdisk /dev/sdb

Command (m for help): <-- n
Command action
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
<-- p
Partition number (1-4, default 1): <-- 1
First sector (2048-8388607, default 2048): <-- <ENTER> 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-8388607, default 8388607): <-- +2G
Partition 1 of type Linux and of size 2 GiB is set

Command (m for help): <-- t
Selected partition 1
Hex code (type L to list codes): <-- 8e
Changed type of partition 'Linux' to 'Linux LVM'

Command (m for help): <-- w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Adding Physical Volume to existing Volume Group (Extending VG)

The following command lists existing rhel Volume Group on /dev/sda2 Physical Volume.

Note: Use vgdisplay and pvdisplay to display more information.

# vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  rhel   1   2   0 wz--n- <23.00g    0

# pvs
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sda2  rhel lvm2 a--  <23.00g    0

The rhel Volume Group can be extended using the newly created Physical Volume /dev/sdb1.

vgextend rhel /dev/sdb1

Re-running the commands, we get can see the newly extended existing rhel Volume Group has 2G free space.

# vgs
  VG   #PV #LV #SN Attr   VSize  VFree
  rhel   2   2   0 wz--n- 24.99g <2.00g

# pvs
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sda2  rhel lvm2 a--  <23.00g     0
  /dev/sdb1  rhel lvm2 a--   <2.00g <2.00g

Extending Logical Volume

Now the root Logical Volume can be resized with using the available free space. The Physical Extend (PE) available for can be displayed using the vgdisplay command. (511 / < 2.00GB). The LV path (/dev/rhel/root) can be displayed using lvdisplay command.

# vgdisplay
  --- Volume group ---
  VG Name               rhel
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               24.99 GiB
  PE Size               4.00 MiB
  Total PE              6398
  Alloc PE / Size       5887 / <23.00 GiB
  Free  PE / Size       511 / <2.00 GiB
  VG UUID               hBrQOm-K28x-qMzo-HXGQ-uMzS-6pCc-xW8ALx

# lvextend -l +511 /dev/rhel/root
  Size of logical volume rhel/root changed from 20.59 GiB (5272 extents) to <22.59 GiB (5783 extents).
  Logical volume rhel/root successfully resized.

# xfs_growfs /dev/rhel/root

Note: After Extending, we need to re-size the file-system using xfs_growfs. For EXT filesystem, the command is resize2fs.

Now the lsblk and df -h . commands will show the following.

# df -h .
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root   23G  916M   22G   4% /

# lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda             8:0    0   24G  0 disk
├─sda1          8:1    0    1G  0 part /boot
└─sda2          8:2    0   23G  0 part
  ├─rhel-root 253:0    0 22.6G  0 lvm  /
  └─rhel-swap 253:1    0  2.4G  0 lvm  [SWAP]
sdb             8:16   0    4G  0 disk
└─sdb1          8:17   0    2G  0 part
  └─rhel-root 253:0    0 22.6G  0 lvm  /
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment