Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Walkthrough of what I did to increase performance on my Synology NAS box during an expansion, and afterwards.

Performance on Synology RAIDs

(especially while expanding)

Warning: The exact commands may not match for your particular linux OS / Synology(NAS) device. I had to customize the commands after exploring my particular system's setup.

If you're new to linux, or this is a new piece of hardware / a new synology device, jump down to the section called "Inspecting a setup"

Contents

Links

  1. https://lucatnt.com/2013/06/improve-software-raid-speed-on-linux/
  2. https://www.simplicate.info/2014/05/07/speeding-up-synology-volume-expansion/
  3. Gist by @stevenharman
  4. https://www.cyberciti.biz/tips/linux-raid-increase-resync-rebuild-speed.html

Figuring out I had a problem:

I wanted to expand my Synology SHR raid by adding 1 new drive. I followed their instructions; but 2 days after installing the device, I noticed in the GUI that I was at less than 20% finished checking parity. It also looked like my services were running slower than normal / possible.

$ cat /proc/mdstat

The output (snipped to relevant line):

      [<ascii art progress bar>]  reshape = <current percentage>% (<raw blocks/raw total blocks>) finish=<time-in-minutes>min speed=<your speed>/sec

The finish time was on the order of 10 days! -- That's 10 days from the time I install a drive until I get to use the added capacity!

Fixing the problem (Temporarily)

aka if you want to temporarily sacrifice RAM/CPU for increased performance in the raid setup.

I had 8GB of RAM and a CPU that was clearly not doing much, so I decided to figure out how to make use of them to speed things up.

The referenced links above talked about several techniques:

  1. Increasing stripe_cache_size at the cost of RAM
  2. Increasing speed_limit_min -- a minimum goal target for performance -- at the cost of increased dedicated CPU
  3. Increasing read_ahead_kb -- volume read-ahead which could increase read-speed for workloads where you're scanning most of the drive.
  4. Enabling "Bitmap Option" via mdadm -- this improves rebuilds when you had a drive crash, or had to remove & readd a device, but the data is still present. You should not have this on normally, so make sure to disable after the rebuild is complete.
  5. Disabling "NCQ - Native Command Queueing" -- this is a drive feature, but I believe Synology has this already disabled or it doesn't apply to my drives.

For the raid expansion, I interactively checked the values of the first 3 options, and determined that the values were comparatively low.

$ cat /sys/block/md2/md/stripe_cache_size
256
$ cat /proc/sys/dev/raid/speed_limit_min
10000
$ cat /sys/block/md2/queue/read_ahead_kb
256 #-- Note I don't remember exactly what the initial value was for my specific device, and an untimely console clear lost it 🤦‍♂️

I switched the values with the following commands:

$ echo 32768 > /sys/block/md2/md/stripe_cache_size   # This is the max value, and it takes up 32Mib to synchronize read/write operations while the array is degraded
$ echo 50000 > /proc/sys/dev/raid/speed_limit_min    # This is a hint that you want more focus on the sync-expansion task
$ echo 32768 > /sys/block/md2/queue/read_ahead_kb    # This is how far ahead of a read request the drive array will preload

Results

After the above changes:

      [=======>.............]  reshape = 39.5% (2316072704/5855691456) finish=1459.7min speed=40414K/sec

This means that I moved the completion from 8ish days remaining to 23 hours, and in actual practice, it was done in less than 16 hours! #Success!

Cleanup

After the resync was complete, I checked the settings again, and weirdly, the stripe_cache_size reverted to 4096 (not the default I saw of 256)

I reset all the values back to normal before starting to write this article.

Improving Performance Permanently

My NAS is mostly a home media & backup server. I'm not running any databases, so most of my workload is sequential streaming of relatively large files. Based on this, I decided to set the read_ahead_kb to 2048 -- based on my readings, this gives you the max benefit out of read-ahead's ability to limit unnecessary seeking.

This was done by writing a script to be called on startup, automatically:

#!/bin/ash

# Increase the read_ahead_kb to 2048 to maximise sequential large-file read/write performance.

# Put this in /usr/local/etc/rc.d/
# chown this to root
# chmod this to 755
# Must be run as root!

onStart() {
	echo "Starting $0…"
	echo 2048 > /sys/block/md2/queue/read_ahead_kb
	echo "Started $0."
}

onStop() {
	echo "Stopping $0…"
	echo 1024 > /sys/block/md2/queue/read_ahead_kb
	echo "Stopped $0."
}

case $1 in
	start) onStart ;;
	stop) onEnd ;;
	*) echo "Usage: $0 [start|stop]" ;;
esac

Inspecting a setup

  1. Some of these commands work with sudo, but some require being logged in as root

    $ sudo su - # to log in as root@<yourhost>
  2. Look at mdstat to learn about your raids

    $ cat /proc/mdstat
    
    Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] 
    md2 : active raid5 sdc5[2] sda5[0] sdb5[1]
          11711382912 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
    
    md1 : active raid1 sdc2[2] sda2[0] sdb2[1]
          2097088 blocks [5/3] [UUU__]
    
    md0 : active raid1 sdc1[2] sda1[0] sdb1[1]
          2490176 blocks [5/3] [UUU__]
    
    unused devices: <none>

    Interpretation: This means that there are 3 raids, 1 big one, and two ~2GB raids

    This means that our real raid is probably /dev/md2

  3. Use mdadm to give you details:

    $ mdadm --detail /dev/md0
    $ mdadm --detail /dev/md1
    $ mdadm --detail /dev/md2
  4. Look at the results, and learn about your system!

    On my Synology, md0 and md1 were raid1 (mirroring) devices configured across all my drives, and 2gb in size. I didn't see any specific documentation, but I assume this is used by the Synology OS/GUI

    /dev/md2 was a raid device with 3 drives with relevant status lines of:

         Raid Level : raid5
         Array Size : 11711382912 (11168.85 GiB 11992.46 GB)
      Used Dev Size : 5855691456 (5584.42 GiB 5996.23 GB)

    Results: now I know that /dev/md2 is the relevant device!

Contents

sudo su -
cat /sys/block/md2/md/stripe_cache_size
cat /proc/sys/dev/raid/speed_limit_min
cat /sys/block/md2/queue/read_ahead_kb
cat /proc/mdstat
echo 32768 > /sys/block/md2/md/stripe_cache_size
echo 50000 > /proc/sys/dev/raid/speed_limit_min
echo 32768 > /sys/block/md2/queue/read_ahead_kb
echo 4096 > /sys/block/md2/md/stripe_cache_size
echo 10000 > /proc/sys/dev/raid/speed_limit_min
echo 256 > /sys/block/md2/queue/read_ahead_kb
@d8ahazard

This comment has been minimized.

Copy link

d8ahazard commented Feb 15, 2019

FWIW, I was able to set my speed_limit_min all the way up to 150000 without any issue. Looking at a rebuild time of about 8 hours on a single disk in a 4x5TB array for 13.5 total size.

@d8ahazard

This comment has been minimized.

Copy link

d8ahazard commented Feb 15, 2019

image

@mqahtani

This comment has been minimized.

Copy link

mqahtani commented May 3, 2019

One more important thing to add: disable SMART testing schedule and will immediately increase read/write speed. Disable it in the schedule and stop any running process, you can do that from GUI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.