Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Walkthrough of what I did to increase performance on my Synology NAS box during an expansion, and afterwards.

Performance on Synology RAIDs

(especially while expanding)

Warning: The exact commands may not match for your particular linux OS / Synology(NAS) device. I had to customize the commands after exploring my particular system's setup.

If you're new to linux, or this is a new piece of hardware / a new synology device, jump down to the section called "Inspecting a setup"

Contents

Links

  1. https://lucatnt.com/2013/06/improve-software-raid-speed-on-linux/
  2. https://www.simplicate.info/2014/05/07/speeding-up-synology-volume-expansion/
  3. Gist by @stevenharman
  4. https://www.cyberciti.biz/tips/linux-raid-increase-resync-rebuild-speed.html

Figuring out I had a problem:

I wanted to expand my Synology SHR raid by adding 1 new drive. I followed their instructions; but 2 days after installing the device, I noticed in the GUI that I was at less than 20% finished checking parity. It also looked like my services were running slower than normal / possible.

$ cat /proc/mdstat

The output (snipped to relevant line):

      [<ascii art progress bar>]  reshape = <current percentage>% (<raw blocks/raw total blocks>) finish=<time-in-minutes>min speed=<your speed>/sec

The finish time was on the order of 10 days! -- That's 10 days from the time I install a drive until I get to use the added capacity!

Fixing the problem (Temporarily)

aka if you want to temporarily sacrifice RAM/CPU for increased performance in the raid setup.

I had 8GB of RAM and a CPU that was clearly not doing much, so I decided to figure out how to make use of them to speed things up.

The referenced links above talked about several techniques:

  1. Increasing stripe_cache_size at the cost of RAM
  2. Increasing speed_limit_min -- a minimum goal target for performance -- at the cost of increased dedicated CPU
  3. Increasing read_ahead_kb -- volume read-ahead which could increase read-speed for workloads where you're scanning most of the drive.
  4. Enabling "Bitmap Option" via mdadm -- this improves rebuilds when you had a drive crash, or had to remove & readd a device, but the data is still present. You should not have this on normally, so make sure to disable after the rebuild is complete.
  5. Disabling "NCQ - Native Command Queueing" -- this is a drive feature, but I believe Synology has this already disabled or it doesn't apply to my drives.

For the raid expansion, I interactively checked the values of the first 3 options, and determined that the values were comparatively low.

$ cat /sys/block/md2/md/stripe_cache_size
256
$ cat /proc/sys/dev/raid/speed_limit_min
10000
$ cat /sys/block/md2/queue/read_ahead_kb
256 #-- Note I don't remember exactly what the initial value was for my specific device, and an untimely console clear lost it 🤦‍♂️

I switched the values with the following commands:

$ echo 32768 > /sys/block/md2/md/stripe_cache_size   # This is the max value, and it takes up 32Mib to synchronize read/write operations while the array is degraded
$ echo 50000 > /proc/sys/dev/raid/speed_limit_min    # This is a hint that you want more focus on the sync-expansion task
$ echo 32768 > /sys/block/md2/queue/read_ahead_kb    # This is how far ahead of a read request the drive array will preload

Results

After the above changes:

      [=======>.............]  reshape = 39.5% (2316072704/5855691456) finish=1459.7min speed=40414K/sec

This means that I moved the completion from 8ish days remaining to 23 hours, and in actual practice, it was done in less than 16 hours! #Success!

Cleanup

After the resync was complete, I checked the settings again, and weirdly, the stripe_cache_size reverted to 4096 (not the default I saw of 256)

I reset all the values back to normal before starting to write this article.

Improving Performance Permanently

My NAS is mostly a home media & backup server. I'm not running any databases, so most of my workload is sequential streaming of relatively large files. Based on this, I decided to set the read_ahead_kb to 2048 -- based on my readings, this gives you the max benefit out of read-ahead's ability to limit unnecessary seeking.

This was done by writing a script to be called on startup, automatically:

#!/bin/bash

# Increase the read_ahead_kb to 2048 to maximise sequential large-file read/write performance.

# Put this in /usr/local/etc/rc.d/
# chown this to root
# chmod this to 755
# Must be run as root!

onStart() {
	echo "Starting $0…"
	echo 2048 > /sys/block/md2/queue/read_ahead_kb
	echo "Started $0."
}

onStop() {
	echo "Stopping $0…"
	echo 1024 > /sys/block/md2/queue/read_ahead_kb
	echo "Stopped $0."
}

case $1 in
	start) onStart ;;
	stop) onEnd ;;
	*) echo "Usage: $0 [start|stop]" ;;
esac

Inspecting a setup

  1. Some of these commands work with sudo, but some require being logged in as root

    $ sudo su - # to log in as root@<yourhost>
  2. Look at mdstat to learn about your raids

    $ cat /proc/mdstat
    
    Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] 
    md2 : active raid5 sdc5[2] sda5[0] sdb5[1]
          11711382912 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
    
    md1 : active raid1 sdc2[2] sda2[0] sdb2[1]
          2097088 blocks [5/3] [UUU__]
    
    md0 : active raid1 sdc1[2] sda1[0] sdb1[1]
          2490176 blocks [5/3] [UUU__]
    
    unused devices: <none>

    Interpretation: This means that there are 3 raids, 1 big one, and two ~2GB raids

    This means that our real raid is probably /dev/md2

  3. Use mdadm to give you details:

    $ mdadm --detail /dev/md0
    $ mdadm --detail /dev/md1
    $ mdadm --detail /dev/md2
  4. Look at the results, and learn about your system!

    On my Synology, md0 and md1 were raid1 (mirroring) devices configured across all my drives, and 2gb in size. I didn't see any specific documentation, but I assume this is used by the Synology OS/GUI

    /dev/md2 was a raid device with 3 drives with relevant status lines of:

         Raid Level : raid5
         Array Size : 11711382912 (11168.85 GiB 11992.46 GB)
      Used Dev Size : 5855691456 (5584.42 GiB 5996.23 GB)

    Results: now I know that /dev/md2 is the relevant device!

Contents

sudo su -
cat /sys/block/md2/md/stripe_cache_size
cat /proc/sys/dev/raid/speed_limit_min
cat /sys/block/md2/queue/read_ahead_kb
cat /proc/mdstat
echo 32768 > /sys/block/md2/md/stripe_cache_size
echo 50000 > /proc/sys/dev/raid/speed_limit_min
echo 32768 > /sys/block/md2/queue/read_ahead_kb
echo 4096 > /sys/block/md2/md/stripe_cache_size
echo 10000 > /proc/sys/dev/raid/speed_limit_min
echo 512 > /sys/block/md2/queue/read_ahead_kb
@d8ahazard

This comment has been minimized.

Copy link

@d8ahazard d8ahazard commented Feb 15, 2019

FWIW, I was able to set my speed_limit_min all the way up to 150000 without any issue. Looking at a rebuild time of about 8 hours on a single disk in a 4x5TB array for 13.5 total size.

@d8ahazard

This comment has been minimized.

Copy link

@d8ahazard d8ahazard commented Feb 15, 2019

image

@mqahtani

This comment has been minimized.

Copy link

@mqahtani mqahtani commented May 3, 2019

One more important thing to add: disable SMART testing schedule and will immediately increase read/write speed. Disable it in the schedule and stop any running process, you can do that from GUI

@GiampaoloGabba

This comment has been minimized.

Copy link

@GiampaoloGabba GiampaoloGabba commented Feb 8, 2020

Another little tip: make sure to close the DSM interface. For me this made an huge difference: from 1000+ mins to 270 just closing the DSM.
I dont know why, but i tried 3 times to open rthe DSM in the browser and everytime the rebuild speed started to slow down a lot.

@kevindd992002

This comment has been minimized.

Copy link

@kevindd992002 kevindd992002 commented Aug 13, 2020

Are these changes still applicable even though the DSM GUI already has a "make RAID resync faster" option which changes the "speed_limit_min" to "300000"? Is it advisable to still make the stripe_cache_size and read_ahead_kb values changes and make them persistent? I too am using my NAS for home use and for media files alone (no databases and stuff). I also use it for a couple of docker containers.

@kevindd992002

This comment has been minimized.

Copy link

@kevindd992002 kevindd992002 commented Aug 13, 2020

@fbartho

This comment has been minimized.

Copy link
Owner Author

@fbartho fbartho commented Aug 13, 2020

@kevindd992002 -- I haven't seen that DSM GUI Feature -- do you have a link to it, or a screenshot?

I haven't tested that feature yet, but I would imagine that if it's a GUI Feature that it's probably good enough / perhaps even better than what I wrote here almost 2.5 years ago!

@kevindd992002

This comment has been minimized.

Copy link

@kevindd992002 kevindd992002 commented Aug 13, 2020

@kevindd992002 -- I haven't seen that DSM GUI Feature -- do you have a link to it, or a screenshot?

I haven't tested that feature yet, but I would imagine that if it's a GUI Feature that it's probably good enough / perhaps even better than what I wrote here almost 2.5 years ago!

https://www.synology.com/en-us/knowledgebase/DSM/help/DSM/StorageManager/storage_pool_adjust_resync_speed

Yeah, that's what I thought. But that GUI option only modifies one of the three options you discussed. I asked about the stripe_cache_size and read_ahead_kb from Synology support but they do not support it.

@fbartho

This comment has been minimized.

Copy link
Owner Author

@fbartho fbartho commented Sep 10, 2020

Turns out, I have a drive that's getting increasing numbers of Bad Sector Errors, so I'm going to swap it out -- and I guess I'll need to follow these instructions!

@kevindd992002

This comment has been minimized.

Copy link

@kevindd992002 kevindd992002 commented Sep 11, 2020

Yeah, it practically has the same effect.

@Milananas

This comment has been minimized.

Copy link

@Milananas Milananas commented Jan 14, 2021

We had this exact issue on multiple systems that we have (RS2818RP+) and none of the above increased our speed which was stuck at +- 2000K/ps. After an extensive Google search we ended up here:
https://eprints.bbk.ac.uk/id/eprint/30457/1/2020-01-03-accelerating-synology-raid-6-reshapes.markdown
https://community.synology.com/enu/forum/1/post/130936

The last command:
echo max > /sys/block/md2/md/sync_max
did the trick for us, upping the speed from 2k to 30k. Even though the already specified value was already a quite high number.

@fbartho

This comment has been minimized.

Copy link
Owner Author

@fbartho fbartho commented Jun 14, 2021

Thanks @Milananas! max was already the value in our sync_max, so unfortunately, that didn't help me (just replaced a drive today). I wonder why that differed from what you saw!

@Ryeera

This comment has been minimized.

Copy link

@Ryeera Ryeera commented Jun 22, 2021

We had this exact issue on multiple systems that we have (RS2818RP+) and none of the above increased our speed which was stuck at +- 2000K/ps. After an extensive Google search we ended up here:
https://eprints.bbk.ac.uk/id/eprint/30457/1/2020-01-03-accelerating-synology-raid-6-reshapes.markdown
https://community.synology.com/enu/forum/1/post/130936

The last command:
echo max > /sys/block/md2/md/sync_max
did the trick for us, upping the speed from 2k to 30k. Even though the already specified value was already a quite high number.

It works freaking wonders! I wonder if it only works for a change to RAID 6 though... Thank you so much!

@rjblake

This comment has been minimized.

Copy link

@rjblake rjblake commented Jun 30, 2021

I'm busy expanding my array (3x 16TB Exos drives to 4x 16TB) on DS1821+. Checked the sync_max which was already set to 'max' and it is currently running at between 80000L/sec and 85480K/sec. My setup is SHR-1. Contemplating adding another disk once this is done and migrating to SHR-2 but may just keep drive as a hot spare. I'm using DSM7.0.

@fbartho

This comment has been minimized.

Copy link
Owner Author

@fbartho fbartho commented Jun 30, 2021

@rjblake thanks for the data point!

I haven’t had a chance to upgrade to DSM7.x — how safe is it? Any problems you ran into?

I’m mostly running stuff in Docker, so as long as that keeps working, I think it’ll be fine, but I’m super hesitant to do the upgrade until I find evidence of stability, haha.

I’m really hoping it’ll finally upgrade smb such that I can do a network-based time machine backup, and not have the experience be terribly slow and unstable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment