Skip to content

Instantly share code, notes, and snippets.

@fbartho
Last active February 29, 2024 01:15
Show Gist options
  • Star 79 You must be signed in to star a gist
  • Fork 12 You must be signed in to fork a gist
  • Save fbartho/2cb998dc1f10d13c124bf736286fd757 to your computer and use it in GitHub Desktop.
Save fbartho/2cb998dc1f10d13c124bf736286fd757 to your computer and use it in GitHub Desktop.
Walkthrough of what I did to increase performance on my Synology NAS box during an expansion, and afterwards.

Performance on Synology RAIDs

(especially while expanding)

Warning: The exact commands may not match for your particular linux OS / Synology(NAS) device. I had to customize the commands after exploring my particular system's setup.

If you're new to linux, or this is a new piece of hardware / a new synology device, jump down to the section called "Inspecting a setup"

Contents

Links

  1. https://lucatnt.com/2013/06/improve-software-raid-speed-on-linux/
  2. https://www.simplicate.info/2014/05/07/speeding-up-synology-volume-expansion/
  3. Gist by @stevenharman
  4. https://www.cyberciti.biz/tips/linux-raid-increase-resync-rebuild-speed.html

Figuring out I had a problem:

I wanted to expand my Synology SHR raid by adding 1 new drive. I followed their instructions; but 2 days after installing the device, I noticed in the GUI that I was at less than 20% finished checking parity. It also looked like my services were running slower than normal / possible.

$ cat /proc/mdstat

The output (snipped to relevant line):

      [<ascii art progress bar>]  reshape = <current percentage>% (<raw blocks/raw total blocks>) finish=<time-in-minutes>min speed=<your speed>/sec

The finish time was on the order of 10 days! -- That's 10 days from the time I install a drive until I get to use the added capacity!

Fixing the problem (Temporarily)

aka if you want to temporarily sacrifice RAM/CPU for increased performance in the raid setup.

I had 8GB of RAM and a CPU that was clearly not doing much, so I decided to figure out how to make use of them to speed things up.

The referenced links above talked about several techniques:

  1. Increasing stripe_cache_size at the cost of RAM
  2. Increasing speed_limit_min -- a minimum goal target for performance -- at the cost of increased dedicated CPU
  3. Increasing read_ahead_kb -- volume read-ahead which could increase read-speed for workloads where you're scanning most of the drive.
  4. Enabling "Bitmap Option" via mdadm -- this improves rebuilds when you had a drive crash, or had to remove & readd a device, but the data is still present. You should not have this on normally, so make sure to disable after the rebuild is complete.
  5. Disabling "NCQ - Native Command Queueing" -- this is a drive feature, but I believe Synology has this already disabled or it doesn't apply to my drives.

For the raid expansion, I interactively checked the values of the first 3 options, and determined that the values were comparatively low.

$ cat /sys/block/md2/md/stripe_cache_size
256
$ cat /proc/sys/dev/raid/speed_limit_min
10000
$ cat /sys/block/md2/queue/read_ahead_kb
256 #-- Note I don't remember exactly what the initial value was for my specific device, and an untimely console clear lost it 🤦‍♂️

I switched the values with the following commands:

$ echo 32768 > /sys/block/md2/md/stripe_cache_size   # This is the max value, and it takes up 32Mib to synchronize read/write operations while the array is degraded
$ echo 50000 > /proc/sys/dev/raid/speed_limit_min    # This is a hint that you want more focus on the sync-expansion task
$ echo 32768 > /sys/block/md2/queue/read_ahead_kb    # This is how far ahead of a read request the drive array will preload

Results

After the above changes:

      [=======>.............]  reshape = 39.5% (2316072704/5855691456) finish=1459.7min speed=40414K/sec

This means that I moved the completion from 8ish days remaining to 23 hours, and in actual practice, it was done in less than 16 hours! #Success!

Cleanup

After the resync was complete, I checked the settings again, and weirdly, the stripe_cache_size reverted to 4096 (not the default I saw of 256)

I reset all the values back to normal before starting to write this article.

Improving Performance Permanently

My NAS is mostly a home media & backup server. I'm not running any databases, so most of my workload is sequential streaming of relatively large files. Based on this, I decided to set the read_ahead_kb to 2048 -- based on my readings, this gives you the max benefit out of read-ahead's ability to limit unnecessary seeking.

This was done by writing a script to be called on startup, automatically:

#!/bin/bash

# Increase the read_ahead_kb to 2048 to maximise sequential large-file read/write performance.

# Put this in /usr/local/etc/rc.d/
# chown this to root
# chmod this to 755
# Must be run as root!

onStart() {
	echo "Starting $0…"
	echo 2048 > /sys/block/md2/queue/read_ahead_kb
	echo "Started $0."
}

onStop() {
	echo "Stopping $0…"
	echo 1024 > /sys/block/md2/queue/read_ahead_kb
	echo "Stopped $0."
}

case $1 in
	start) onStart ;;
	stop) onEnd ;;
	*) echo "Usage: $0 [start|stop]" ;;
esac

Inspecting a setup

  1. Some of these commands work with sudo, but some require being logged in as root

    $ sudo su - # to log in as root@<yourhost>
  2. Look at mdstat to learn about your raids

    $ cat /proc/mdstat
    
    Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] 
    md2 : active raid5 sdc5[2] sda5[0] sdb5[1]
          11711382912 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
    
    md1 : active raid1 sdc2[2] sda2[0] sdb2[1]
          2097088 blocks [5/3] [UUU__]
    
    md0 : active raid1 sdc1[2] sda1[0] sdb1[1]
          2490176 blocks [5/3] [UUU__]
    
    unused devices: <none>

    Interpretation: This means that there are 3 raids, 1 big one, and two ~2GB raids

    This means that our real raid is probably /dev/md2

  3. Use mdadm to give you details:

    $ mdadm --detail /dev/md0
    $ mdadm --detail /dev/md1
    $ mdadm --detail /dev/md2
  4. Look at the results, and learn about your system!

    On my Synology, md0 and md1 were raid1 (mirroring) devices configured across all my drives, and 2gb in size. I didn't see any specific documentation, but I assume this is used by the Synology OS/GUI

    /dev/md2 was a raid device with 3 drives with relevant status lines of:

         Raid Level : raid5
         Array Size : 11711382912 (11168.85 GiB 11992.46 GB)
      Used Dev Size : 5855691456 (5584.42 GiB 5996.23 GB)

    Results: now I know that /dev/md2 is the relevant device!

Contents

sudo su -
cat /sys/block/md2/md/stripe_cache_size
cat /proc/sys/dev/raid/speed_limit_min
cat /sys/block/md2/queue/read_ahead_kb
cat /proc/mdstat
echo 32768 > /sys/block/md2/md/stripe_cache_size
echo 50000 > /proc/sys/dev/raid/speed_limit_min
echo 32768 > /sys/block/md2/queue/read_ahead_kb
echo 4096 > /sys/block/md2/md/stripe_cache_size
echo 10000 > /proc/sys/dev/raid/speed_limit_min
echo 512 > /sys/block/md2/queue/read_ahead_kb
@d8ahazard
Copy link

FWIW, I was able to set my speed_limit_min all the way up to 150000 without any issue. Looking at a rebuild time of about 8 hours on a single disk in a 4x5TB array for 13.5 total size.

@d8ahazard
Copy link

image

@mqahtani
Copy link

mqahtani commented May 3, 2019

One more important thing to add: disable SMART testing schedule and will immediately increase read/write speed. Disable it in the schedule and stop any running process, you can do that from GUI

@GiampaoloGabba
Copy link

GiampaoloGabba commented Feb 8, 2020

Another little tip: make sure to close the DSM interface. For me this made an huge difference: from 1000+ mins to 270 just closing the DSM.
I dont know why, but i tried 3 times to open rthe DSM in the browser and everytime the rebuild speed started to slow down a lot.

@kevindd992002
Copy link

Are these changes still applicable even though the DSM GUI already has a "make RAID resync faster" option which changes the "speed_limit_min" to "300000"? Is it advisable to still make the stripe_cache_size and read_ahead_kb values changes and make them persistent? I too am using my NAS for home use and for media files alone (no databases and stuff). I also use it for a couple of docker containers.

@kevindd992002
Copy link

@fbartho
Copy link
Author

fbartho commented Aug 13, 2020

@kevindd992002 -- I haven't seen that DSM GUI Feature -- do you have a link to it, or a screenshot?

I haven't tested that feature yet, but I would imagine that if it's a GUI Feature that it's probably good enough / perhaps even better than what I wrote here almost 2.5 years ago!

@kevindd992002
Copy link

@kevindd992002 -- I haven't seen that DSM GUI Feature -- do you have a link to it, or a screenshot?

I haven't tested that feature yet, but I would imagine that if it's a GUI Feature that it's probably good enough / perhaps even better than what I wrote here almost 2.5 years ago!

https://www.synology.com/en-us/knowledgebase/DSM/help/DSM/StorageManager/storage_pool_adjust_resync_speed

Yeah, that's what I thought. But that GUI option only modifies one of the three options you discussed. I asked about the stripe_cache_size and read_ahead_kb from Synology support but they do not support it.

@fbartho
Copy link
Author

fbartho commented Sep 10, 2020

Turns out, I have a drive that's getting increasing numbers of Bad Sector Errors, so I'm going to swap it out -- and I guess I'll need to follow these instructions!

@kevindd992002
Copy link

kevindd992002 commented Sep 11, 2020

Yeah, it practically has the same effect.

@Milananas
Copy link

We had this exact issue on multiple systems that we have (RS2818RP+) and none of the above increased our speed which was stuck at +- 2000K/ps. After an extensive Google search we ended up here:
https://eprints.bbk.ac.uk/id/eprint/30457/1/2020-01-03-accelerating-synology-raid-6-reshapes.markdown
https://community.synology.com/enu/forum/1/post/130936

The last command:
echo max > /sys/block/md2/md/sync_max
did the trick for us, upping the speed from 2k to 30k. Even though the already specified value was already a quite high number.

@fbartho
Copy link
Author

fbartho commented Jun 14, 2021

Thanks @Milananas! max was already the value in our sync_max, so unfortunately, that didn't help me (just replaced a drive today). I wonder why that differed from what you saw!

@Ryeera
Copy link

Ryeera commented Jun 22, 2021

We had this exact issue on multiple systems that we have (RS2818RP+) and none of the above increased our speed which was stuck at +- 2000K/ps. After an extensive Google search we ended up here:
https://eprints.bbk.ac.uk/id/eprint/30457/1/2020-01-03-accelerating-synology-raid-6-reshapes.markdown
https://community.synology.com/enu/forum/1/post/130936

The last command:
echo max > /sys/block/md2/md/sync_max
did the trick for us, upping the speed from 2k to 30k. Even though the already specified value was already a quite high number.

It works freaking wonders! I wonder if it only works for a change to RAID 6 though... Thank you so much!

@rjblake
Copy link

rjblake commented Jun 30, 2021

I'm busy expanding my array (3x 16TB Exos drives to 4x 16TB) on DS1821+. Checked the sync_max which was already set to 'max' and it is currently running at between 80000L/sec and 85480K/sec. My setup is SHR-1. Contemplating adding another disk once this is done and migrating to SHR-2 but may just keep drive as a hot spare. I'm using DSM7.0.

@fbartho
Copy link
Author

fbartho commented Jun 30, 2021

@rjblake thanks for the data point!

I haven’t had a chance to upgrade to DSM7.x — how safe is it? Any problems you ran into?

I’m mostly running stuff in Docker, so as long as that keeps working, I think it’ll be fine, but I’m super hesitant to do the upgrade until I find evidence of stability, haha.

I’m really hoping it’ll finally upgrade smb such that I can do a network-based time machine backup, and not have the experience be terribly slow and unstable.

@dotWee
Copy link

dotWee commented Jan 9, 2022

We had this exact issue on multiple systems that we have (RS2818RP+) and none of the above increased our speed which was stuck at +- 2000K/ps. After an extensive Google search we ended up here:
https://eprints.bbk.ac.uk/id/eprint/30457/1/2020-01-03-accelerating-synology-raid-6-reshapes.markdown
https://community.synology.com/enu/forum/1/post/130936

The last command:
echo max > /sys/block/md2/md/sync_max
did the trick for us, upping the speed from 2k to 30k. Even though the already specified value was already a quite high number.

Had to reshape a SHR system with WD Red's HDDs. While idle'ing it ran at finish=7127.8min speed=8753K/sec. After setting sync_max it went straight up finish=1043.5min speed=59017K/sec. Thats a 6x improvement!

It continued over night and finished the next day without any issues, thanks @Milananas!

@mikehardy
Copy link

mikehardy commented Aug 20, 2022

@fbartho imagine my needing to increase the speed of array expansion on my synology and seeing you with the helpful gist :-), small world. Fantastic! Thanks for writing this up, and to the commenters for fantastic additions

Just a note - performance is always a tradeoff, in my case if you bump these numbers up too much, and it is a more expensive operation (like a raid5 reshape to grow an array, not just a resync), you can make your diskstation non-responsive. In my experience on a DS412+ it can handle 55000 or so on a resync but only 20000 on a reshape. Maybe that's useful? Still faster than 10000. Cheers

@winedog
Copy link

winedog commented Jan 5, 2023

@fbartho in your original post you talk about:

Enabling "Bitmap Option" via mdadm -- this improves rebuilds when you had a drive crash, or had to remove & readd a device, but the data is still present. You should not have this on normally, so make sure to disable after the rebuild is complete.

However, I don't see any commands about how you might enable this option?

@fbartho
Copy link
Author

fbartho commented Jan 6, 2023

@winedog I couldn’t personally see any real improvements using the bitmap option, so I don’t have that as part of my final scripts. One of the articles I link to discusses it further.

Do report back if you try it out and see interesting results!

@GCustom
Copy link

GCustom commented Feb 7, 2023

I'm getting Permission denied trying to use sync_max set to max now, this used to work like magic on my system

@GCustom
Copy link

GCustom commented Feb 7, 2023

cat /sys/block/md2/md/sync_max shows max so I guess it doesn't matter it wouldn't let me set it

@MyChickenNinja
Copy link

We had this exact issue on multiple systems that we have (RS2818RP+) and none of the above increased our speed which was stuck at +- 2000K/ps. After an extensive Google search we ended up here: https://eprints.bbk.ac.uk/id/eprint/30457/1/2020-01-03-accelerating-synology-raid-6-reshapes.markdown https://community.synology.com/enu/forum/1/post/130936

The last command: echo max > /sys/block/md2/md/sync_max did the trick for us, upping the speed from 2k to 30k. Even though the already specified value was already a quite high number.

Setting max was absolutely the right way to go for me. Was running an expansion and reshape from 6x 16tb SHR-1 to 8x 16tb SHR-2 (DS1821+). It has been running for 16 days now and only at 40% reshaping. Changed sync_max to max and now looks like it should finish reshaping in little more than a day.

image

Thanks!

@AnLuoRidge
Copy link

AnLuoRidge commented Aug 31, 2023

We had this exact issue on multiple systems that we have (RS2818RP+) and none of the above increased our speed which was stuck at +- 2000K/ps. After an extensive Google search we ended up here: https://eprints.bbk.ac.uk/id/eprint/30457/1/2020-01-03-accelerating-synology-raid-6-reshapes.markdown https://community.synology.com/enu/forum/1/post/130936

The last command: echo max > /sys/block/md2/md/sync_max did the trick for us, upping the speed from 2k to 30k. Even though the already specified value was already a quite high number.

I can confirm again that echo max > /sys/block/md2/md/sync_max is the key. The original sync_max on my end (DS920+) is 3535306752. By changing it to max, my speed increased by ~4.5x.

I also found an explanation.

The value of max for sync_max effectively disables the limit. When a resync is active, the value can only ever be increased, never decreased. The value of 0 is the minimum for sync_min. (doc)

@nextlevel2023
Copy link

We had this exact issue on multiple systems that we have (RS2818RP+) and none of the above increased our speed which was stuck at +- 2000K/ps. After an extensive Google search we ended up here:
https://eprints.bbk.ac.uk/id/eprint/30457/1/2020-01-03-accelerating-synology-raid-6-reshapes.markdown
https://community.synology.com/enu/forum/1/post/130936
The last command:
echo max > /sys/block/md2/md/sync_max
did the trick for us, upping the speed from 2k to 30k. Even though the already specified value was already a quite high number.

It works freaking wonders! I wonder if it only works for a change to RAID 6 though... Thank you so much!

Unbelievable cut down 4 weeks to 2 days on RAID5->RAID6 reshape over 10 x 16TB disks - incredible. I didn't know max existed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment