Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
detect all ephemeral disks on EC2 then stripe together in a raid-0 vol mounted at /mnt
#!/bin/bash
#
# this script will attempt to detect any ephemeral drives on an EC2 node and create a RAID-0 stripe
# mounted at /mnt. It should be run early on the first boot of the system.
#
# Beware, This script is NOT fully idempotent.
#
METADATA_URL_BASE="http://169.254.169.254/2012-01-12"
yum -y -d0 install mdadm curl
# Configure Raid - take into account xvdb or sdb
root_drive=`df -h | grep -v grep | awk 'NR==2{print $1}'`
if [ "$root_drive" == "/dev/xvda1" ]; then
echo "Detected 'xvd' drive naming scheme (root: $root_drive)"
DRIVE_SCHEME='xvd'
else
echo "Detected 'sd' drive naming scheme (root: $root_drive)"
DRIVE_SCHEME='sd'
fi
# figure out how many ephemerals we have by querying the metadata API, and then:
# - convert the drive name returned from the API to the hosts DRIVE_SCHEME, if necessary
# - verify a matching device is available in /dev/
drives=""
ephemeral_count=0
ephemerals=$(curl --silent $METADATA_URL_BASE/meta-data/block-device-mapping/ | grep ephemeral)
for e in $ephemerals; do
echo "Probing $e .."
device_name=$(curl --silent $METADATA_URL_BASE/meta-data/block-device-mapping/$e)
# might have to convert 'sdb' -> 'xvdb'
device_name=$(echo $device_name | sed "s/sd/$DRIVE_SCHEME/")
device_path="/dev/$device_name"
# test that the device actually exists since you can request more ephemeral drives than are available
# for an instance type and the meta-data API will happily tell you it exists when it really does not.
if [ -b $device_path ]; then
echo "Detected ephemeral disk: $device_path"
drives="$drives $device_path"
ephemeral_count=$((ephemeral_count + 1 ))
else
echo "Ephemeral disk $e, $device_path is not present. skipping"
fi
done
if [ "$ephemeral_count" = 0 ]; then
echo "No ephemeral disk detected. exiting"
exit 0
fi
# ephemeral0 is typically mounted for us already. umount it here
umount /mnt
# overwrite first few blocks in case there is a filesystem, otherwise mdadm will prompt for input
for drive in $drives; do
dd if=/dev/zero of=$drive bs=4096 count=1024
done
partprobe
mdadm --create --verbose /dev/md0 --level=0 -c256 --raid-devices=$ephemeral_count $drives
echo DEVICE $drives | tee /etc/mdadm.conf
mdadm --detail --scan | tee -a /etc/mdadm.conf
blockdev --setra 65536 /dev/md0
mkfs -t ext3 /dev/md0
mount -t ext3 -o noatime /dev/md0 /mnt
# Remove xvdb/sdb from fstab
chmod 777 /etc/fstab
sed -i "/${DRIVE_SCHEME}b/d" /etc/fstab
# Make raid appear on reboot
echo "/dev/md0 /mnt ext3 noatime 0 0" | tee -a /etc/fstab
@felbus

This comment has been minimized.

Copy link

commented Jan 14, 2014

thanks

@TheCorp

This comment has been minimized.

Copy link

commented Jan 15, 2014

Ya this was awesome thank you

@AlexeiBlue

This comment has been minimized.

Copy link

commented Jul 11, 2014

Awesome script. One thing to add:
mdadm --create --verbose /dev/md0 --level=0 -c256 --raid-devices=$ephemeral_count $drives

May given a warning if you only have 1 ephemeral drive, just add --force before --raid-devices.

@hughe

This comment has been minimized.

Copy link

commented Aug 15, 2014

Awesome. Thanks, you've just saved me a day.

@hey-jude

This comment has been minimized.

Copy link

commented Nov 28, 2014

It has critical issue when reboot. I saw this message in EC2 system log. and I cannot connect this instance.
( c3.xlarge, Ubuntu 14.04 )

 * Starting enable remaining boot-time encrypted block devices�[74G[ OK ]
The disk drive for /mnt is not ready yet or not present.
keys:Continue to wait, or Press S to skip mounting or M for manual recovery

I think that needs to add in /etc/fstab the ‘nobootwait’ option.
http://askubuntu.com/questions/120/how-do-i-avoid-the-s-to-skip-message-on-boot
But it can only continue boot process, not fix (reformat md0 device). I have no idea to fix on boot time, but it really needed.

@UnitedMarsupials

This comment has been minimized.

Copy link

commented Jan 28, 2015

I'm curious, why bother RAID-ing the ephemeral devices together to mount /foo instead of simply using them as swap and mounting the same /foo as tmpfs? One would think, the swap-handling code in kernel is much more mature than the RAID0...

@kjwierenga

This comment has been minimized.

Copy link

commented Jun 24, 2015

Thanks hey-jude! I had the same issue with my ephemeral RAID0. Adding the nobootwait option in fstab fixed the issue for me too.

@rickrussell

This comment has been minimized.

Copy link

commented Mar 22, 2016

For some images I'm getting 'udev' error due to the awk command hitting the first line looking for root. It looks like the root drive is the 3rd line down for some images, so I changed the script as follows:
root_drive=df -h | grep -v grep | awk 'NR==4{print $1}'`
which was finally finding the correct root partition and the script is working as intended. I'm using this as a user_data snippet on boot.

@danjger

This comment has been minimized.

Copy link

commented Mar 22, 2016

I think the issue of the drive not being ready at boot is because there is a difference in the state of the ephemeral drives on a reboot versus a stop and then start. You get new unformatted drives when using a stop then restart but you get the same drives on a reboot. I'm trying to figure out how to detect that on boot and run the mkfs if that is the case.

@stwaph

This comment has been minimized.

Copy link

commented Apr 5, 2016

Thanks! Was getting 'resource busy' at create RAID step but your umounting of the cloud-config mounted ephemeral drive solved my problem.

@mconigliaro

This comment has been minimized.

Copy link

commented Apr 19, 2016

If you trust the docs, there's a much easier way to find all the ephemeral volumes:

find /dev -regextype posix-egrep -regex '.*/(sd|xvd)[b-e]'
@BillBrower

This comment has been minimized.

Copy link

commented May 15, 2016

On Ubuntu 14.04

root_drive=df -h | grep -v grep | awk 'NR==2{print $1}'`

was pulling the first drive, not necessarily the root drive, for me until I changed it to

root_drive=df -h | grep -P '^.*/$' | awk 'NF==6{print $1}'`.

@kung-foo

This comment has been minimized.

Copy link

commented Nov 8, 2016

On Ubuntu 16.04 I am using this to get the ephemeral volumes:

root@ip-10-228-67-215:~# lsblk -I 202 -d -n --output NAME,MAJ:MIN | grep -v "202:0" | awk '{print $1}'
xvdf
xvdg
@jimbrowne

This comment has been minimized.

Copy link

commented May 22, 2017

Note that Ubuntu 16.04+ uses systemd and it does not support the nobootwait mount option; use nofail instead.

@jpriolo

This comment has been minimized.

Copy link

commented Dec 8, 2017

Great script. Saved me some time. thanks! - I did add some code for this to run efficiently as a startup script... details below. Cheers! - Joseph P.

PS: The correct startup log on EC2 (As of this date) is: "/var/log/boot.log"

Additions:

  1. created as 'startup' script on redhat / centos
  2. added code to verify script is running on startup
  3. added code so that this script will not execute on reboot if the raid disk is already mounted. (In case of a shutdown it will typically execute as ephemeral disks are wiped out.)
  4. added code to prevent /etc/fstab mount point entry from being written multiple times.

additional code

#######  To use as startup script  ########
### 1. Copy script to directory: /opt/aws/
# mkdir -p /opt/aws/
# cp 'myscript.sh' /opt/aws/
# chmod 755 /opt/aws/myscript.sh
# 
### 2: Add script path to 'rc.local' to execute at startup:
# echo "/opt/aws/myscript.sh" | tee -a /etc/rc.d/rc.local
#
### Intended use: 
# EC2 reboot: raid volume and data will be persistent on "/dev/md0"
# EC2 shutdown: all ephemeral storage is wiped. This script will initialize all instance stores and mount raid disk on boot.
#

# checksum to verify script is executing at reboot
DATE=$(date +'%F %H:%M:%S')
DIR=/tmp
echo "Current date and time: $DATE" > $DIR/ephem_bootscript_lastrun.txt

# check if Raid 0 disk is mounted | IF mounted then exit - if not continue
if grep '/dev/md0' /etc/mtab > /dev/null 2>&1; then
echo "'/dev/md0' is mounted...exiting" && exit
else
echo "'/dev/md0' not mounted"
echo "Continuing..."
fi

# Make raid volume mount on reboot (Prevent writing entry to /etc/fstab multiple times)
MP=/yourmountpoint
MOUNT_PATH="/dev/md0 ${MP} ext3 noatime 0 0"
FILE="/etc/fstab"
grep -qF "$MOUNT_PATH" "$FILE" || echo "$MOUNT_PATH" >> "$FILE"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.