Skip to content

Instantly share code, notes, and snippets.

@joemiller
Last active October 23, 2023 21:53
Show Gist options
  • Save joemiller/6049831 to your computer and use it in GitHub Desktop.
Save joemiller/6049831 to your computer and use it in GitHub Desktop.
detect all ephemeral disks on EC2 then stripe together in a raid-0 vol mounted at /mnt
#!/bin/bash
#
# this script will attempt to detect any ephemeral drives on an EC2 node and create a RAID-0 stripe
# mounted at /mnt. It should be run early on the first boot of the system.
#
# Beware, This script is NOT fully idempotent.
#
METADATA_URL_BASE="http://169.254.169.254/2012-01-12"
yum -y -d0 install mdadm curl
# Configure Raid - take into account xvdb or sdb
root_drive=`df -h | grep -v grep | awk 'NR==2{print $1}'`
if [ "$root_drive" == "/dev/xvda1" ]; then
echo "Detected 'xvd' drive naming scheme (root: $root_drive)"
DRIVE_SCHEME='xvd'
else
echo "Detected 'sd' drive naming scheme (root: $root_drive)"
DRIVE_SCHEME='sd'
fi
# figure out how many ephemerals we have by querying the metadata API, and then:
# - convert the drive name returned from the API to the hosts DRIVE_SCHEME, if necessary
# - verify a matching device is available in /dev/
drives=""
ephemeral_count=0
ephemerals=$(curl --silent $METADATA_URL_BASE/meta-data/block-device-mapping/ | grep ephemeral)
for e in $ephemerals; do
echo "Probing $e .."
device_name=$(curl --silent $METADATA_URL_BASE/meta-data/block-device-mapping/$e)
# might have to convert 'sdb' -> 'xvdb'
device_name=$(echo $device_name | sed "s/sd/$DRIVE_SCHEME/")
device_path="/dev/$device_name"
# test that the device actually exists since you can request more ephemeral drives than are available
# for an instance type and the meta-data API will happily tell you it exists when it really does not.
if [ -b $device_path ]; then
echo "Detected ephemeral disk: $device_path"
drives="$drives $device_path"
ephemeral_count=$((ephemeral_count + 1 ))
else
echo "Ephemeral disk $e, $device_path is not present. skipping"
fi
done
if [ "$ephemeral_count" = 0 ]; then
echo "No ephemeral disk detected. exiting"
exit 0
fi
# ephemeral0 is typically mounted for us already. umount it here
umount /mnt
# overwrite first few blocks in case there is a filesystem, otherwise mdadm will prompt for input
for drive in $drives; do
dd if=/dev/zero of=$drive bs=4096 count=1024
done
partprobe
mdadm --create --verbose /dev/md0 --level=0 -c256 --raid-devices=$ephemeral_count $drives
echo DEVICE $drives | tee /etc/mdadm.conf
mdadm --detail --scan | tee -a /etc/mdadm.conf
blockdev --setra 65536 /dev/md0
mkfs -t ext3 /dev/md0
mount -t ext3 -o noatime /dev/md0 /mnt
# Remove xvdb/sdb from fstab
chmod 777 /etc/fstab
sed -i "/${DRIVE_SCHEME}b/d" /etc/fstab
# Make raid appear on reboot
echo "/dev/md0 /mnt ext3 noatime 0 0" | tee -a /etc/fstab
@kjwierenga
Copy link

Thanks hey-jude! I had the same issue with my ephemeral RAID0. Adding the nobootwait option in fstab fixed the issue for me too.

@rickrussell
Copy link

For some images I'm getting 'udev' error due to the awk command hitting the first line looking for root. It looks like the root drive is the 3rd line down for some images, so I changed the script as follows:
root_drive=df -h | grep -v grep | awk 'NR==4{print $1}'`
which was finally finding the correct root partition and the script is working as intended. I'm using this as a user_data snippet on boot.

@danjger
Copy link

danjger commented Mar 22, 2016

I think the issue of the drive not being ready at boot is because there is a difference in the state of the ephemeral drives on a reboot versus a stop and then start. You get new unformatted drives when using a stop then restart but you get the same drives on a reboot. I'm trying to figure out how to detect that on boot and run the mkfs if that is the case.

@stwaph
Copy link

stwaph commented Apr 5, 2016

Thanks! Was getting 'resource busy' at create RAID step but your umounting of the cloud-config mounted ephemeral drive solved my problem.

@mconigliaro
Copy link

mconigliaro commented Apr 19, 2016

If you trust the docs, there's a much easier way to find all the ephemeral volumes:

find /dev -regextype posix-egrep -regex '.*/(sd|xvd)[b-e]'

@BillBrower
Copy link

BillBrower commented May 15, 2016

On Ubuntu 14.04

root_drive=df -h | grep -v grep | awk 'NR==2{print $1}'`

was pulling the first drive, not necessarily the root drive, for me until I changed it to

root_drive=df -h | grep -P '^.*/$' | awk 'NF==6{print $1}'`.

@kung-foo
Copy link

kung-foo commented Nov 8, 2016

On Ubuntu 16.04 I am using this to get the ephemeral volumes:

root@ip-10-228-67-215:~# lsblk -I 202 -d -n --output NAME,MAJ:MIN | grep -v "202:0" | awk '{print $1}'
xvdf
xvdg

@jimbrowne
Copy link

Note that Ubuntu 16.04+ uses systemd and it does not support the nobootwait mount option; use nofail instead.

@jpriolo
Copy link

jpriolo commented Dec 8, 2017

Great script. Saved me some time. thanks! - I did add some code for this to run efficiently as a startup script... details below. Cheers! - Joseph P.

PS: The correct startup log on EC2 (As of this date) is: "/var/log/boot.log"

Additions:

  1. created as 'startup' script on redhat / centos
  2. added code to verify script is running on startup
  3. added code so that this script will not execute on reboot if the raid disk is already mounted. (In case of a shutdown it will typically execute as ephemeral disks are wiped out.)
  4. added code to prevent /etc/fstab mount point entry from being written multiple times.

additional code

#######  To use as startup script  ########
### 1. Copy script to directory: /opt/aws/
# mkdir -p /opt/aws/
# cp 'myscript.sh' /opt/aws/
# chmod 755 /opt/aws/myscript.sh
# 
### 2: Add script path to 'rc.local' to execute at startup:
# echo "/opt/aws/myscript.sh" | tee -a /etc/rc.d/rc.local
#
### Intended use: 
# EC2 reboot: raid volume and data will be persistent on "/dev/md0"
# EC2 shutdown: all ephemeral storage is wiped. This script will initialize all instance stores and mount raid disk on boot.
#

# checksum to verify script is executing at reboot
DATE=$(date +'%F %H:%M:%S')
DIR=/tmp
echo "Current date and time: $DATE" > $DIR/ephem_bootscript_lastrun.txt

# check if Raid 0 disk is mounted | IF mounted then exit - if not continue
if grep '/dev/md0' /etc/mtab > /dev/null 2>&1; then
echo "'/dev/md0' is mounted...exiting" && exit
else
echo "'/dev/md0' not mounted"
echo "Continuing..."
fi

# Make raid volume mount on reboot (Prevent writing entry to /etc/fstab multiple times)
MP=/yourmountpoint
MOUNT_PATH="/dev/md0 ${MP} ext3 noatime 0 0"
FILE="/etc/fstab"
grep -qF "$MOUNT_PATH" "$FILE" || echo "$MOUNT_PATH" >> "$FILE"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment