Skip to content

Instantly share code, notes, and snippets.

View fmcmac's full-sized avatar
🎯
Focusing

Francis fmcmac

🎯
Focusing
View GitHub Profile
@fmcmac
fmcmac / emr_drill.md
Last active December 1, 2021 20:03
AWS EMR Apache Drill using SSM and not bootstrap

Apache Drill on EMR

Taking over the management of an EMR cluster which is created on demand and terminated after use I set about "tidying it up". Part of the tidy up was to simplify the installion of Apache Drill.
Any improvements greatfully accepted.

Approach

I abandoned the use of a bootstrap script on the cluster to install Drill as I wanted to use the default installation of Zookeeper.
Additionally the use of SSM meant that I could install and tune Drill after the cluster was created without the risk of the bootstrap failing and blocking cluster creation.

Step 1: Install SSM

AWS says that SSM is installed "Amazon Linux base AMIs dated 2017.09 and later" (https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-install-ssm-agent.html).
I used the below to install it and ran this script as a bootstrap script on the EMR creation. "bootstrap_for_ssm.bash"