EMR & EC2 Notes

emr_notes.md

EMR

Submit an S3DistCP step using awscli

# AMI 3.x
aws emr add-steps\
  --cluster-id j-xxxxxxxxxxx\
  --steps 'Type=CUSTOM_JAR,Name="s3distcpstep",Jar=/home/hadoop/lib/emr-s3distcp-1.0.jar,Args=["--src=s3://","--dest=s3://"]'

# AMI 4.x
aws emr add-steps\
  --cluster-id j-xxxxxxxxxxx\
  --steps 'Type=CUSTOM_JAR,Name="s3distcp step",Jar=command-runner.jar,Args=["s3-dist-cp","--src=s3://","--dest=s3://"]'

EC2

Get instance metadata

curl http://169.254.169.254/latest/meta-data/

See http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html