yuanzhaoYZ/hadoop copy from s3

Created April 11, 2017 15:43

Star () You must be signed in to star a gist
Fork () You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/yuanzhaoYZ/b906fc56da60530a76a7a9e82ad9fd78.js"></script>
Save yuanzhaoYZ/b906fc56da60530a76a7a9e82ad9fd78 to your computer and use it in GitHub Desktop.

Download ZIP

Raw

hadoop copy from s3

hadoop distcp -Dmapreduce.map.memory.mb=4096 -Dfs.s3a.awsAccessKeyId=XXX -Dfs.s3a.awsSecretAccessKey=XXXX  -m 250 hdfs:///data/* s3a://api-v3-data-sources/output/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment