Skip to content

Instantly share code, notes, and snippets.

@robcowie

robcowie/distcp_examples.sh

Created June 20, 2017 19:10
Show Gist options
  • Save robcowie/3c2fa2519e242823c3c14b5ac66257ba to your computer and use it in GitHub Desktop.
Save robcowie/3c2fa2519e242823c3c14b5ac66257ba to your computer and use it in GitHub Desktop.
Download ZIP
distcp Examples
Raw
distcp_examples.sh
# export HADOOP_OPTS=-Xmx28G
export HADOOP_CLIENT_OPTS="-Xmx2048m"
hadoop distcp -p "hdfs://plat/data/level1/clicks/datehour=2016-12-*" "gs://data-events/data/level1/clicks/"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment