Skip to content

Instantly share code, notes, and snippets.

@dgadiraju

dgadiraju/cdh-admin-distcp.sh

Created January 23, 2019 23:48
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save dgadiraju/085ee66867a891108e91cd092dc0f90b to your computer and use it in GitHub Desktop.
Save dgadiraju/085ee66867a891108e91cd092dc0f90b to your computer and use it in GitHub Desktop.
Download ZIP
Raw
cdh-admin-distcp.sh
# This command fails on largedeck.txt as the block size of that file is 64 MB
hadoop distcp \
hdfs://nameservice1/user/itversity/cards \
hdfs://nameservice1/user/itversity/cards1
hadoop fs -ls /user/itversity/cards1
hadoop distcp \
-update \
-skipcrccheck \
hdfs://nameservice1/user/itversity/cards \
hdfs://nameservice1/user/itversity/cards1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment