Skip to content

Instantly share code, notes, and snippets.

@mdespriee

mdespriee/stream_to_hdfs.sh

Created March 16, 2017 11:29
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mdespriee/4f020b4f8106f8aebe835a80118ff146 to your computer and use it in GitHub Desktop.
Save mdespriee/4f020b4f8106f8aebe835a80118ff146 to your computer and use it in GitHub Desktop.
Download ZIP
stream data in/out of hdfs through a edge node
Raw
stream_to_hdfs.sh
ssh edge_node "hdfs dfs -cat /some/path/part-*" | cat > file
cat file | ssh edge_node "hdfs dfs -put - /target/path"
# think of using a named pipe (mkfifo) to sream directly for application output
rm -f stream
mkfifo stream
some_app > stream &
cat stream | ssh edge_node "hdfs dfs -put - /target/path"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment