Skip to content

Instantly share code, notes, and snippets.

@jorisbontje
Last active December 13, 2015 23:49
Show Gist options
  • Save jorisbontje/4994035 to your computer and use it in GitHub Desktop.
Save jorisbontje/4994035 to your computer and use it in GitHub Desktop.
WebHDFS
from webhdfs.webhdfs import WebHDFS
import os, tempfile
import time
webhdfs = WebHDFS("localhost", 50070, "training")
webhdfs.mkdir("/hello-world/")
# create a temporary file
f = tempfile.NamedTemporaryFile()
f.write(b'Hello world!\n')
f.flush()
print "Upload file: " + f.name
webhdfs.copyFromLocal(f.name,
"hello-world/test.txt")
webhdfs.copyToLocal("hello-world/test.txt",
"/tmp/test1.txt")
for i in webhdfs.listdir("/hello-world/"):
print str(i)
f.close()
curl -O http://python-distribute.org/distribute_setup.py
sudo python distribute_setup.py
curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py
sudo python get-pip.py
sudo pip install webhdfs
cp /usr/lib/python2.6/site-packages/webhdfs/example.py .
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment