Skip to content

Instantly share code, notes, and snippets.

@X140Yu
Created July 13, 2018 08:27
Show Gist options
  • Save X140Yu/84d6fef6189654d0e263d2314ff3c6de to your computer and use it in GitHub Desktop.
Save X140Yu/84d6fef6189654d0e263d2314ff3c6de to your computer and use it in GitHub Desktop.
use hadoop hdfs in ruby
class HDFSClient
attr_accessor :prefix_url
def initialize(prefix_url)
prefix_url.append("/") unless prefix_url.end_with?("/")
@prefix_url = prefix_url
end
def upload(file_name, file_path)
system "curl -i -L -s -X PUT -T #{file_path} \
#{@prefix_url}#{file_name}\\?op\\=CREATE"
raise "😱 upload failed" unless $?.exitstatus == 0
end
def download(file_name)
cmd = "curl \
#{@prefix_url}#{file_name}\\?op\\=OPEN \
--output #{file_name}"
system cmd
raise "😱 download failed" unless $?.exitstatus == 0
end
end
# usage
# hdfs = HDFSClient.new("http://hadoop02.dev.rack.zhihu.com:50070/webhdfs/v1/user/ios/osee2binary/")
# hdfs.upload("upload.py", File.expand_path("upload.py", __dir__))
# system "rm -rf upload.py"
# hdfs.download("upload.py")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment