Last active
July 14, 2016 12:20
-
-
Save sideshowcoder/1cab4589e0ac0abbf1536d66ae7e3b54 to your computer and use it in GitHub Desktop.
Transfer a huge file over a flaky SCP connection
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# On the server side split the core file into 50M chunks naming them core.0000000000 etc. | |
# $ split -a 10 -b 50M -d core core. | |
# reassamble on the other side via | |
# $ cat core.* > core | |
chunk_count = 538 | |
root_name = "core." # name on server is 'core.00001 etc' | |
host = "myhost.com" | |
path = "/tmp" | |
target = "put-this-here" | |
(0..chunk_count).map do |n| | |
fname = sprintf "#{root_name}%010d", n # in my infinite wisdom I used 10 element long numeric chunks | |
begin | |
puts "downloading #{fname}" | |
`scp #{host}:#{path}/#{fname} ./#{target}/` | |
fail "download failed" unless $?.success? # SCP why are you dying on me?! | |
rescue | |
puts "retrying #{fname}" | |
retry | |
end | |
end |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment