-
-
Save Nerian/989563 to your computer and use it in GitHub Desktop.
Merger
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
require 'json' | |
module Resque | |
module InMemoryMerger | |
def self.merge_partial_images_from_tasks(image_hash) | |
puts 'Merging image' | |
top_partial = image_hash['0'] | |
image_hash.delete('0') | |
image_hash.each { |k,v| v.slice!(0,18) } | |
final_image = [top_partial,(image_hash.keys.sort{|a,b| a.to_i <=> b.to_i}).collect{ |i| image_hash[i] }.join].join | |
puts 'Image merged' | |
final_image | |
end | |
end | |
module Merger | |
def self.merge_partial_images_from_tasks(hash_OrderImage) | |
#puts 'Merging image' | |
partial_images_names = save_partial_images_to_tmp_folder(hash_OrderImage) | |
#puts partial_images_names.inspect | |
partial_images_names.each_cons 2 do |origin, destiny| | |
#puts "\ttail -c +19 #{origin} >> #{destiny}" | |
`tail -c +19 #{origin} >> #{destiny}` | |
end | |
#puts 'Image merged' | |
final_image = File.read(partial_images_names.last) | |
end | |
private | |
def self.save_partial_images_to_tmp_folder(partial_images) | |
files = [] | |
tmp_folder = '/tmp/dpovray/merger'+ rand(10000).to_s + '/' | |
system("mkdir -p #{tmp_folder}") | |
keys = partial_images.keys.sort{|a,b| a.to_i <=> b.to_i}.reverse | |
puts 'Keys: '+keys.inspect | |
keys.each do |key| | |
image = partial_images[key.to_s] | |
files << tmp_folder+'image'+key.to_s+'.tga' | |
File.open(tmp_folder+'image'+key.to_s+'.tga', "w") do |f| | |
f.write(image) | |
end | |
end | |
files | |
end | |
end | |
end | |
partial_image = JSON.parse(File.read(File.dirname(__FILE__)+"/code")) | |
image = Resque::Merger.merge_partial_images_from_tasks(partial_image) | |
File.open("/tmp/image-tail-merger.tga", "w") do |f| | |
f.write(image) | |
end | |
partial_image = JSON.parse(File.read(File.dirname(__FILE__)+"/code")) | |
image = Resque::InMemoryMerger.merge_partial_images_from_tasks(partial_image) | |
File.open("/tmp/image-"+Time.at(Time.now).usec.to_s+'.tga', "w") do |f| | |
f.write(image) | |
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Download the hash at: | |
http://dl.dropbox.com/u/834494/hash |
@diedthreetimes: I discover how to make it right! juujuuuuuu slice!(0,18) and the array of keys must be sorted and reversed :) Woa this feelings is goood :) The refactor is awesome it makes the merge in an instant. It saves around 50 seconds for an image of 2000x2000. Thanks a lot for your help!!!
@diedthreetimes: Since my code is based on your solution and you helped me a lot I will accept your answer :) thanks!
@Nerian I'm glad it worked :) Thats odd that slice!(0,18) did the trick. According to the docs http://ruby-doc.org/core/classes/String.html#M001213 the second argument should be length and not an index. Maybe slice! opperates slightly different.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
@diedthreetimes: I think I didn't explained well. I will try to be very very detailed:
I am using The Povray Rendering Engine to render an image using a cluster of machines. Each machine is going to render just a part of the total image. So I divide the total image in tasks. The hash with the tasks has a key that represents the order of the task from top ('0') to bottom('49').
The partial images are all in the TGA format. The TGA format has a magic number of 19 bytes. Each partial image is a true TGA image and so it has the 19 bytes of magic number.
Normally I would do this to get the final image:
...
The image5 – except the first 19 bytes – is concatenated to image4
The image4 – except the first 19 bytes – is concatenated to image3
The image3 – except the first 19 bytes – is concatenated to image2
The image2 – except the first 19 bytes – is concatenated to image1
Notice that image0 do not need to have the first 19 bytes removed.
But this is really slow when you have got a lot of files. So I want a solution that runs all in memory.
Check the gist updated code. It is all the code needed and I added more explanation at stackoverflow
http://stackoverflow.com/questions/6115518/how-can-i-achieve-a-unix-tail-operation-without-using-files-in-ruby