Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
An illustration how we can use ruby thread for accelerating our normal RAKE tasks. (intended for posting in a local Ruby developers group https://www.facebook.com/groups/rubydevs/)
namespace :somestuffs do
desc 'Importing some big stuff that requies lotta network stuff'
task :import
file, tasks = ENV['FILE'], Queue.new
# Parse out CSV file and retrieve each row and queue them up
# for further processing.
#
# Keeping it a thread allows us to let this process continue while other
# threads are not in RUNNING state.
Thread.new do
CSV.open(file) { |row| tasks << row }
end
# Use something that matters in your envrionment,
# if you are using some network call which takes quite long time,
# then you can use 10/20 threads / CPU core
4.times do
Thread.new do
while (row = tasks.pop)
# Do network transfer related blocking task.
end
end
end
# Control thread which keeps current process blocked until queue becomes zero
# In our case we are expecting tasks.size could get zero whenever no more data to process.
Thread.new do
sleep 1 until tasks.size.zero?
end.join
# Calling "join" allows current process to wait until the JOINED thread is finished.
# Thus we can ensure our parent process doesn't exit until all tasks are done.
end
end
@rubyrider

This comment has been minimized.

Copy link

rubyrider commented Dec 16, 2013

nice!

@mftaher

This comment has been minimized.

Copy link

mftaher commented Dec 19, 2013

Just wondering, won't it lead to memory leaks since your threads will live forever and every time you'll call import! it will be spawning 4 new threads?

@we4tech

This comment has been minimized.

Copy link
Owner Author

we4tech commented Dec 19, 2013

hi @mftaher bhai, actually i took out this snapshot from one of my rake tasks, so in that case it wasn't an issue since rake task spins in and do it job then exit.

Otherwise the middle block is leaky if you are trying to use it in normal long running process. I'm adding additional changes so it does reflect my intent.

thanks for mentioning :)

@we4tech

This comment has been minimized.

Copy link
Owner Author

we4tech commented Dec 19, 2013

hi @mftaher bhai check out it now :)

@mftaher

This comment has been minimized.

Copy link

mftaher commented Dec 19, 2013

@we4tech now it looks elegant and make sense ... more ruby like

@we4tech

This comment has been minimized.

Copy link
Owner Author

we4tech commented Dec 19, 2013

thanks @mftaher bhai :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.