An illustration how we can use ruby thread for accelerating our normal RAKE tasks. (intended for posting in a local Ruby developers group https://www.facebook.com/groups/rubydevs/)

rthread_importer.rb

namespace :somestuffs do
  desc 'Importing some big stuff that requies lotta network stuff'
  task :import
    file, tasks = ENV['FILE'], Queue.new

    # Parse out CSV file and retrieve each row and queue them up 
    # for further processing. 
    #
    # Keeping it a thread allows us to let this process continue while other 
    # threads are not in RUNNING state.
    Thread.new do
      CSV.open(file) { |row| tasks << row }
    end
    
    # Use something that matters in your envrionment, 
    # if you are using some network call which takes quite long time, 
    # then you can use 10/20 threads / CPU core
    4.times do
      Thread.new do
        while (row = tasks.pop)
          # Do network transfer related blocking task.
        end
      end
    end
    
    # Control thread which keeps current process blocked until queue becomes zero
    # In our case we are expecting tasks.size could get zero whenever no more data to process.
    Thread.new do
      sleep 1 until tasks.size.zero?
    end.join 
    
    # Calling "join" allows current process to wait until the JOINED thread is finished. 
    # Thus we can ensure our parent process doesn't exit until all tasks are done.
  end
end

nice!

Just wondering, won't it lead to memory leaks since your threads will live forever and every time you'll call import! it will be spawning 4 new threads?

hi @mftaher bhai, actually i took out this snapshot from one of my rake tasks, so in that case it wasn't an issue since rake task spins in and do it job then exit.

Otherwise the middle block is leaky if you are trying to use it in normal long running process. I'm adding additional changes so it does reflect my intent.

thanks for mentioning :)

hi @mftaher bhai check out it now :)

@we4tech now it looks elegant and make sense ... more ruby like

thanks @mftaher bhai :)