Skip to content

Instantly share code, notes, and snippets.

@benmwendwa

benmwendwa/sisekiq background processing

Created February 19, 2015 06:47
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save benmwendwa/349c1ca665e97ff2f848 to your computer and use it in GitHub Desktop.
Save benmwendwa/349c1ca665e97ff2f848 to your computer and use it in GitHub Desktop.
Download ZIP
using sidekiq to process a background job in sinatra.
Raw
sisekiq background processing
# in one file I have required all the needed resources.
require 'rubygems'
require 'google/api_client'
require 'google/api_client/client_secrets'
require 'google/api_client/auth/file_storage'
require 'sinatra'
require 'logger'
require 'mail'
require 'mysql2'
require 'json'
require 'oauth2'
require 'google_contacts_api'
require 'sinatra/redirect_with_flash'
require 'sinatra/flash'
require 'haml'
require 'oauth'
require 'rest_client'
require 'securerandom'
require 'sidekiq'
require 'sidekiq/web'
# I want to test if the sidekiq background worker can get contacts in the background for one token which is passed to it from another function.
# this is the method call to the sidekiq worker:
PygmentsWorker.perform_async("google",token, refresh_token, row["user_email"], row["user_id"])
# the background worker method looks like this:
class PygmentsWorker
include Sidekiq::Worker
def perform(provider, token, refresh_token, user_email, user_id)
response = GoogleContactsApi::User.new(token)
contacts = response.contacts
puts "\n\nGetting all the user contacts and saving them saving them into MySql table\n" if response
# the contacts are returned as an hash and I loop through then to get the emails
contacts.each do |contact|
contact_email = contact.primary_email
unless contact_email.nil?
# call another method to save the email address
sidekiq_save_contact(user_email, user_id, contact_email, refresh_token, 1, provider)
end
end
# update token status
update_token_status(refresh_token)
end
def sidekiq_save_contact(user_email, user_id, contact_email, refresh_token_used, token_status, provider_used)
begin
con = Mysql2::Client.new($con)
con.query("INSERT INTO Contacts (user_email, user_id, contact_email, refresh_token_used, token_status, provider_used) SELECT * FROM (SELECT '#{user_email}' AS 'user_email', '#{user_id}' AS 'user_id', '#{contact_email}' AS 'contact_email', '#{refresh_token_used}' AS 'refresh_token_used', '#{token_status}' AS 'token_status', '#{provider_used}' AS 'provider_used') AS tmp WHERE NOT EXISTS (SELECT contact_email FROM Contacts WHERE contact_email = '#{contact_email}') LIMIT 1;")
rescue Mysql2::Error => e
puts e.errno
puts e.error
ensure
con.close if con
end
end
end
# running process:
1. I run the sinatra application.
2. I start the redis-server
3. I run this command ~>$ bundle exec sidekiq -r ./others/sidekiq.rb
4. I push the job to the background. i.e call this method: PygmentsWorker.perform_async("google",token, refresh_token, row["user_email"], row["user_id"])
# this is the output that i'm getting on the terminal where I had run sidekiq(bundle exec sidekiq...)
2015-02-19T06:40:01.380Z 9595 TID-apid4 INFO: Running in ruby 2.1.5p273 (2014-11-13 revision 48405) [x86_64-linux]
2015-02-19T06:40:01.380Z 9595 TID-apid4 INFO: See LICENSE and the LGPL-3.0 for licensing details.
2015-02-19T06:40:01.380Z 9595 TID-apid4 INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org/pro
2015-02-19T06:40:01.380Z 9595 TID-apid4 INFO: Starting processing, hit Ctrl-C to stop
2015-02-19T06:40:01.433Z 9595 TID-gxfug INFO: Booting Sidekiq 3.3.2 with redis options {}
2015-02-19T06:40:01.434Z 9595 TID-amb70 INFO: Booting Sidekiq 3.3.2 with redis options {}
2015-02-19T06:40:23.664Z 9595 TID-bsjt8 PygmentsWorker JID-946e711788266c914a4946b6 INFO: start
2015-02-19T06:40:23.666Z 9595 TID-bsjt8 PygmentsWorker JID-946e711788266c914a4946b6 INFO: fail: 0.001 sec
2015-02-19T06:40:23.667Z 9595 TID-bsjt8 WARN: {"retry"=>true, "queue"=>"default", "class"=>"PygmentsWorker", "args"=>["google", "#<OAuth2::AccessToken:0x00000002807738>", "1/B3_fx3slHEfOccyy5fmcK9q5KMkq3LJ4XI3lFEAnmucMEudVrK5jSpoR30zcRFq6", "", "6FJM8dH53xjc8g"], "jid"=>"946e711788266c914a4946b6", "enqueued_at"=>1424328023.6608443, "error_message"=>"uninitialized constant PygmentsWorker::GoogleContactsApi", "error_class"=>"NameError", "failed_at"=>1424328023.665422, "retry_count"=>0}
2015-02-19T06:40:23.668Z 9595 TID-bsjt8 WARN: uninitialized constant PygmentsWorker::GoogleContactsApi
2015-02-19T06:40:23.668Z 9595 TID-bsjt8 WARN: /home/ben/rprojects/mailer/others/sidekiq.rb:5:in `perform'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/processor.rb:75:in `execute_job'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/processor.rb:52:in `block (2 levels) in process'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:127:in `call'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:127:in `block in invoke'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/logging.rb:24:in `with_context'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/server/logging.rb:7:in `call'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:132:in `call'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:132:in `invoke'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/processor.rb:51:in `block in process'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/processor.rb:98:in `stats'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/processor.rb:50:in `process'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `public_send'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `dispatch'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/calls.rb:122:in `dispatch'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/cell.rb:60:in `block in invoke'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/cell.rb:71:in `block in task'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/actor.rb:357:in `block in task'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/tasks.rb:57:in `block in initialize'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/tasks/task_fiber.rb:15:in `block in create'
2015-02-19T06:41:06.969Z 9595 TID-bsjt8 PygmentsWorker JID-946e711788266c914a4946b6 INFO: start
2015-02-19T06:41:06.970Z 9595 TID-bsjt8 PygmentsWorker JID-946e711788266c914a4946b6 INFO: fail: 0.002 sec
2015-02-19T06:41:06.972Z 9595 TID-bsjt8 WARN: {"retry"=>true, "queue"=>"default", "class"=>"PygmentsWorker", "args"=>["google", "#<OAuth2::AccessToken:0x00000002807738>", "1/B3_fx3slHEfOccyy5fmcK9q5KMkq3LJ4XI3lFEAnmucMEudVrK5jSpoR30zcRFq6", "", "6FJM8dH53xjc8g"], "jid"=>"946e711788266c914a4946b6", "enqueued_at"=>1424328023.6608443, "error_message"=>"uninitialized constant PygmentsWorker::GoogleContactsApi", "error_class"=>"NameError", "failed_at"=>1424328023.665422, "retry_count"=>1, "retried_at"=>1424328066.9699829}
2015-02-19T06:41:06.973Z 9595 TID-bsjt8 WARN: uninitialized constant PygmentsWorker::GoogleContactsApi
2015-02-19T06:41:06.973Z 9595 TID-bsjt8 WARN: /home/ben/rprojects/mailer/others/sidekiq.rb:5:in `perform'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/processor.rb:75:in `execute_job'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/processor.rb:52:in `block (2 levels) in process'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:127:in `call'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:127:in `block in invoke'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/logging.rb:24:in `with_context'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/server/logging.rb:7:in `call'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:132:in `call'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:132:in `invoke'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/processor.rb:51:in `block in process'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/processor.rb:98:in `stats'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/processor.rb:50:in `process'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `public_send'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `dispatch'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/calls.rb:122:in `dispatch'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/cell.rb:60:in `block in invoke'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/cell.rb:71:in `block in task'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/actor.rb:357:in `block in task'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/tasks.rb:57:in `block in initialize'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/tasks/task_fiber.rb:15:in `block in create'
2015-02-19T06:41:55.473Z 9595 TID-bsjt8 PygmentsWorker JID-946e711788266c914a4946b6 INFO: start
2015-02-19T06:41:55.474Z 9595 TID-bsjt8 PygmentsWorker JID-946e711788266c914a4946b6 INFO: fail: 0.001 sec
2015-02-19T06:41:55.475Z 9595 TID-bsjt8 WARN: {"retry"=>true, "queue"=>"default", "class"=>"PygmentsWorker", "args"=>["google", "#<OAuth2::AccessToken:0x00000002807738>", "1/B3_fx3slHEfOccyy5fmcK9q5KMkq3LJ4XI3lFEAnmucMEudVrK5jSpoR30zcRFq6", "", "6FJM8dH53xjc8g"], "jid"=>"946e711788266c914a4946b6", "enqueued_at"=>1424328023.6608443, "error_message"=>"uninitialized constant PygmentsWorker::GoogleContactsApi", "error_class"=>"NameError", "failed_at"=>1424328023.665422, "retry_count"=>2, "retried_at"=>1424328115.473585}
2015-02-19T06:41:55.476Z 9595 TID-bsjt8 WARN: uninitialized constant PygmentsWorker::GoogleContactsApi
2015-02-19T06:41:55.476Z 9595 TID-bsjt8 WARN: /home/ben/rprojects/mailer/others/sidekiq.rb:5:in `perform'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/processor.rb:75:in `execute_job'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/processor.rb:52:in `block (2 levels) in process'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:127:in `call'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:127:in `block in invoke'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/logging.rb:24:in `with_context'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/server/logging.rb:7:in `call'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:132:in `call'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:132:in `invoke'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/processor.rb:51:in `block in process'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/processor.rb:98:in `stats'
/home/ben/.rvm/gems/ruby-2.1.5/gems/sidekiq-3.3.2/lib/sidekiq/processor.rb:50:in `process'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `public_send'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `dispatch'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/calls.rb:122:in `dispatch'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/cell.rb:60:in `block in invoke'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/cell.rb:71:in `block in task'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/actor.rb:357:in `block in task'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/tasks.rb:57:in `block in initialize'
/home/ben/.rvm/gems/ruby-2.1.5/gems/celluloid-0.16.0/lib/celluloid/tasks/task_fiber.rb:15:in `block in create'
You can help resolve this please.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment