draffensperger

In MPDX, the Google Contacts sync job takes a long time and the google accounts loop to sync each job could benefit from find_each(batch_size: 1). Basically it seems likes find_each pulls in the records in batches and then saves them in an array to enumerate through. Here's a comparison of the memory results with different batch_sizes that I did using a similar, but contrived MemHungry model. To setup, first do 6.times { MemHungry.create }.

Using the default batch_size of 1000 the memory at the end reflecs all 6 objects and the RAM they hold onto:

[1] pry(main)> MemHungry.all.find_each { |m| puts m.id; m.eat_memory; }
  MemHungry Load (0.5ms)  SELECT  "mem_hungries".* FROM "mem_hungries"   ORDER BY "mem_hungries"."id" ASC LIMIT 1000
1
Memory before GC: 112.63671875
Memory before allocation: 159.90234375
Memory after allocation: 1759.90234375

draffensperger

module JobDuplicateChecker
  def duplicate_job?(*args)
    job_in_retries?(args) || older_job_running?(args)
  end

  private

  def older_job_running?(args)
    workers = Sidekiq::Workers.new
    self_worker = workers.find { |_, _, work| work['payload']['jid'] == jid }

draffensperger

Keybase proof

I hereby claim:

• I am draffensperger on github.
• I am draffensperger (https://keybase.io/draffensperger) on keybase.
• I have a public key whose fingerprint is 11AD 4270 6B18 7B21 BDCF 7A62 66E0 7480 D06A 9FE6

To claim this, I am signing this object: