Skip to content

Instantly share code, notes, and snippets.

Mike Perham mperham

Block or report user

Report or block mperham

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
View gist:12ca6ff49ed0b665fd971fbada546ea5
~/src/sidekiq (master *=)$ bundle exec bin/sidekiqload
ruby 2.6.0preview3 (2018-11-06 trunk 65578) [x86_64-darwin17]
2018-11-06T18:17:06.966Z 34703 TID-ovsoikzhj INFO: Booting Sidekiq 5.2.3 with redis options {:db=>13, :id=>"Sidekiq-server-PID-34703", :url=>nil}
2018-11-06T18:17:10.046Z 34703 TID-ovsoikzhj ERROR: Created 100000 jobs
2018-11-06T18:17:11.058Z 34703 TID-ovsofju03 ERROR: RSS: 56880 Pending: 94926
2018-11-06T18:17:12.067Z 34703 TID-ovsofju03 ERROR: RSS: 57084 Pending: 89629
2018-11-06T18:17:13.080Z 34703 TID-ovsofju03 ERROR: RSS: 57104 Pending: 84563
2018-11-06T18:17:14.093Z 34703 TID-ovsofju03 ERROR: RSS: 57104 Pending: 79536
2018-11-06T18:17:15.105Z 34703 TID-ovsofju03 ERROR: RSS: 57104 Pending: 74438
2018-11-06T18:17:16.117Z 34703 TID-ovsofju03 ERROR: RSS: 57108 Pending: 69405
@mperham
mperham / frag.rb
Last active Jun 4, 2019
memory fragmentation on ruby 2.5.1
View frag.rb
=begin
This script attempts to reproduce poor glibc allocator behavior within Ruby, leading
to extreme memory fragmentation and process RSS bloat.
glibc allocates memory using per-thread "arenas". These blocks can easily fragment when
some objects are free'd and others are long-lived.
Our script runs multiple threads, all allocating randomly sized "large" Strings between 4,000
and 40,000 bytes in size. This simulates Rails views with ERB creating large chunks of HTML
to output to the browser. Some of these strings are kept around and some are discarded.
View bulk_bug_test.go
package storage
import (
"crypto/rand"
"encoding/base64"
"fmt"
"os"
"testing"
"time"
@mperham
mperham / gist:544327d176f9693df05d4d60548b0b16
Last active Sep 8, 2016
Fast counters with Redis, persisting to DB
View gist:544327d176f9693df05d4d60548b0b16
class Counter
include Sidekiq::Worker
# I'd use a separate Redis instance from Sidekiq
REDIS = ConnectionPool.new(size: 5) { Redis.new(...) }
# Call this API in your Rails app code to increment counters
# as the user does things.
def self.incr(name, amount=1)
key = "counter-#{name}-#{current_window}"
View gist:399f0ed0eca321aeaf9ee20cf104a849
~/src/ent/myapp (master *=)$ bundle exec derailed bundle:objects
Measuring objects created by gems in groups [:default, "production"]
Total allocated: 4246025 bytes (34726 objects)
Total retained: 437421 bytes (4179 objects)
allocated memory by gem
-----------------------------------
1348174 puma-3.6.0
749251 sidekiq/lib
538517 pro/lib
View load.cr
require "./src/sidekiq"
`redis-cli flushdb`
class LoadWorker
include Sidekiq::Worker
perform_types Int64
def perform(idx)
end
@mperham
mperham / locktest.rb
Created Apr 23, 2016
Distributed Locking with Redis and Ruby
View locktest.rb
require 'benchmark'
require 'sidekiq-ent'
require 'redis-lock'
require 'redis-semaphore'
require 'ruby_redis_lock'
# monkey patch to remove exponential backoff in pmckee11-redis-lock,
# otherwise this benchmark does not complete successfully.
class Redis
class Lock
View gist:d35154a06946cee8a64b
$ ruby -v
ruby 2.0.0p598 (2014-11-13) [x86_64-linux]
$ irb
irb(main):001:0> require 'json'
=> true
irb(main):002:0> quit
$ bundle exec irb
rirb(main):001:0> require 'json'
LoadError: cannot load such file -- json
View example.rb
class MyCode
def process
do_something if leader?
end
def leader?
Sidekiq::Senate.leader?
end
end
@mperham
mperham / repro.md
Created Jul 22, 2015
Sidekiq / Rails 5 interlockup
View repro.md
git clone git://github.com/mperham/sidekiq
cd sidekiq && git checkout rails5
cd myapp && bundle
bundle exec sidekiq
# in another terminal tab
bundle exec rails s

Now hit http://localhost:3000/work to create a random set of jobs for sidekiq to perform.

You can’t perform that action at this time.