Skip to content

Instantly share code, notes, and snippets.

View mperham's full-sized avatar

Mike Perham mperham

View GitHub Profile
@mperham
mperham / bench.rb
Last active October 4, 2024 16:25
Atomic counters: MutexFixnum vs concurrent-ruby's AtomicFixnum
require "benchmark/ips"
require "concurrent"
class MutexFixnum
def initialize
@value = 0
@lock = Mutex.new
end
def increment(amount = 1)
@mperham
mperham / IRB.txt
Created February 15, 2024 00:26
OpenSSL error handled despite not being required?!?!?!
irb(main):004> OpenSSL
(irb):4:in `<main>': uninitialized constant OpenSSL (NameError)
Did you mean? OpenStruct
from /Users/mperham/.gem/ruby/3.2.0/gems/irb-1.10.1/exe/irb:9:in `<top (required)>'
from /Users/mperham/.gem/ruby/3.2.0/bin/irb:25:in `load'
from /Users/mperham/.gem/ruby/3.2.0/bin/irb:25:in `<main>'
irb(main):005> ENV["FAKTORY_URL"] = "tcp+tls://test.contribsys.com:7419"
=> "tcp+tls://test.contribsys.com:7419"
irb(main):006> f = Faktory::Client.new
/Users/mperham/src/fwr/lib/faktory/client.rb:318:in `rescue in open_socket': Server not using TLS? Use FAKTORY_URL=tcp://... to disable encryption (RuntimeError)
@mperham
mperham / wikitext.md
Created January 29, 2024 19:42
Sidekiq + DragonflyDB

Sidekiq can use Dragonfly, an open source datastore compatible with Redis.

By default, Sidekiq connects localhost:6379. Since Dragonfly (hereafter, DF) is 100% compatible with Redis clients, your configuration will look identical to Redis configuration.

Using an ENV variable

You can configure DF's location using an environment variable. The easiest option is to set REDIS_URL; Sidekiq will pick it up and use it. A Redis URL looks like redis://[hostname]:[port]/[dbnumber], e.g. redis://redis.example.com:7777/11.

@mperham
mperham / gist:dcf7fca8e14ddee235be9c3bed9bcda6
Last active January 7, 2022 17:03
Truffle 21.3.0 vs MRI 2.7.2

This benchmark creates 100,000 jobs in Redis, boots Sidekiq with 10 threads and waits for all jobs to be processed. This is not a microbenchmark -- it is a "macrobenchmark" designed to test real system throughput.

Code is at https://github.com/mperham/sidekiq/blob/main/bin/sidekiqload

❯ bundle exec bin/sidekiqload
truffleruby 21.3.0, like ruby 2.7.4, GraalVM CE Native [x86_64-darwin]
2022-01-07T16:56:17.644Z pid=40281 tid=29xt INFO: Booting Sidekiq 6.3.2 with redis options {:db=>13, :port=>6380}
2022-01-07T16:56:36.557Z pid=40281 tid=29xt ERROR: Created 100000 jobs
@mperham
mperham / Benchmark.rb
Last active October 7, 2023 18:44
Leaky bucket limiter usage with Sidekiq Enterprise
require 'benchmark'
require 'sidekiq-ent/limiter'
COUNT = 10_000
Benchmark.bmbm(30) do |x|
x.report("leaky") do
COUNT.times do |count|
lmt = Sidekiq::Limiter.leaky("leaky_#{count%100}", 10, 10, wait_timeout: 0, policy: :skip)
lmt.within_limit do
❯ make runruby
./miniruby -I./lib -I. -I.ext/common ./tool/runruby.rb --extout=.ext -- --disable-gems ./test.rb
<internal:ractor>:38: warning: Ractor is experimental, and the behavior may change in future versions of Ruby! Also there are many implementation issues.
#<Thread:0x00007fadd701fa90 run> terminated with exception (report_on_exception is true):
<internal:ractor>:124:in `take': thrown by remote Ractor. (Ractor::RemoteError)
from ./test.rb:11:in `<main>'
./test.rb:5:in `logger': can not access instance variables of classes/modules from non-main Ractors (RuntimeError)
from ./test.rb:10:in `block in <main>'
make: *** [runruby] Error 1
@mperham
mperham / aj.txt
Created May 8, 2020 15:54
AJ vs Sidekiq backtraces
2020-05-08T15:52:22.241Z pid=10710 tid=2me WARN: RuntimeError: boom
2020-05-08T15:52:22.241Z pid=10710 tid=2me WARN: /Users/mikeperham/src/sidekiq/myapp/app/jobs/some_job.rb:5:in `perform'
/Users/mikeperham/.gem/ruby/2.7.1/gems/activejob-6.0.3/lib/active_job/execution.rb:40:in `block in perform_now'
/Users/mikeperham/.gem/ruby/2.7.1/gems/activesupport-6.0.3/lib/active_support/callbacks.rb:112:in `block in run_callbacks'
/Users/mikeperham/.gem/ruby/2.7.1/gems/i18n-1.8.2/lib/i18n.rb:313:in `with_locale'
/Users/mikeperham/.gem/ruby/2.7.1/gems/activejob-6.0.3/lib/active_job/translation.rb:9:in `block (2 levels) in <module:Translation>'
/Users/mikeperham/.gem/ruby/2.7.1/gems/activesupport-6.0.3/lib/active_support/callbacks.rb:121:in `instance_exec'
/Users/mikeperham/.gem/ruby/2.7.1/gems/activesupport-6.0.3/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/Users/mikeperham/.gem/ruby/2.7.1/gems/activesupport-6.0.3/lib/active_support/core_ext/time/zones.rb:66:in `use_zone'
/Users/mikeperham/.gem/ruby/2
~/src/sidekiq (master *=)$ bundle exec bin/sidekiqload
ruby 2.6.0preview3 (2018-11-06 trunk 65578) [x86_64-darwin17]
2018-11-06T18:17:06.966Z 34703 TID-ovsoikzhj INFO: Booting Sidekiq 5.2.3 with redis options {:db=>13, :id=>"Sidekiq-server-PID-34703", :url=>nil}
2018-11-06T18:17:10.046Z 34703 TID-ovsoikzhj ERROR: Created 100000 jobs
2018-11-06T18:17:11.058Z 34703 TID-ovsofju03 ERROR: RSS: 56880 Pending: 94926
2018-11-06T18:17:12.067Z 34703 TID-ovsofju03 ERROR: RSS: 57084 Pending: 89629
2018-11-06T18:17:13.080Z 34703 TID-ovsofju03 ERROR: RSS: 57104 Pending: 84563
2018-11-06T18:17:14.093Z 34703 TID-ovsofju03 ERROR: RSS: 57104 Pending: 79536
2018-11-06T18:17:15.105Z 34703 TID-ovsofju03 ERROR: RSS: 57104 Pending: 74438
2018-11-06T18:17:16.117Z 34703 TID-ovsofju03 ERROR: RSS: 57108 Pending: 69405
@mperham
mperham / frag.rb
Last active April 26, 2023 18:22
memory fragmentation on ruby 2.5.1
=begin
This script attempts to reproduce poor glibc allocator behavior within Ruby, leading
to extreme memory fragmentation and process RSS bloat.
glibc allocates memory using per-thread "arenas". These blocks can easily fragment when
some objects are free'd and others are long-lived.
Our script runs multiple threads, all allocating randomly sized "large" Strings between 4,000
and 40,000 bytes in size. This simulates Rails views with ERB creating large chunks of HTML
to output to the browser. Some of these strings are kept around and some are discarded.
package storage
import (
"crypto/rand"
"encoding/base64"
"fmt"
"os"
"testing"
"time"