mperham

Go has a number of low-level crypto APIs which check off marketing bullet-points (got FIPS supprt, check!) but is missing an high-level API usable by mere mortal programmers. Imagine you want to create a document, sign it and verify that document later. Now check out Go's crypto APIs and give up in frustration after an hour of Googling.

The API should encapsulate a half-dozen common operations and make them as easy as possible. Avoid choice where possible, just pick something reasonably secure in 2014 for me and use it! I'm speaking specifically of a few basic actions (yes, this API is very naive/non-idiomatic), call it crypto/easy:

• Create me a public/private key pair and save it to the filesystem.

// create and persist a keypair to the current directory.
// this is just a one-time operation, now we have a keypair to use.
easy.CreateKeyPair()

mperham

irb(main):004> OpenSSL
(irb):4:in `<main>': uninitialized constant OpenSSL (NameError)
Did you mean?  OpenStruct
	from /Users/mperham/.gem/ruby/3.2.0/gems/irb-1.10.1/exe/irb:9:in `<top (required)>'
	from /Users/mperham/.gem/ruby/3.2.0/bin/irb:25:in `load'
	from /Users/mperham/.gem/ruby/3.2.0/bin/irb:25:in `<main>'
irb(main):005> ENV["FAKTORY_URL"] = "tcp+tls://test.contribsys.com:7419"
=> "tcp+tls://test.contribsys.com:7419"
irb(main):006> f = Faktory::Client.new
/Users/mperham/src/fwr/lib/faktory/client.rb:318:in `rescue in open_socket': Server not using TLS? Use FAKTORY_URL=tcp://... to disable encryption (RuntimeError)

mperham

Sidekiq can use Dragonfly, an open source datastore compatible with Redis.

By default, Sidekiq connects localhost:6379. Since Dragonfly (hereafter, DF) is 100% compatible with Redis clients, your configuration will look identical to Redis configuration.

Using an ENV variable

You can configure DF's location using an environment variable. The easiest option is to set REDIS_URL; Sidekiq will pick it up and use it. A Redis URL looks like redis://[hostname]:[port]/[dbnumber], e.g. redis://redis.example.com:7777/11.

mperham

require 'benchmark'
require 'sidekiq-ent/limiter'

COUNT = 10_000

Benchmark.bmbm(30) do |x|
  x.report("leaky") do
    COUNT.times do |count|
      lmt = Sidekiq::Limiter.leaky("leaky_#{count%100}", 10, 10, wait_timeout: 0, policy: :skip)
      lmt.within_limit do

mperham

=begin
This script attempts to reproduce poor glibc allocator behavior within Ruby, leading
to extreme memory fragmentation and process RSS bloat.

glibc allocates memory using per-thread "arenas".  These blocks can easily fragment when
some objects are free'd and others are long-lived.

Our script runs multiple threads, all allocating randomly sized "large" Strings between 4,000
and 40,000 bytes in size.  This simulates Rails views with ERB creating large chunks of HTML
to output to the browser. Some of these strings are kept around and some are discarded.

mperham

require 'sidekiq'

module Sidekiq

  # Encapsulates a single job awaiting retry
  class Retry
    attr_reader :score, :item
    def initialize(score, item)
      @score = score
      @item = Sidekiq.load_json(item)

mperham

This benchmark creates 100,000 jobs in Redis, boots Sidekiq with 10 threads and waits for all jobs to be processed. This is not a microbenchmark -- it is a "macrobenchmark" designed to test real system throughput.

Code is at https://github.com/mperham/sidekiq/blob/main/bin/sidekiqload

❯ bundle exec bin/sidekiqload
truffleruby 21.3.0, like ruby 2.7.4, GraalVM CE Native [x86_64-darwin]
2022-01-07T16:56:17.644Z pid=40281 tid=29xt INFO: Booting Sidekiq 6.3.2 with redis options {:db=>13, :port=>6380}
2022-01-07T16:56:36.557Z pid=40281 tid=29xt ERROR: Created 100000 jobs

mperham

  desc "convert a latin1 database with utf8 data into proper utf8"
  task :convert_to_utf8 => :environment do
    puts Time.now
    dryrun = ENV['DOIT'] != '1'
    conn = ActiveRecord::Base.connection
    if dryrun
      def conn.run_sql(sql)
        puts(sql)
      end
    else

mperham

❯ make runruby
./miniruby -I./lib -I. -I.ext/common  ./tool/runruby.rb --extout=.ext  -- --disable-gems ./test.rb
<internal:ractor>:38: warning: Ractor is experimental, and the behavior may change in future versions of Ruby! Also there are many implementation issues.
#<Thread:0x00007fadd701fa90 run> terminated with exception (report_on_exception is true):
<internal:ractor>:124:in `take': thrown by remote Ractor. (Ractor::RemoteError)
	from ./test.rb:11:in `<main>'
./test.rb:5:in `logger': can not access instance variables of classes/modules from non-main Ractors (RuntimeError)
	from ./test.rb:10:in `block in <main>'
make: *** [runruby] Error 1

mperham

2020-05-08T15:52:22.241Z pid=10710 tid=2me WARN: RuntimeError: boom
2020-05-08T15:52:22.241Z pid=10710 tid=2me WARN: /Users/mikeperham/src/sidekiq/myapp/app/jobs/some_job.rb:5:in `perform'
/Users/mikeperham/.gem/ruby/2.7.1/gems/activejob-6.0.3/lib/active_job/execution.rb:40:in `block in perform_now'
/Users/mikeperham/.gem/ruby/2.7.1/gems/activesupport-6.0.3/lib/active_support/callbacks.rb:112:in `block in run_callbacks'
/Users/mikeperham/.gem/ruby/2.7.1/gems/i18n-1.8.2/lib/i18n.rb:313:in `with_locale'
/Users/mikeperham/.gem/ruby/2.7.1/gems/activejob-6.0.3/lib/active_job/translation.rb:9:in `block (2 levels) in <module:Translation>'
/Users/mikeperham/.gem/ruby/2.7.1/gems/activesupport-6.0.3/lib/active_support/callbacks.rb:121:in `instance_exec'
/Users/mikeperham/.gem/ruby/2.7.1/gems/activesupport-6.0.3/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/Users/mikeperham/.gem/ruby/2.7.1/gems/activesupport-6.0.3/lib/active_support/core_ext/time/zones.rb:66:in `use_zone'
/Users/mikeperham/.gem/ruby/2