Skip to content

Instantly share code, notes, and snippets.

Chris Hanks chanks

Block or report user

Report or block chanks

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@chanks
chanks / script.lua
Last active Jan 8, 2020
Lua script to delete/trim all processed messages from a Redis stream - an updated version is maintained here: https://gist.github.com/FSX/fb86595c64751201497e2050aeb722e2
View script.lua
-- The goal of this script is to trim messages that have been processed by
-- all extant groups from the a given Redis stream. It returns the number
-- of messages that were deleted from the stream, if any. I make no
-- guarantees about its performance, particularly if the stream is large
-- and not fully processed (so a simple XTRIM isn't possible).
-- First off, bail out early if the stream doesn't exist.
if redis.call("EXISTS", KEYS[1]) == 0 then
return false
end
View backtrace.rb
["/home/chris/.rvm/rubies/ruby-2.5.0/lib/ruby/2.5.0/monitor.rb:109:in `sleep'",
"/home/chris/.rvm/rubies/ruby-2.5.0/lib/ruby/2.5.0/monitor.rb:109:in `wait'",
"/home/chris/.rvm/rubies/ruby-2.5.0/lib/ruby/2.5.0/monitor.rb:109:in `wait'",
"/home/chris/.rvm/rubies/ruby-2.5.0/lib/ruby/2.5.0/monitor.rb:121:in `wait_while'",
"/home/chris/.rvm/gems/ruby-2.5.0@rails-test/gems/activesupport-5.1.6/lib/active_support/concurrency/share_lock.rb:219:in `wait_for'",
"/home/chris/.rvm/gems/ruby-2.5.0@rails-test/gems/activesupport-5.1.6/lib/active_support/concurrency/share_lock.rb:81:in `block (2 levels) in start_exclusive'",
"/home/chris/.rvm/gems/ruby-2.5.0@rails-test/gems/activesupport-5.1.6/lib/active_support/concurrency/share_lock.rb:185:in `yield_shares'",
"/home/chris/.rvm/gems/ruby-2.5.0@rails-test/gems/activesupport-5.1.6/lib/active_support/concurrency/share_lock.rb:80:in `block in start_exclusive'",
"/home/chris/.rvm/rubies/ruby-2.5.0/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'",
"/home/chris
View gist:6fd90efd875331535441
require 'sequel'
require 'logger'
DB = Sequel.connect "postgres:///scrap"
DB.loggers << Logger.new(STDOUT)
Sequel::Model.plugin :tactical_eager_loading
DB.drop_table? :posts
@chanks
chanks / gist:7585810
Last active Apr 3, 2020
Turning PostgreSQL into a queue serving 10,000 jobs per second
View gist:7585810

Turning PostgreSQL into a queue serving 10,000 jobs per second

RDBMS-based job queues have been criticized recently for being unable to handle heavy loads. And they deserve it, to some extent, because the queries used to safely lock a job have been pretty hairy. SELECT FOR UPDATE followed by an UPDATE works fine at first, but then you add more workers, and each is trying to SELECT FOR UPDATE the same row (and maybe throwing NOWAIT in there, then catching the errors and retrying), and things slow down.

On top of that, they have to actually update the row to mark it as locked, so the rest of your workers are sitting there waiting while one of them propagates its lock to disk (and the disks of however many servers you're replicating to). QueueClassic got some mileage out of the novel idea of randomly picking a row near the front of the queue to lock, but I can't still seem to get more than an an extra few hundred jobs per second out of it under heavy load.

So, many developers have started going straight t

@chanks
chanks / gist:4719494
Created Feb 6, 2013
Quick alternate implementation for Dataset#query.
View gist:4719494
module Sequel
class Dataset
class Query
attr_reader :dataset
def initialize(dataset)
@dataset = dataset
end
def method_missing(method, *args, &block)
@chanks
chanks / .rubinius_last_error
Created Nov 11, 2012
Rubinius crash with sequel_pg gem.
View .rubinius_last_error
5275 6269 6e69 7573 2043 7261 7368 2052
6570 6f72 7420 2372 6278 6372 6173 6872
6570 6f72 740a 0a45 7272 6f72 3a20 7369
676e 616c 2000 5349 4753 4547 560a 0a5b
5b42 6163 6b74 7261 6365 5d5d 0a2f 686f
6d65 2f63 6872 6973 2f2e 7262 656e 762f
7665 7273 696f 6e73 2f72 6278 2d32 2e30
2e30 2d64 6576 2f62 696e 2f72 6278 5b30
7835 6239 6161 305d 0a2f 6c69 622f 7838
365f 3634 2d6c 696e 7578 2d67 6e75 2f6c
@chanks
chanks / gist:2829829
Created May 29, 2012
Sequel nested attributes possible API
View gist:2829829
require 'rubygems'
require 'sequel'
DB = Sequel.sqlite
DB.create_table :questions do
primary_key :id
String :content
end
@chanks
chanks / gist:2820828
Created May 28, 2012
Sequel nested attributes spec
View gist:2820828
it "should support creating new objects with composite primary keys" do
a = @Artist.new({:name=>'Ar', :concerts_attributes=>[{:tour => 'To', :date => "2004-04-05"}]})
@db.sqls.should == []
a.save
check_sql_array("INSERT INTO artists (name) VALUES ('Ar')",
["INSERT INTO concerts (tour, date, artist_id) VALUES ('To', '2004-04-05', 1)",
"INSERT INTO concerts (artist_id, tour, date) VALUES (1, 'To', '2004-04-05')"
]) # etc.
end
@chanks
chanks / gist:2289822
Created Apr 3, 2012
Sequel bug with cursors and statement parameterization.
View gist:2289822
Loading development environment (Rails 3.2.2)
irb(main):001:0> DB.loggers << Logger.new(STDOUT)
=> [...array of loggers...]
irb(main):002:0> Problem.dataset.first
(0.001232s) SELECT * FROM "problems" LIMIT $1::int4; [1]
=> #<Problem @values={:id=>1, :content=>...}>
irb(main):003:0> Problem.dataset.use_cursor.first
(0.000347s) BEGIN
PGError: ERROR: there is no parameter $1
LINE 1: ...SOR WITHOUT HOLD FOR SELECT * FROM "problems" LIMIT $1::int4
@chanks
chanks / gist:1119590
Created Aug 2, 2011
Annoying bug in Sequel
View gist:1119590
require "sequel"
DB = Sequel.sqlite
DB.create_table :users do
primary_key :id
end
DB.create_table :quizzes do
Integer :user_id
You can’t perform that action at this time.