Skip to content

Instantly share code, notes, and snippets.

chanks / script.lua
Last active May 26, 2023 05:07
Lua script to delete/trim all processed messages from a Redis stream - an updated version is maintained here:
-- The goal of this script is to trim messages that have been processed by
-- all extant groups from the a given Redis stream. It returns the number
-- of messages that were deleted from the stream, if any. I make no
-- guarantees about its performance, particularly if the stream is large
-- and not fully processed (so a simple XTRIM isn't possible).
-- First off, bail out early if the stream doesn't exist.
if"EXISTS", KEYS[1]) == 0 then
return false
["/home/chris/.rvm/rubies/ruby-2.5.0/lib/ruby/2.5.0/monitor.rb:109:in `sleep'",
"/home/chris/.rvm/rubies/ruby-2.5.0/lib/ruby/2.5.0/monitor.rb:109:in `wait'",
"/home/chris/.rvm/rubies/ruby-2.5.0/lib/ruby/2.5.0/monitor.rb:109:in `wait'",
"/home/chris/.rvm/rubies/ruby-2.5.0/lib/ruby/2.5.0/monitor.rb:121:in `wait_while'",
"/home/chris/.rvm/gems/ruby-2.5.0@rails-test/gems/activesupport-5.1.6/lib/active_support/concurrency/share_lock.rb:219:in `wait_for'",
"/home/chris/.rvm/gems/ruby-2.5.0@rails-test/gems/activesupport-5.1.6/lib/active_support/concurrency/share_lock.rb:81:in `block (2 levels) in start_exclusive'",
"/home/chris/.rvm/gems/ruby-2.5.0@rails-test/gems/activesupport-5.1.6/lib/active_support/concurrency/share_lock.rb:185:in `yield_shares'",
"/home/chris/.rvm/gems/ruby-2.5.0@rails-test/gems/activesupport-5.1.6/lib/active_support/concurrency/share_lock.rb:80:in `block in start_exclusive'",
"/home/chris/.rvm/rubies/ruby-2.5.0/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'",
require 'sequel'
require 'logger'
DB = Sequel.connect "postgres:///scrap"
DB.loggers <<
Sequel::Model.plugin :tactical_eager_loading
DB.drop_table? :posts
chanks / gist:7585810
Last active February 29, 2024 03:50
Turning PostgreSQL into a queue serving 10,000 jobs per second

Turning PostgreSQL into a queue serving 10,000 jobs per second

RDBMS-based job queues have been criticized recently for being unable to handle heavy loads. And they deserve it, to some extent, because the queries used to safely lock a job have been pretty hairy. SELECT FOR UPDATE followed by an UPDATE works fine at first, but then you add more workers, and each is trying to SELECT FOR UPDATE the same row (and maybe throwing NOWAIT in there, then catching the errors and retrying), and things slow down.

On top of that, they have to actually update the row to mark it as locked, so the rest of your workers are sitting there waiting while one of them propagates its lock to disk (and the disks of however many servers you're replicating to). QueueClassic got some mileage out of the novel idea of randomly picking a row near the front of the queue to lock, but I can't still seem to get more than an an extra few hundred jobs per second out of it under heavy load.

So, many developers have started going straight t

chanks / gist:4719494
Created February 6, 2013 01:46
Quick alternate implementation for Dataset#query.
module Sequel
class Dataset
class Query
attr_reader :dataset
def initialize(dataset)
@dataset = dataset
def method_missing(method, *args, &block)
chanks / .rubinius_last_error
Created November 11, 2012 20:14
Rubinius crash with sequel_pg gem.
5275 6269 6e69 7573 2043 7261 7368 2052
6570 6f72 7420 2372 6278 6372 6173 6872
6570 6f72 740a 0a45 7272 6f72 3a20 7369
676e 616c 2000 5349 4753 4547 560a 0a5b
5b42 6163 6b74 7261 6365 5d5d 0a2f 686f
6d65 2f63 6872 6973 2f2e 7262 656e 762f
7665 7273 696f 6e73 2f72 6278 2d32 2e30
2e30 2d64 6576 2f62 696e 2f72 6278 5b30
7835 6239 6161 305d 0a2f 6c69 622f 7838
365f 3634 2d6c 696e 7578 2d67 6e75 2f6c
chanks / gist:2829829
Created May 29, 2012 18:13
Sequel nested attributes possible API
require 'rubygems'
require 'sequel'
DB = Sequel.sqlite
DB.create_table :questions do
primary_key :id
String :content
chanks / gist:2820828
Created May 28, 2012 19:35
Sequel nested attributes spec
it "should support creating new objects with composite primary keys" do
a ={:name=>'Ar', :concerts_attributes=>[{:tour => 'To', :date => "2004-04-05"}]})
@db.sqls.should == []
check_sql_array("INSERT INTO artists (name) VALUES ('Ar')",
["INSERT INTO concerts (tour, date, artist_id) VALUES ('To', '2004-04-05', 1)",
"INSERT INTO concerts (artist_id, tour, date) VALUES (1, 'To', '2004-04-05')"
]) # etc.
chanks / gist:2289822
Created April 3, 2012 06:41
Sequel bug with cursors and statement parameterization.
Loading development environment (Rails 3.2.2)
irb(main):001:0> DB.loggers <<
=> [...array of loggers...]
irb(main):002:0> Problem.dataset.first
(0.001232s) SELECT * FROM "problems" LIMIT $1::int4; [1]
=> #<Problem @values={:id=>1, :content=>...}>
irb(main):003:0> Problem.dataset.use_cursor.first
(0.000347s) BEGIN
PGError: ERROR: there is no parameter $1
chanks / gist:1119590
Created August 2, 2011 04:14
Annoying bug in Sequel
require "sequel"
DB = Sequel.sqlite
DB.create_table :users do
primary_key :id
DB.create_table :quizzes do
Integer :user_id