dvliman

Turning PostgreSQL into a queue serving 10,000 jobs per second

RDBMS-based job queues have been criticized recently for being unable to handle heavy loads. And they deserve it, to some extent, because the queries used to safely lock a job have been pretty hairy. SELECT FOR UPDATE followed by an UPDATE works fine at first, but then you add more workers, and each is trying to SELECT FOR UPDATE the same row (and maybe throwing NOWAIT in there, then catching the errors and retrying), and things slow down.

On top of that, they have to actually update the row to mark it as locked, so the rest of your workers are sitting there waiting while one of them propagates its lock to disk (and the disks of however many servers you're replicating to). QueueClassic got some mileage out of the novel idea of randomly picking a row near the front of the queue to lock, but I can't still seem to get more than an an extra few hundred jobs per second out of it under heavy load.

So, many developers have started going straight t

dvliman

package httplib

import (
	"net/http"
	"net/http/httptest"
	"net/url"
	"testing"


	"github.com/manyminds/api2go"

dvliman

set -gx PATH $PATH /usr/local/sbin
set -gx GOPATH $HOME/go
set -gx GOBIN $GOPATH/bin
set -gx PATH $PATH $GOBIN
set -gx GOOGLE_APPLICATION_CREDENTIALS /Users/dliman/lereta/acq-plt/app-credentials.json
set -gx GOGOOGLE_APPLICATION_CREDENTIALS /Users/dliman/lereta/acq-plt/app-credentials.json
set -gx APP_ENV local
set -gx PROJECT_ID ltf-eng-dliman-cb8b

function set_psql

dvliman

Time.now.strftime("%Y%m%d%H%M%S")

dvliman

# rename files to have underscores instead of ' ', '(', and ')' - runs on * or takes extension argument: i.e., `rename_filename_chars txt`
rename_filename_chars() { for file in *"$1"; do mv -n "$file" $(echo $file | sed -e "s/ /_/g" | sed -e "s/(/_/g" | sed -e  "s/)/_/g"); done }

dvliman

;; -*- mode: emacs-lisp -*-
(defun dotspacemacs/layers ()
  (setq-default
   dotspacemacs-distribution 'spacemacs
   dotspacemacs-enable-lazy-installation 'unused
   dotspacemacs-ask-for-lazy-installation t
   ;; If non-nil layers with lazy install support are lazy installed.
   ;; List of additional paths where to look for configuration layers.
   ;; Paths must have a trailing slash (i.e. `~/.mycontribs/')
   dotspacemacs-configuration-layer-path '()

dvliman

public static void main(String[] args) throws Exception {
    Security.addProvider(new BouncyCastleProvider());
    KeyPairGenerator keygen = KeyPairGenerator.getInstance("ECDH", "BC");
    keygen.initialize(ECNamedCurveTable.getParameterSpec("P-256"), new SecureRandom());

    KeyPair keypair1 = keygen.generateKeyPair();
    KeyPair keypair2 = keygen.generateKeyPair();

    String secret1 = computeSecret((ECPrivateKey) keypair1.getPrivate(), (ECPublicKey) keypair2.getPublic());
    String secret2 = computeSecret((ECPrivateKey) keypair2.getPrivate(), (ECPublicKey) keypair1.getPublic());

dvliman

Client-side SSL

For excessively paranoid client authentication.

---

Updated Apr 5 2019:

because this is a gist from 2011 that people stumble into and maybe you should AES instead of 3DES in the year of our lord 2019.

some other notes:

dvliman

Reader Macros in Common Lisp

This post also appears on lisper.in.

Reader macros are perhaps not as famous as ordinary macros. While macros are a great way to create your own DSL, reader macros provide even greater flexibility by allowing you to create entirely new syntax on top of Lisp.

Paul Graham explains them very well in [On Lisp][] (Chapter 17, Read-Macros):

The three big moments in a Lisp expression's life are read-time, compile-time, and runtime. Functions are in control at runtime. Macros give us a chance to perform transformations on programs at compile-time. ...read-macros... do their work at read-time.

dvliman

    (defun spacemacs/go-run-directory ()
          (interactive)
          (shell-command
                (format "go run %s" (concat (file-name-directory (buffer-file-name)) "*.go"))))