pkuphy

Turning PostgreSQL into a queue serving 10,000 jobs per second

RDBMS-based job queues have been criticized recently for being unable to handle heavy loads. And they deserve it, to some extent, because the queries used to safely lock a job have been pretty hairy. SELECT FOR UPDATE followed by an UPDATE works fine at first, but then you add more workers, and each is trying to SELECT FOR UPDATE the same row (and maybe throwing NOWAIT in there, then catching the errors and retrying), and things slow down.

On top of that, they have to actually update the row to mark it as locked, so the rest of your workers are sitting there waiting while one of them propagates its lock to disk (and the disks of however many servers you're replicating to). QueueClassic got some mileage out of the novel idea of randomly picking a row near the front of the queue to lock, but I can't still seem to get more than an an extra few hundred jobs per second out of it under heavy load.

So, many developers have started going straight t

pkuphy

Keybase proof

I hereby claim:

• I am pkuphy on github.
• I am caigen1337 (https://keybase.io/caigen1337) on keybase.
• I have a public key ASBfQLIxQeVd6Nwi79AZ8zO_o3xfrXZjWirPqIrRcmKHpwo

To claim this, I am signing this object:

pkuphy

# Do the first 6 steps only once.
1. pip install --user alembic
2. bash: ``cd {{my_project}} && alembic init alembic``
3. bash: ``text_editor {{my_project}}/alembic.ini``
4. Change: "sqlalchemy.url = postgres://{{username}}:{{password}}@{{address}}/{{db_name}}"
5. bash: ``text_editor {{my_project}}/alembic/env.py``
6. Now, import your metadata/db object from your app.:

    # {{my_project}}/{{my_project_dir}}/app.py
    

pkuphy

// npm install eosjs-ecc

const crypto = require('crypto');
const assert = require('assert');
const eosecc = require('eosjs-ecc');

const publicKey = 'EOS5bmVmmRRRcdNERtn23Sc4H32e7rGLLXS8hvajUZVfP2PDicBCH';

const hash = (s) => {
  const hash256 = crypto.createHash('sha256');

pkuphy

View Click

Instead of the verbose setOnClickListener:

RxView.clicks(submitButton).subscribe(o -> log("submit button clicked!"));

Filter even numbers

Observable
    .just(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

pkuphy

server {
  listen 80;
  server_name a.fasdasdfc.com;

  location / {
    proxy_pass http://localhost:8888;
    proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;
    proxy_set_header        Host            $host;
    proxy_set_header        X-Real-IP       $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;

pkuphy

# encoding: utf-8
from flask import Flask, request

app = Flask(__name__)


@app.route('/config')
def config():
    echostr = request.args.get('echostr')
    return echostr

pkuphy

// DON'T FORGET TO IMPORT proxy.conf TOO

[General]
loglevel = notify
skip-proxy = 192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12,127.0.0.0/24
bypass-tun = 192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12,127.0.0.0/24

// DNS OVERRIDE, REMOVE # IF YOU NEED
# dns-server = 223.6.6.6,223.5.5.5,114.114.114.114,114.114.115.115

pkuphy

# First:
lsbom -f -l -s -pf /var/db/receipts/org.nodejs.pkg.bom | while read f; do  sudo rm /usr/local/${f}; done
sudo rm -rf /usr/local/lib/node /usr/local/lib/node_modules /var/db/receipts/org.nodejs.*

# To recap, the best way (I've found) to completely uninstall node + npm is to do the following:

#go to /usr/local/lib and delete any node and node_modules
cd /usr/local/lib
sudo rm -rf node*

pkuphy

#!flask/bin/python
from flask import Flask, jsonify, abort, request, make_response, url_for
from flask.ext.httpauth import HTTPBasicAuth

app = Flask(__name__, static_url_path = "")
auth = HTTPBasicAuth()

@auth.get_password
def get_password(username):
    if username == 'miguel':