Skip to content

Instantly share code, notes, and snippets.

View random-ua's full-sized avatar

Alexander Kostyuchenko random-ua

View GitHub Profile
@chanks
chanks / gist:7585810
Last active June 22, 2024 19:01
Turning PostgreSQL into a queue serving 10,000 jobs per second

Turning PostgreSQL into a queue serving 10,000 jobs per second

RDBMS-based job queues have been criticized recently for being unable to handle heavy loads. And they deserve it, to some extent, because the queries used to safely lock a job have been pretty hairy. SELECT FOR UPDATE followed by an UPDATE works fine at first, but then you add more workers, and each is trying to SELECT FOR UPDATE the same row (and maybe throwing NOWAIT in there, then catching the errors and retrying), and things slow down.

On top of that, they have to actually update the row to mark it as locked, so the rest of your workers are sitting there waiting while one of them propagates its lock to disk (and the disks of however many servers you're replicating to). QueueClassic got some mileage out of the novel idea of randomly picking a row near the front of the queue to lock, but I can't still seem to get more than an an extra few hundred jobs per second out of it under heavy load.

So, many developers have started going straight t

DOWNLOAD_HANDLERS = {
'http': 'myspider.socks5_http.Socks5DownloadHandler',
'https': 'myspider.socks5_http.Socks5DownloadHandler'
}
@pudquick
pudquick / exec-priv.sh
Created August 18, 2014 00:14
An example of using execute-with-privileges verb of security command to bypass GUI for a CLI authorization prompt
# This expects that you're already running as sudo'd / root account
cat <<EOF | security -i
authorize -C adminUserNameHere com.apple.uninstalld.uninstall
execute-with-privileges /usr/sbin/uninstall /Applications/SomeMASapp.app
EOF
# In the example above, if you're already running as root / sudo'd admin,
# you still get a prompt for the password of the account in question.
# We aren't dodging proving we know the password - we're just doing it
'use strict';
var gulp = require('gulp'),
watch = require('gulp-watch'),
prefixer = require('gulp-autoprefixer'),
uglify = require('gulp-uglify'),
sass = require('gulp-sass'),
sourcemaps = require('gulp-sourcemaps'),
rigger = require('gulp-rigger'),
cssmin = require('gulp-minify-css'),
@sbuzonas
sbuzonas / nginx-cors.conf
Created September 8, 2015 15:36
Nginx CORS maps
map $http_origin $allow_origin {
default "";
"~^https?://(?:[^/]*\.)?(stevebuzonas\.(?:com|local))(?::[0-9]+)?$" "$http_origin";
}
map $request_method $cors_method {
default "allowed";
"OPTIONS" "preflight";
}
@markjlorenz
markjlorenz / simple-exec-with-docker-remote-api.sh
Created July 23, 2016 16:05
Using the exec command with the docker API, and capturing output
#! /usr/bin/env bash
# Create the container with a long running process. If PID1 exists before
# we send the exec commands, they will fail because the container is not running
#
CONTAINER_NAME="TEST-`uuidgen`"
curl --silent --unix-socket /var/run/docker.sock "http:/containers/create?name=${CONTAINER_NAME}" -XPOST \
-H "Content-Type: application/json" \
-d '{
"Image": "ruby:latest",
@sanchezzzhak
sanchezzzhak / clickhouse-get-tables-size.sql
Created January 18, 2018 13:43
clickhouse get tables size
SELECT table,
formatReadableSize(sum(bytes)) as size,
min(min_date) as min_date,
max(max_date) as max_date
FROM system.parts
WHERE active
GROUP BY table
@milesbxf
milesbxf / monzo-alertmanager-config.yaml
Last active June 24, 2024 04:45
Monzo's Alertmanager Slack templates
###################################################
##
## Alertmanager YAML configuration for routing.
##
## Will route alerts with a code_owner label to the slack-code-owners receiver
## configured above, but will continue processing them to send to both a
## central Slack channel (slack-monitoring) and PagerDuty receivers
## (pd-warning and pd-critical)
##
@chromko
chromko / monzo-alertmanager-config.yaml
Created August 27, 2019 15:41 — forked from milesbxf/monzo-alertmanager-config.yaml
Monzo's Alertmanager Slack templates
###################################################
##
## Alertmanager YAML configuration for routing.
##
## Will route alerts with a code_owner label to the slack-code-owners receiver
## configured above, but will continue processing them to send to both a
## central Slack channel (slack-monitoring) and PagerDuty receivers
## (pd-warning and pd-critical)
##
@figassis
figassis / README.md
Last active April 6, 2024 15:53
Deploy AWS Storage Gateway on VirtualBox

Update Ubuntu

sudo apt-get update && sudo apt-get dist-upgrade && sudo apt-get autoremove

Install Required Linux Headers

sudo apt-get -y install gcc make linux-headers-$(uname -r) dkms ufw apt-transport-https ca-certificates curl software-properties-common

Add VirtualBox Repository and key