Skip to content

Instantly share code, notes, and snippets.

View 17twenty's full-sized avatar

Nick Glynn 17twenty

View GitHub Profile
@17twenty
17twenty / gist:ca503d3be60330e3b3d07799fcdbdc33
Created November 14, 2019 19:03 — forked from chanks/gist:7585810
Turning PostgreSQL into a queue serving 10,000 jobs per second

Turning PostgreSQL into a queue serving 10,000 jobs per second

RDBMS-based job queues have been criticized recently for being unable to handle heavy loads. And they deserve it, to some extent, because the queries used to safely lock a job have been pretty hairy. SELECT FOR UPDATE followed by an UPDATE works fine at first, but then you add more workers, and each is trying to SELECT FOR UPDATE the same row (and maybe throwing NOWAIT in there, then catching the errors and retrying), and things slow down.

On top of that, they have to actually update the row to mark it as locked, so the rest of your workers are sitting there waiting while one of them propagates its lock to disk (and the disks of however many servers you're replicating to). QueueClassic got some mileage out of the novel idea of randomly picking a row near the front of the queue to lock, but I can't still seem to get more than an an extra few hundred jobs per second out of it under heavy load.

So, many developers have started going straight t

@17twenty
17twenty / WorkingRemote.md
Created July 24, 2019 21:02
Top tips for working remotely

Be as available as possible This is probably the best piece of advice I received from a friend prior to starting my new job. Things move quickly back at the home office, and since you’re not there, there’s a built-in delay for someone wishing to communicate with you.

If it always takes hours for someone to get your ear for just a minute, you’ll build a reputation as someone who’s “never around”–even if only subconsciously–among your coworkers.

As a remote employee, you have to work harder at being available to avoid the impression that you’re never available.

Strive to build a reputation as someone who is quick to respond. Be eager to hear the latest developments on projects so that you’re never out of sync with everyone else in your office.

@17twenty
17twenty / readme.md
Created July 22, 2019 04:08
Using golang-migrate AKA mattes/migrate with Docker, Postgres and creating new schemas.

Using golang-migrate AKA mattes/migrate with Docker, Postgres and creating new schemas.

Prerequisites

We use the golang-migrate tool (formerly known as mattes/migrate) - build it with postgres support.

$ go get -v -u -tags 'postgres' github.com/golang-migrate/migrate/cli
$ migrate -h
Usage: migrate OPTIONS COMMAND [arg...]
@17twenty
17twenty / testing.go
Created August 15, 2018 06:37
Make your life easier by copying these into your Golang tests package
import (
"fmt"
"path/filepath"
"runtime"
"reflect"
"testing"
)
// assert fails the test if the condition is false.
func assert(tb testing.TB, condition bool, msg string, v ...interface{}) {
@17twenty
17twenty / readme.md
Last active August 9, 2018 10:12
Getting Oracle Enterprise in a Container AND installing client libraries

Team member was fighting trying to get a docker container for Oracle setup on his Mac as there's a lot of confusing and conflicting information out there. This was tried and tested!

You will need to aquire access to the container via the standard Oracle data grab here.

$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
@17twenty
17twenty / 1_kubernetes_on_macOS.md
Created June 18, 2018 03:45 — forked from kevin-smets/1_kubernetes_on_macOS.md
Local Kubernetes setup on macOS with minikube on VirtualBox and local Docker registry

Requirements

Minikube requires that VT-x/AMD-v virtualization is enabled in BIOS. To check that this is enabled on OSX / macOS run:

sysctl -a | grep machdep.cpu.features | grep VMX

If there's output, you're good!

Prerequisites

@17twenty
17twenty / sigdemo.go
Created December 8, 2017 00:13
Sig demo
package main
import (
"fmt"
"log"
"os"
"os/signal"
"strconv"
"syscall"
)
@17twenty
17twenty / bash_profile
Last active October 3, 2018 02:20
My bash_profile for MacOS
if [ -f /usr/local/etc/bash_completion ]; then
. /usr/local/etc/bash_completion
fi
PS1='\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
unset color_prompt force_color_prompt
# If this is an xterm set the title to user@host:dir
case "$TERM" in
xterm*|rxvt*)
@17twenty
17twenty / gupdate.sh
Created October 13, 2017 02:10
Update all git repos in a directory
#!/usr/bin/env bash
# Store PWD (could use pushd/popd but meh)
CUR_DIR=$(pwd)
echo -e "\n\033[1mPulling in latest changes for all repositories...\033[0m\n"
# Find all git repositories and update to master
for i in $(find . -name ".git" | cut -c 3-); do
echo "";
@17twenty
17twenty / demo.go
Created September 19, 2017 05:27
Simple demo of setting up the usage preamble
package main
import (
"flag"
"fmt"
"os"
"path"
)
var (