Skip to content

Instantly share code, notes, and snippets.

View ranjanprj's full-sized avatar

ranjanprj ranjanprj

  • Pune
View GitHub Profile
@chanks
chanks / gist:7585810
Last active July 19, 2024 10:16
Turning PostgreSQL into a queue serving 10,000 jobs per second

Turning PostgreSQL into a queue serving 10,000 jobs per second

RDBMS-based job queues have been criticized recently for being unable to handle heavy loads. And they deserve it, to some extent, because the queries used to safely lock a job have been pretty hairy. SELECT FOR UPDATE followed by an UPDATE works fine at first, but then you add more workers, and each is trying to SELECT FOR UPDATE the same row (and maybe throwing NOWAIT in there, then catching the errors and retrying), and things slow down.

On top of that, they have to actually update the row to mark it as locked, so the rest of your workers are sitting there waiting while one of them propagates its lock to disk (and the disks of however many servers you're replicating to). QueueClassic got some mileage out of the novel idea of randomly picking a row near the front of the queue to lock, but I can't still seem to get more than an an extra few hundred jobs per second out of it under heavy load.

So, many developers have started going straight t

@coco98
coco98 / kube-registry.yaml
Created May 2, 2017 16:31
Docker registry on minikube
apiVersion: v1
kind: ReplicationController
metadata:
name: kube-registry-v0
namespace: kube-system
labels:
k8s-app: kube-registry
version: v0
spec:
replicas: 1
@danielepolencic
danielepolencic / README.md
Last active May 20, 2024 18:07
Create 3 nodes Kubernetes cluster locally with Vagrant

3 Virtual Machines Kubernetes cluster

Dependencies

You should install VirtualBox and Vagrant before you start.

Creating the cluster

You should create a Vagrantfile in an empty directory with the following content:

@pahud
pahud / main.workflow
Last active July 24, 2023 08:20
Github Actions with Amazon EKS CI/CD
workflow "Demo workflow" {
on = "push"
resolves = ["SNS Notification"]
}
action "Build Image" {
uses = "actions/docker/cli@c08a5fc9e0286844156fefff2c141072048141f6"
runs = ["/bin/sh", "-c", "docker build -t $IMAGE_URI ."]
env = {
IMAGE_URI = "xxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/github-action-demo:latest"
@jctosta
jctosta / init_postgres_windows.md
Last active April 29, 2024 12:28
A simple guide on initializing the Windows version of Postgres using the zip file instead of msi installer
@pavankjadda
pavankjadda / Sonarqube with Postgres database using docker.md
Last active June 24, 2024 05:30
Setup Sonarqube with Postgres database using docker

Setup Sonarqube with Postgres database using docker

Both Sonarqube and Postgres database should be same network in order communicate with each other. First create docker network using the following commands

Create docker network

Delete existing docker network and create new docker network sonarqube

docker network rm sonarqube_network
docker network create sonarqube_network

Run PostGres DB container

#define _GNU_SOURCE
#include <errno.h>
#include <sched.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <sys/types.h>
@kissgyorgy
kissgyorgy / listen.py
Created September 4, 2020 16:37
How to use PostgreSQL's LISTEN/NOTIFY as a simple message queue with psycopg2 and asyncio
import asyncio
import psycopg2
# dbname should be the same for the notifying process
conn = psycopg2.connect(host="localhost", dbname="example", user="example", password="example")
conn.set_isolation_level(psycopg2.extensions.ISOLATION_LEVEL_AUTOCOMMIT)
cursor = conn.cursor()
cursor.execute(f"LISTEN match_updates;")
@joelonsql
joelonsql / PostgreSQL-EXTENSIONs.md
Last active July 13, 2024 02:57
1000+ PostgreSQL EXTENSIONs

🗺🐘 1000+ PostgreSQL EXTENSIONs

This is a list of URLs to PostgreSQL EXTENSION repos, listed in alphabetical order of parent repo, with active forks listed under each parent.

⭐️ >= 10 stars
⭐️⭐️ >= 100 stars
⭐️⭐️⭐️ >= 1000 stars
Numbers of stars might not be up-to-date.

@AlbertRapp
AlbertRapp / connected_ggiraph.R
Created January 25, 2023 19:39
connected_ggiraph.R
library(dplyr)
library(ggplot2)
library(patchwork)
library(ggiraph)
dat <- gapminder::gapminder |>
janitor::clean_names() |>
mutate(
# ID that is shared for boxplots (this one uses factors, i.e. numbers, as ID instead of continents)