Skip to content

Instantly share code, notes, and snippets.

View lukaszx0's full-sized avatar

Lukasz Strzalkowski lukaszx0

  • Column
View GitHub Profile
@Hunsin
Hunsin / router.go
Last active August 26, 2023 16:23
The package wraps julienschmidt's httprouter, making it support functions such as middlewares, sub/group routing with same prefix. Written in Go (Golang).
package router
import (
"context"
"net/http"
gpath "path"
"github.com/julienschmidt/httprouter"
)

Scaling your API with rate limiters

The following are examples of the four types rate limiters discussed in the accompanying blog post. In the examples below I've used pseudocode-like Ruby, so if you're unfamiliar with Ruby you should be able to easily translate this approach to other languages. Complete examples in Ruby are also provided later in this gist.

In most cases you'll want all these examples to be classes, but I've used simple functions here to keep the code samples brief.

Request rate limiter

This uses a basic token bucket algorithm and relies on the fact that Redis scripts execute atomically. No other operations can run between fetching the count and writing the new count.

Redis in Production at Smyte

To be clear we continue to run many Redis services in our production environment. It’s a great tool for prototyping and small workloads. For our use case however, we believe the cost and complexity of our setup justifies urgently finding alternate solutions.

  • Each of our Redis servers are clearly numbered with a current leader in one availability zone, and a follower in another zone.
  • The servers run ~16 different individual Redis processes. This helps us utilize CPUs (as Redis is single-threaded) but it also means we only need an extra 1/16th memory in order to safely perform a BGSAVE (due to copy-on-write), though in practice it’s closer to 1/8 because it’s not always evenly balanced.
  • Our leaders do not every run BGSAVE unless we’re bringing up a new slave which is carefully done manually. Since issues with the slave should not affect the leader and new slave connections might trigger an unsafe BGSAVE on the leader, slave Redis processes are set to not automatically rest
Priority Queues
═══════════════
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
INSERT DEL-MIN MIN DEC-KEY DELETE MERGE
binary log n log n 1 log n log n n
binomial 1 log n 1 log n log n log n
Fibonacci 1 log n† 1 1† log n† log n
Pairing 1† log n† 1† 1† log n† 1†
Brodal-Okasaki 1 log n 1 1 log n 1
@mislav
mislav / netflix.sh
Last active February 21, 2024 16:56
Watch Netflix as if you were in the US by proxying DNS through a DigitalOcean instance.
#!/bin/bash
set -e
droplet=netflix
interface=Wi-Fi
us_regions=( nyc1 nyc2 nyc3 )
random_region() {
echo ${us_regions[RANDOM % ${#us_regions[@]}]}
}
syntax = "proto2";
// Copyright 2012, Square Inc.
// This file provides the default Sake service exported on all Sake services. Useful for debugging
// remote hosts from the commandline or other tools.
package squareup.sake;
option java_package = "com.squareup.protos.sake";
option java_generic_services = true;
@louiscryan
louiscryan / README
Created December 14, 2015 21:51
GRPC - Curl, HTTP/1.1 & nghttp misc
# Build the GRPC Java server which is used for interop testing.
# Interface defined here
# https://github.com/grpc/grpc-java/blob/master/interop-testing/src/main/proto/io/grpc/testing/integration/test.proto
./gradlew :grpc-interop-testing:installDist
# Run it with TLS turned on so ALPN is enabled. Uses a dummy cert
./interop-testing/build/install/grpc-interop-testing/bin/test-server --port=8080 --use_tls=true
# Run the awesome nghttp2 as a proxy in front of it.
# See https://nghttp2.org/documentation/package_README.html#building-from-git
#!/usr/bin/perl
# This filter changes all words to Title Caps, and attempts to be clever
# about *un*capitalizing small words like a/an/the in the input.
#
# The list of "small words" which are not capped comes from
# the New York Times Manual of Style, plus 'vs' and 'v'.
#
# 10 May 2008
# Original version by John Gruber:
@sddamico
sddamico / LICENSE
Last active April 29, 2018 04:26
Exponential Backoff Transformer
Copyright (c) 2016 Stephen D'Amico
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE
# speed up pluck
class ActiveRecord::Relation
class RailsDateTimeDecoder < PG::SimpleDecoder
def decode(string, tuple=nil, field=nil)
if Rails.version >= "4.2.0"
@caster ||= ActiveRecord::Type::DateTime.new
@caster.type_cast_from_database(string)
else