valyala

Optimizing postgresql table for more than 100K inserts per second

• Create UNLOGGED table. This reduces the amount of data written to persistent storage by up to 2x.
• Set WITH (autovacuum_enabled=false) on the table. This saves CPU time and IO bandwidth on useless vacuuming of the table (since we never DELETE or UPDATE the table).
• Insert rows with COPY FROM STDIN. This is the fastest possible approach to insert rows into table.
• Minimize the number of indexes in the table, since they slow down inserts. Usually an index on time timestamp with time zone is enough.
• Add synchronous_commit = off to postgresql.conf.
• Use table inheritance for fast removal of old data:

ptarjan

Scaling your API with rate limiters

The following are examples of the four types rate limiters discussed in the accompanying blog post. In the examples below I've used pseudocode-like Ruby, so if you're unfamiliar with Ruby you should be able to easily translate this approach to other languages. Complete examples in Ruby are also provided later in this gist.

In most cases you'll want all these examples to be classes, but I've used simple functions here to keep the code samples brief.

Request rate limiter

This uses a basic token bucket algorithm and relies on the fact that Redis scripts execute atomically. No other operations can run between fetching the count and writing the new count.

grampelberg

# zookeeper connection string
{{- define "zookeepers" -}}{{ include "_zookeepers" . | trim | replace "\n\n" "," | nospace }}{{- end -}}

{{- define "_zookeepers" -}}
{{ $values := . }}
{{ range $i, $e := until (int .Values.zookeeper.replicaCount) }}
  {{ template "zookeeperName" $values }}-{{ $i }}.{{ template "headlessZookeeper" $values }}
{{ end }}
{{- end -}}

grampelberg

# Convert resources.requests.memory into JVM heap sizes.
{{- define "memoryRequest" -}}{{ include "_memoryRequest" . | trim | trimSuffix "i" | lower }}{{- end -}}

{{- define "_memoryRequest" -}}
{{- $default := "256m" -}}
{{ if .Values.resources }}
  {{ if (hasKey .Values.resources "requests") }}
    {{ if (hasKey .Values.resources.requests "memory") }}
      {{ .Values.resources.requests.memory }}
    {{ else }}

bobrik

Overly aggressive CFS

We're looking at CPU bandwidth control via CFS:

• https://www.kernel.org/doc/Documentation/scheduler/sched-bwc.txt

Reproduction program

Program does number of iterations, in each iteration we burn CPU in small chunks until we get 5ms of real time spent. On each iteration we also print how much

enricofoltran

package main

import (
	"context"
	"flag"
	"fmt"
	"log"
	"net/http"
	"os"
	"os/signal"

AndrewJakubowicz

[package]
name = "snake_game"
version = "0.1.0"
authors = ["youcodethings <spyr1014@gmail.com>"]

[dependencies]
piston = "0.36.0"
piston2d-graphics = "0.26.0"
pistoncore-glutin_window = "0.45.0"
piston2d-opengl_graphics = "0.52.0"

keyan

Recently I have had to deal with Salt configuration. I am doing product work so I didn't want to focus too much on deeply grasping Salt, but instead knowing the terminology equivalents to Chef/Puppet which I am comfortable with.

Salt	Chef	Puppet
state	resource	resource
states	recipes	manifests
formula*	cookbook	module
pillars	databags	hiera
grains	ohai	facter

HenningTimm

Memory profiling Rust code with heaptrack in 2019

I recently ran into a classic case of "our code is using way more memory than it should". So I took my first dive into memory profiling Rust code. I read several posts about this, including the following

• KodrAus' gist on profiling rust
• This reddit thread
• Brendan D. Gregg's post about memory flamegraphs
• Bradlee Speice's A Case Study in Heaptrack

qoega

CREATE TABLE payment_types (
    payment_type UInt8,
    description String
)
ENGINE = MergeTree
ORDER BY payment_type;

INSERT INTO payment_types(payment_type, description) VALUES
(1, 'credit card'),
(2, 'cash'),