frsyuki

My thoughts on MessagePack

Hi. My name is Sadayuki "Sada" Furuhashi. I am the author of the MessagePack serialization format as well as its implementation in C/C++/Ruby.

Recently, MessagePack made it to the front page of Hacker News with this blog entry by Olaf, the creator of the Facebook game ZeroPilot. In the comment thread, there were several criticisms for the blog post as well as MessagePack itself, and I thought this was a good opportunity for me to address the questions and share my thoughts.

My high-level response to the comments

To the best of my understanding, roughly speaking, the criticisms fell into the following two categories.

John07

# required: ffmpeg (e.g. from homebrew), terminal-notifier from https://github.com/alloy/terminal-notifier 
# you can schedule this with launchd to run e.g. weekly

# Specify in seconds how long the script should record (default here is 1 hour).
seconds=3600

# Date format for the recording file name
DATE=`date "+%d-%m-%y_%H-%M"`

# start ffmpeg recording

chanks

Turning PostgreSQL into a queue serving 10,000 jobs per second

RDBMS-based job queues have been criticized recently for being unable to handle heavy loads. And they deserve it, to some extent, because the queries used to safely lock a job have been pretty hairy. SELECT FOR UPDATE followed by an UPDATE works fine at first, but then you add more workers, and each is trying to SELECT FOR UPDATE the same row (and maybe throwing NOWAIT in there, then catching the errors and retrying), and things slow down.

On top of that, they have to actually update the row to mark it as locked, so the rest of your workers are sitting there waiting while one of them propagates its lock to disk (and the disks of however many servers you're replicating to). QueueClassic got some mileage out of the novel idea of randomly picking a row near the front of the queue to lock, but I can't still seem to get more than an an extra few hundred jobs per second out of it under heavy load.

So, many developers have started going straight t

sepehr

#!/usr/bin/perl
 
#
# @file
# Converter tool, from Apache Common Log file to CSV.
#
# All code is released under the GNU General Public License.
# See COPYRIGHT.txt and LICENSE.txt.
#
 

josiahcarlson

'''
rate_limit2.py

Copyright 2014, Josiah Carlson - josiah.carlson@gmail.com
Released under the MIT license

This module intends to show how to perform standard and sliding-window rate
limits as a companion to the two articles posted on Binpress entitled
"Introduction to rate limiting with Redis", parts 1 and 2:

josephspurrier

package main

import (
	"bufio"
	"crypto/aes"
	"crypto/cipher"
	"crypto/rand"
	"fmt"
	"io"
	"io/ioutil"

josephspurrier

package main

import (
	"crypto/aes"
	"crypto/cipher"
	"crypto/rand"
	"fmt"
	"io"
	"io/ioutil"
	"os"

gusennan

#!/bin/sh

PREREQ="dropbear"

prereqs() {
  echo "$PREREQ"
}

case "$1" in
  prereqs)

harlow

Golang Workers / Job Queue

A running example of the code from:

• http://marcio.io/2015/07/handling-1-million-requests-per-minute-with-golang
• http://nesv.github.io/golang/2014/02/25/worker-queues-in-go.html

This gist creates a working example from blog post, and a alternate example using simple worker pool.

TLDR: if you want simple and controlled concurrency use a worker pool.

TheGU

package main

import (
	"encoding/json"
	"fmt"
	"io/ioutil"
	"log"
	"net/http"
	"net/url"
	"os"