frutik

System Design Cheatsheet

Picking the right architecture = Picking the right battles + Managing trade-offs

Basic Steps

1. Clarify and agree on the scope of the system

• User cases (description of sequences of events that, taken together, lead to a system doing something useful)
  • Who is going to use it?
  • How are they going to use it?

frutik

function permutations(list)
{
	// Empty list has one permutation
	if (list.length == 0)
		return [[]];
		
		
	var result = [];
	
	for (var i=0; i<list.length; i++)

frutik

/*jslint sloppy:true, white:true, vars:true, plusplus:true */

var permutation = function (collection){
    var current,
        subarray,
        result = [],
        currentArray = [],
        newResultArray = [];

    if (collection.length){

frutik

Thread Pools

Thread pools on the JVM should usually be divided into the following three categories:

1. CPU-bound
2. Blocking IO
3. Non-blocking IO polling

Each of these categories has a different optimal configuration and usage pattern.

frutik

Set the default size for new Docker for Mac disk images

If you are getting the error: No space left on device

Configuring the qcow2 size cap is possible in the current versions:

# my disk is currently 64GiB
$ /Applications/Docker.app/Contents/MacOS/qemu-img info ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
image: /Users/djs/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2

frutik

A warning occurred (42 apples)
An error occurred

frutik

1. General Background and Overview

• Probabilistic Data Structures for Web Analytics and Data Mining : A great overview of the space of probabilistic data structures and how they are used in approximation algorithm implementation.
• Models and Issues in Data Stream Systems
• Philippe Flajolet’s contribution to streaming algorithms : A presentation by Jérémie Lumbroso that visits some of the hostorical perspectives and how it all began with Flajolet
• Approximate Frequency Counts over Data Streams by Gurmeet Singh Manku & Rajeev Motwani : One of the early papers on the subject.
• [Methods for Finding Frequent Items in Data Streams](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.187.9800&rep

frutik

Next Steps

• Measure time spend on index, flush, refresh, merge, query, etc. (TD - done)
• Take hot threads snapshots under read+write, read-only, write-only (TD - done)
• Adjust refresh time to 10s (from 1s) and see how load changes (TD)
• Measure time of a rolling restart doing disable_flush and disable_recovery (TD)
• Specify routing on query -- make it choose same node for each shard each time (MD)
• GC new generation size (TD)
• Warmers

• measure before/after of client query time with and without warmers (MD)

frutik

# cat /etc/redhat-release
CentOS release 6.5 (Final)

golang from EPEL:

# rpm -qi golang
Name        : golang                       Relocations: (not relocatable)
Version     : 1.1.2                             Vendor: Fedora Project
Release     : 4.el6                         Build Date: Fri 20 Sep 2013 16:21:49 BST
Install Date: Thu 09 Jan 2014 15:14:13 GMT      Build Host: buildvm-17.phx2.fedoraproject.org

frutik

# Run me with:
#
#     $ nginx -p /path/to/this/file/ -c nginx.conf
#
# All requests are then routed to authenticated user's index, so
#
#    GET http://user:password@localhost:8080/_search?q=*
#
# is rewritten to:
#