Skip to content

Instantly share code, notes, and snippets.

#!/usr/bin/env python
import argparse
import json
import random
import sys
def get_topic( args, data ):
topic = data['partitions'][0]['topic']
for partition in data['partitions']:
conf = pyspark.SparkConf()
conf.set("spark.sql.tungsten.enabled", "false")
sc = getOrCreateSparkContext(conf)
View classpath

Internet Scale Services Checklist

A checklist for designing and developing internet scale services, inspired by James Hamilton's 2007 paper "On Desgining and Deploying Internet-Scale Services."

Basic tenets

  • Does the design expect failures to happen regularly and handle them gracefully?
  • Have we kept things as simple as possible?
import shutil, errno
def copyanything(src, dst):
shutil.copytree(src, dst)
except OSError as exc: # python >2.5
if exc.errno == errno.ENOTDIR:
shutil.copy(src, dst)
else: raise
anandnalya / css-js-stopwords
Created Jul 29, 2013
List of css and javascript identifiers/properties that can be used as stopwords while indexing
View css-js-stopwords
View decoded.json
"filtered" : {
"query" : {
"range" : {
"c100" : {
"from" : "0",
"to" : "5293983999",
"include_lower" :true,
"include_upper" : true
View elasticsearch.log
[root@ct-0088 ~]# ./bin/elasticsearch
[root@ct-0088 ~]# ./logs/elasticsearchStaging_4hr.log
[2013-07-12 14:07:56,405][INFO ][node ] [CT-0088] {0.90.0.RC2}[3049]: starting ...
[2013-07-12 14:07:56,567][INFO ][transport ] [CT-0088] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/]}
[2013-07-12 14:07:59,598][INFO ][cluster.service ] [CT-0088] new_master [CT-0088][Zukt8ivLRd6LIzeZoriR8A][inet[/]]{max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master)
[2013-07-12 14:07:59,620][INFO ][discovery ] [CT-0088] elasticsearchStaging_4hr/Zukt8ivLRd6LIzeZoriR8A
[2013-07-12 14:07:59,659][INFO ][http ] [CT-0088] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/]}
[2013-07-12 14:07:59,660][INFO ][node ] [CT-0088] {0.90.0.RC2}[3049]: started
[2013-07-12 14:08:01,055][INFO ][gateway.local.state.meta ] [CT-0088] [mlivemas
anandnalya / gist:3089221
Created Jul 11, 2012
Overriding one properties file with another optional properties file in Spring. Properties will be first searched in /path/local/ and if not found, in
View gist:3089221
<bean id="propertyConfigurer" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<property name="ignoreResourceNotFound" value="true" />
View clustering.R
x <- read.table("")
cat( "read", length(x[,1]), "records.\n")
# load clustering library
# get number of clusters from user
n <- as.integer( readline("Enter number of clusters: "))
# run kmeans clustering on the dataset