Skip to content

Instantly share code, notes, and snippets.

View ggreg's full-sized avatar

Greg Leclercq ggreg

View GitHub Profile
### Keybase proof
I hereby claim:
* I am ggreg on github.
* I am ggreg (https://keybase.io/ggreg) on keybase.
* I have a public key ASDc3kdDuqjWOdn2YXUta7mG_vHfNujZcBbyN4n_WlV0Iwo
To claim this, I am signing this object:
@ggreg
ggreg / spy_deletes.lua
Created August 8, 2015 21:25
displays processes that delete a file
description = "displays processes that delete a file"
short_description = "spy file deletion"
category = "files"
args =
{
{
name = "path",
description = "the path of the file to monitor",
argtype = "string"
@ggreg
ggreg / pickling_example.scala
Last active August 29, 2015 14:00
An example of using Scala's pickling library to store an instance of a non-trivial (nested case classes) type in a byte array and load it back. It comes from a simple prototype (yes the byte array is copied to deserialize the data) that stores some data in zookeeper.
import scala.pickling._
import json._
import org.apache.curator.framework.{ CuratorFrameworkFactory }
import org.apache.curator.retry.{ ExponentialBackoffRetry }
import org.apache.zookeeper.CreateMode
abstract class State
case class Scheduled extends State
@ggreg
ggreg / test.md
Created February 17, 2014 16:30
test

TEST

@ggreg
ggreg / quadcopter.txt
Created December 20, 2013 17:17
DIY quadcopter lists
- http://www.intorobotics.com/5-best-examples-of-how-to-build-a-diy-quadcopter/
- http://www.instructables.com/id/Sturdy-Quadcopter-Build/
- http://yameb.blogspot.ro/search/label/Quadrotors
- https://lalegiondesquadri.wordpress.com/2013/09/27/construire-un-multi-cest-facile/
- http://wiki.openpilot.org/display/Doc/Basic+QuadCopter
- http://smaccmpilot.org
- http://www.helimag.com/multirotors-complets/34639-turnigy-h-l-2.html
If you want, I can try and help with pointers as to how to improve the indexing speed you get. Its quite easy to really increase it by using some simple guidelines, for example:
- Use create in the index API (assuming you can).
- Relax the real time aspect from 1 second to something a bit higher (index.engine.robin.refresh_interval).
- Increase the indexing buffer size (indices.memory.index_buffer_size), it defaults to the value 10% which is 10% of the heap.
- Increase the number of dirty operations that trigger automatic flush (so the translog won't get really big, even though its FS based) by setting index.translog.flush_threshold (defaults to 5000).
- Increase the memory allocated to elasticsearch node. By default its 1g.
- Start with a lower replica count (even 0), and then once the bulk loading is done, increate it to the value you want it to be using the update_settings API. This will improve things as possibly less shards will be allocated to each machine.
- Increase the number of machines you have so
@ggreg
ggreg / slide24_numbers.txt
Created December 4, 2013 13:38
From Designs, Lessons and Advice from Building Large Distributed Systems http://www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf
Numbers Everyone Should Know
L1 cache reference 0.5 ns
Branch mispredict 5 ns
L2 cache reference 7 ns
Mutex lock/unlock 25 ns
Main memory reference 100 ns
Compress 1K bytes with Zippy 3,000 ns
Send 2K bytes over 1 Gbps network 20,000 ns
Read 1 MB sequentially from memory 250,000 ns
Round trip within same datacenter 500,000 ns
infra1/development
staging
production
webservers.yml
dbservers.yml
backend_servers.yml
group_vars/
infra2/development
staging
production
@ggreg
ggreg / main.yml
Created July 30, 2013 16:27
ec2 + route53 with ansible
- name: provision
local_action: ec2 key_name=greg group="{{default_security_group}}" instance_type="{{instance_type}}" image="{{image_id}}" instance_tags='{{instance_tags | to_json}}' monitoring=yes wait=yes
register: ec2
- name: set_dns_record
route53: >
command=create
zone=botify.com
type=A
value={{item.public_ip}}
with_items: ec2.instances
@ggreg
ggreg / frp_intro.hs
Created April 30, 2013 10:58
Introducing Functional Reactive Programming (FRP)
type Behavior a = Time -> a
type Event a = [(Time,a)]