Skip to content

Instantly share code, notes, and snippets.

(ns shell
"Run OS commands"
(:import [java.io BufferedReader InputStreamReader]))
(defn run
"Executes the given command"
[cmd]
(.. Runtime getRuntime (exec (str cmd))))
(defn pipe-to
@viksit
viksit / uuid.js
Created March 22, 2011 18:06
Javscript generate UUID/GUID
'xxyxxyxyx'.replace(/[xy]/g, function(c) {
var r = Math.random()*16|0, v = c == 'x' ? r : (r&0x3|0x8);
return v.toString(16);
});
@paulkoerbitz
paulkoerbitz / tutorial1.clj
Created October 18, 2011 11:35
Simple webscrape with enlive
(ns tutorial.scrape1
(:require [net.cgrand.enlive-html :as html]))
(def *url* "http://www.belex.rs/trgovanje/prospekt/VZAS/show")
(defn get-page
"Gets the html page from passed url"
[url]
(html/html-resource (java.net.URL. url)))
@methane
methane / gist:2185380
Created March 24, 2012 17:28
Tornado Example: Delegating an blocking task to a worker thread pool from an asynchronous request handler
from time import sleep
from tornado.httpserver import HTTPServer
from tornado.ioloop import IOLoop
from tornado.web import Application, asynchronous, RequestHandler
from multiprocessing.pool import ThreadPool
_workers = ThreadPool(10)
def run_background(func, callback, args=(), kwds={}):
def _callback(result):
@jboner
jboner / latency.txt
Last active June 1, 2024 18:05
Latency Numbers Every Programmer Should Know
Latency Comparison Numbers (~2012)
----------------------------------
L1 cache reference 0.5 ns
Branch mispredict 5 ns
L2 cache reference 7 ns 14x L1 cache
Mutex lock/unlock 25 ns
Main memory reference 100 ns 20x L2 cache, 200x L1 cache
Compress 1K bytes with Zippy 3,000 ns 3 us
Send 1K bytes over 1 Gbps network 10,000 ns 10 us
Read 4K randomly from SSD* 150,000 ns 150 us ~1GB/sec SSD
@jgeurts
jgeurts / install-graphite-ubuntu-12.04.sh
Created July 14, 2012 16:36 — forked from tkoeppen/install-graphite-ubuntu-10.04.sh
Install Graphite 0.9.10 on Ubuntu 12.04
####################################
# BASIC REQUIREMENTS
# http://graphite.wikidot.com/installation
# http://geek.michaelgrace.org/2011/09/how-to-install-graphite-on-ubuntu/
# Last tested & updated 10/13/2011
####################################
cd
sudo apt-get update
sudo apt-get upgrade
@smougenot
smougenot / A_Logstash.conf
Created July 26, 2012 13:59
Logstash Multiline Filter for Java Stacktrace (tested on field)
# stacktrace java as one message
multiline {
#type => "all" # no type means for all inputs
pattern => "(^.+Exception: .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"
what => "previous"
}
@brandonb927
brandonb927 / osx-for-hackers.sh
Last active May 29, 2024 14:52
OSX for Hackers: Yosemite/El Capitan Edition. This script tries not to be *too* opinionated and any major changes to your system require a prompt. You've been warned.
#!/bin/sh
###
# SOME COMMANDS WILL NOT WORK ON macOS (Sierra or newer)
# For Sierra or newer, see https://github.com/mathiasbynens/dotfiles/blob/master/.macos
###
# Alot of these configs have been taken from the various places
# on the web, most from here
# https://github.com/mathiasbynens/dotfiles/blob/5b3c8418ed42d93af2e647dc9d122f25cc034871/.osx
@samuelclay
samuelclay / radix_trie.clj
Created January 22, 2013 17:51
A Radix Trie (aka PATRICIA trie) implemented in Clojure. For my JavaScript implementation, see this interactive JS fiddle: http://jsfiddle.net/jCYAw/
(ns radix
(:require [clojure.string :as string]))
(use 'clojure.java.io)
(use 'clojure.pprint)
(println "Loading names... ")
(time (def names
(with-open
[rdr (reader
"/usr/share/dict/ProperNames")]

Setting up Flume NG, listening to syslog over UDP, with an S3 Sink

My goal was to set up Flume on my web instances, and write all events into s3, so I could easily use other tools like Amazon Elastic Map Reduce, and Amazon Red Shift.

I didn't want to have to deal with log rotation myself, so I setup Flume to read from a syslog UDP source. In this case, Flume NG acts as a syslog server, so as long as Flume is running, my web application can simply write to it in syslog format on the specified port. Most languages have plugins for this.

At the time of this writing, I've been able to get Flume NG up and running on 3 ec2 instances, and all writing to the same bucket.

Install Flume NG on instances