Skip to content

Instantly share code, notes, and snippets.

Avatar
🗿
.

Bob Poekert bobpoekert

🗿
.
View GitHub Profile
@bobpoekert
bobpoekert / gist:685bce859831eabfdc9e
Last active Aug 29, 2015
Transfer-Encoding: Chuxed
View gist:685bce859831eabfdc9e

In HTTP/1.1, Transfer-Encoding: chunked allows servers to stream responses to clients in pieces without knowing the length in advance, which is useful in, for example, chat and live streaming applications.

HTTP/1.1 also allows request pipelining, which lets clients do multiple get requests in parallel, saving the latency overhead of opening a new TCP socket for each one.

It's not currently possible to use both of these things together, because chunks in HTTP chunked encoding don't have any information in them about which stream they came from. Chunks are <length field>\r\n<data>.

But if a new transfer-encoding were added that had stream ids, you could do multiple requsts in parallel even if the responses were streamed. Let's call this encoding Transfer-Encoding: chuxed (for "chunked, multiplexed").

There are two differences between chuxed and chunked:

View sentiments.clj
(def nlp
(let [props (new java.util.Properties)]
(.setProperty props "annotators" "tokenize,ssplit,parse,sentiment")
(new edu.stanford.nlp.pipeline.StanfordCoreNLP props)))
(defn find-sentiment
"Determines the sentiment of each sentence in a given glob of text. Results in a collection of integers ranging from [0-4]: where 0 is 'Very negative', 2 is 'neutral', and 4 is 'Very positive'"
[^String glob]
(let [main-sentiment 0
longest 0]
View gist:f4613bde4fabae5b50bb
  • Doing a query with a has_parent filter when a parent-child relation references a mapping that doesn't exist returns a NullPointerException (instead of a more informative error)
  • Adding a port number to a unicast host in elasticsearch.yml causes that node to recieve invalid (ie unparseable) http requests
  • Missing a newline in a bulk insert request caused subsequent queries on that index to return invalid json
  • Doing a delete by query on an index that removed a significant number of documents caused refresh requests on that index to return NullPointerExceptions
  • Shards moving between nodes for no apparent reason
  • Shards becoming unassigned for no apparent reason
  • Shards becoming unassigned even when all of the shards in the cluster had been routed manually and shard allocation had been disabled
  • Shards losing all of their documents if a write is performed while it's unavailable
View partial.py
class partial(object):
def __init__(self, fn, *args):
self.fn = fn
self.args = args
def __call__(self, *extra_args):
return self.fn(*(self.args + extra_args))
@bobpoekert
bobpoekert / sha256.c
Last active Aug 29, 2015
NaCl's sha256 implementation pulled out into a standalone executable, buildable from a single C file
View sha256.c
#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <sys/mman.h> /* so we don't have to deal with buffering and fread()-ing */
#include <sys/types.h>
#include <unistd.h>
#include <sys/stat.h>
#include <string.h>
/* http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/utopic/nacl/utopic/view/head:/crypto_hashblocks/sha256/inplace/blocks.c */
View postgres_webskale.md

Postgres Webskale(tm)

What would it take to fold the ad-hoc sharding that people do with postgres into postgres? Or, what would it take to make postgres scale like riak and cassandra?

  • A shard routing server kind of like mongos. This could be implemented as a foreign data wrapper that holds a connection pool and routes queries to shards based on qualifiers it has with respect to the column being sharded on. It could get fancy by supporting scatter-gather aggregation and joins but that probably isn't necessary because by the point you need to shard doing aggregation and joins on the production database is already too dangerous (I assert). Updates to the hash ring happen over paxos (or zab or raft or whatever).
  • A server responsible for annointing new users and assigning them to shards. This could happen at random or it could be aware of geographic locality and things.
  • The multicast views described below
@bobpoekert
bobpoekert / gist:648354
Created Oct 27, 2010
Justin.tv Javascript Api Example
View gist:648354
player = jtv_api.new_player(document.getElementById('video_container'), {
width:'100%',
height:'100%',
custom:true,
consumer_key:'yjw6J88EzQ2kCj8NhBXA',
events : {
stream_lost : play_random,
video_not_found : play_random,
}
});
View gist:673274
var tail = function(fname, onData) {
fs.open(fname, 'a+', function(err, fd) {
puts('fd:'+fd);
if (err) {
puts('error!: '+err);
setTimeout(function(){
tail(fname, onData);
}, 10);
} else {
puts('allocating buffer');
View gist:673587
var tail = function(fname, onData, ender) {
fs.open(fname, 'a+', function(err, fd) {
if (err) {
puts('tail error: '+err);
if (ender(fd, 0) !== false) {
setTimeout(function(){
tail(fname, onData);
}, 10);
}
} else {
View gist:673518
var tail = function(fname, onData, ender) {
fs.open(fname, 'a+', function(err, fd) {
if (err) {
puts('error!: '+err);
if (ender(fd, 0) !== false) {
setTimeout(function(){
tail(fname, onData);
}, 10);
}
} else {