Skip to content

Instantly share code, notes, and snippets.

Dominic Tarr dominictarr

Block or report user

Report or block dominictarr

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
View README.md

Consistent Archive Tree

motivation

Despite it's considerable age, Tape ARchive format (tar) is still in widespread use. This is unfortunate because has as several features that are footguns in a modern context.

Firstly, because it contains timestamps (ctime, mtime) then archiving the same files twice will give a tar file that has a few bytes different, and thus will have a different hash. creating a archive deterministically was not a priority when tar was designed.

@dominictarr
dominictarr / README.md
Last active Feb 15, 2016
expandable buffers pool
View README.md

a pool of expandable buffers

pool noodles, thanks @maxogden

anyone who has played around with getting performant IO in node will tell you to get really great performance you need to avoid unnecessary memory allocations and memory copying.

Working on pull-file I could get about 1gb/s read on a warm cache, but if I just reused the same buffer over and over I could get 2gb/s!

reusing memory is easy to do in a benchmark, but in an a real system when you read data in order to do something with it. maybe you write it to another file or socket, or maybe you encrypt it? maybe you pass it to a multiplexer which adds framing and then writes that to a socket?

View best-pull-streams.md

here are the most useful pull streams modules

combining pull streams

pull-stream

a library of simple functions that will be familiar functional programmers.

pull-cat

View pull-streams.md

there are two fundamental types of streams source and sink. There are two composite types of streams through (aka transform) and duplex A through stream is a sink stream that reads what goes into the source. a duplex stream is a pair of {source, sink} streams.

Source streams have two methods.

readable stream: source

A source (readable stream) is an async function that may be called repeatedly until it returns a terminal state. You must not call the read function until the previous call has returned, except for a call to abort the stream. pull-streams have back pressure, but it implicit instead of sending an explicit back pressure signal. If a source

View profile.js
var fs = require('fs')
var ssbc = require('ssb-client')
var pull = require('pull-stream')
var paramap = require('pull-paramap')
var related = false
//scan
ssbc(function (err, sbot) {
if(err) throw err
@dominictarr
dominictarr / logdb.md
Last active Jul 29, 2016
ideas for a modular database
View logdb.md

intro

we've had great success building modular database stuff on top of leveldb with node, but as I have learnt more about databases it's become apparent to me that the idea of a modular database would be better implemented at a slightly lower level.

Level db provides a sorted key:value store, which, because of the sorted property, many things can be implemented on top of. For example, for replication, or for consistent materialized views, we often need a write ahead log. This can easily be implemented via a batch write to level, and writing the log into a section of the leveldb key space which is treated as append only.

@dominictarr
dominictarr / packages.md
Created Sep 18, 2015
the package manager we need
View packages.md

Git tracks the history of changes, but in a pm we don't really care about the history, at least not when we are installing. This is the properties that I think we do need.

  • security: It must be possible to check the current deps, and know if a single bit has been flipped.
  • replication: It must be possible to share that hash, and someone else would do pm checkout <hash> and get exactly the same code as you.
  • determinism: If I install X, Y then Z, and you install Z, then X then Y, we should get the same dep tree. If those deps have deps that need to be shuffeled around, then it should do that and keep it always optimal.
  • portability: package management is too hard a problem to roll another one for every language. They all suck, more or less. This should be usable for any language, or things that have dependencies.
@dominictarr
dominictarr / .travis.yml
Last active Jan 5, 2016
pull-streams crash course
View .travis.yml
language: node_js
node_js:
- '0.10'
- '0.12'
@dominictarr
dominictarr / readme.md
Created Jun 23, 2015
Decentralized Moderation
View readme.md

decentralized moderation

I am working on a p2p protocol for a decentralized social network. I wanted to build decentralized applications (to avoid having central powers who tend to optimize for the wrong thing -- ad clicks) but it's not exactly simple: what makes decentralized protocols hard is a thing called "the sybil attack".

The sybil attack is when a malicious actor generates a large number of fake accounts, and attempts to disrupt the network. This is also a problem in centralized systems - online voting, email spam, sockpuppet armies, gaming reddit

This kind of attack also appears in the human realm! such as astroturfing

@dominictarr
dominictarr / awkward.txt
Last active Aug 29, 2015
many awkward yet grammatically valid sentences
View awkward.txt
There are so many gramatically valid but awkward english sentences.
There are many awkward but gramatically valid english sentences.
There exist many english sentences which are gramatically valid, but awkward to read.
There are many awkward english sentences which are, never the less, gramatically valid.
The english language contains many awkward-to-read sentences that are within the set of gramatically valid sentences.
The set of gramatically valid english sentences contains many that are quite awkward.
Many gramatically valid english sentences are awkward.
The grammatic vallidity of a given engilsh sentence does not preclude it's awkwardness.
There are many awkward, gramatically valid english sentences.
Many awkward sentences are gramatically valid english.
You can’t perform that action at this time.