Skip to content

Instantly share code, notes, and snippets.

Avatar

Julian Gruber juliangruber

View GitHub Profile
View 1-saturn.md

Current Saturn topology, with multi-peer retrieval (TBD) between L1<>L2:

flowchart TB
HttpClient <--> DNS[L1 DNS]
DNS <--> L1'
subgraph L1s[L1s deployed to data centers]
  L1
  L1'
 L1''
View gist:eddb2e9fbc1c96a5f935170c4336d9b5
$ npm run build
> libp2p@0.38.0 build
> aegir build
[08:47:41] tsc [started]
src/circuit/auto-relay.ts:54:80 - error TS2345: Argument of type '(evt: CustomEvent<Connection>) => void' is not assignable to parameter of type 'EventHandler<CustomEvent<Connection>> | null'.
Type '(evt: CustomEvent<Connection>) => void' is not assignable to type 'EventCallback<CustomEvent<Connection>>'.
Types of parameters 'evt' and 'evt' are incompatible.
Type 'CustomEvent<import("/Users/julian/dev/libp2p/js-libp2p/node_modules/@libp2p/interface-connection-manager/node_modules/@libp2p/interface-connection/dist/src/index").Connection>' is not assignable to type 'CustomEvent<import("/Users/julian/dev/libp2p/js-libp2p/node_modules/@libp2p/interface-connection/dist/src/index").Connection>'.
View 00-readme.md

L1<>L2 architecture

Conventions

  • Naming
    • L1: L1 machine
    • l1: Individual l1 worker process
  • The overview graph lists all involved components. In future graphs irrelevant components are omitted for readability, which doesn't mean they have been removed

Problems to solve

View forward-gae-to-papertrail.js
const dgram = require('dgram')
const glossy = require('glossy')
const syslogProducer = new glossy.Produce()
const socket = dgram.createSocket('udp4')
const sendToPapertrail = (chunk, { severity }) => {
const syslog = syslogProducer.produce({
facility: 'user',
severity,
View oops2.js
const fs = require('fs')
const hyperdrive = require('hyperdrive')
const src = `/tmp/${Math.random()}/`
console.log(src)
const storage = require('dat-storage')(src)
const archive = hyperdrive(storage, { latest: true, indexing: true })
archive.writeFile('/dat.json', 'hi', err => {
if (err) throw err
View oops.js
const fs = require('fs')
const src = `/tmp/${Math.random()}`
Dat(src, (err, dat) => {
if (err) throw err
dat.archive.writeFile('/dat.json', 'hi', err => {
if (err) throw err
console.log(fs.readFileSync(`${src}/dat.json`)) // => empty
dat.archive.readFile('/dat.json', (err, data) => {
@juliangruber
juliangruber / 01-results.md
Last active Jan 15, 2018
benchmark: pull-stream vs async-stream
View 01-results.md
$ node bench-01-pull-stream.js
pull3*100000: 1995.798ms
pull_compose*100000: 1968.901ms
pull_chain*100000: 1871.930ms

$ node bench-02-async-stream.js
async3*100000: 3295.666ms
async3*100000: 3145.636ms
async3*100000: 2943.988ms
View async-heap.js
// https://github.com/juliangruber/async-stream
function heap (name, Stream) {
var start = Date.now()
var heap = process.memoryUsage().heapUsed
var a = [], N = 100000
for(var i = 0; i < N; i++)
a.push(Stream())
console.log(name, (process.memoryUsage().heapUsed - heap)/N, (Date.now()-start))
a = null
View gist:fe383095909a9e237c11a5318262920b
const writeToRemote = new PassThrough()
reconnect(async con => {
await sendState(con)
writeToRemote.pipe(con)
}).on('disconnect', () => {
writeToRemote.unpipe()
})
theRestOfYourProgram.pipe(writeToRemote)
View random-access-directory.js
var rad = require('random-access-directory')
var store = rad(procss.env.HOME + '/.dat/')
// create a first store
store = store('hypercore.txt')
store.get(0, ...)
// create another store, based on the first one
// this way modules can pass the store instance