Skip to content

Instantly share code, notes, and snippets.

View juliangruber's full-sized avatar

Julian Gruber juliangruber

View GitHub Profile
@juliangruber
juliangruber / fw.sh
Created March 2, 2023 15:58
l1-firewall
#!/usr/bin/env bash
ipv4ranges=(
10.0.0.0/8
100.64.0.0/10
169.254.0.0/16
172.16.0.0/12
192.0.0.0/24
192.0.2.0/24
192.168.0.0/16

Current Saturn topology, with multi-peer retrieval (TBD) between L1<>L2:

flowchart TB
HttpClient <--> DNS[L1 DNS]
DNS <--> L1'
subgraph L1s[L1s deployed to data centers]
  L1
  L1'
 L1''
$ npm run build
> libp2p@0.38.0 build
> aegir build
[08:47:41] tsc [started]
src/circuit/auto-relay.ts:54:80 - error TS2345: Argument of type '(evt: CustomEvent<Connection>) => void' is not assignable to parameter of type 'EventHandler<CustomEvent<Connection>> | null'.
Type '(evt: CustomEvent<Connection>) => void' is not assignable to type 'EventCallback<CustomEvent<Connection>>'.
Types of parameters 'evt' and 'evt' are incompatible.
Type 'CustomEvent<import("/Users/julian/dev/libp2p/js-libp2p/node_modules/@libp2p/interface-connection-manager/node_modules/@libp2p/interface-connection/dist/src/index").Connection>' is not assignable to type 'CustomEvent<import("/Users/julian/dev/libp2p/js-libp2p/node_modules/@libp2p/interface-connection/dist/src/index").Connection>'.

L1<>L2 architecture

Conventions

  • Naming
    • L1: L1 machine
    • l1: Individual l1 worker process
  • The overview graph lists all involved components. In future graphs irrelevant components are omitted for readability, which doesn't mean they have been removed

Problems to solve

const dgram = require('node:dgram')
const util = require('node:util')
const glossy = require('glossy')
const syslogProducer = new glossy.Produce()
const socket = dgram.createSocket('udp4')
const streams = {
stdout: process.stdout.write.bind(process.stdout),
stderr: process.stderr.write.bind(process.stderr)
const fs = require('fs')
const hyperdrive = require('hyperdrive')
const src = `/tmp/${Math.random()}/`
console.log(src)
const storage = require('dat-storage')(src)
const archive = hyperdrive(storage, { latest: true, indexing: true })
archive.writeFile('/dat.json', 'hi', err => {
if (err) throw err
const fs = require('fs')
const src = `/tmp/${Math.random()}`
Dat(src, (err, dat) => {
if (err) throw err
dat.archive.writeFile('/dat.json', 'hi', err => {
if (err) throw err
console.log(fs.readFileSync(`${src}/dat.json`)) // => empty
dat.archive.readFile('/dat.json', (err, data) => {
@juliangruber
juliangruber / 01-results.md
Last active January 15, 2018 11:16
benchmark: pull-stream vs async-stream
$ node bench-01-pull-stream.js
pull3*100000: 1995.798ms
pull_compose*100000: 1968.901ms
pull_chain*100000: 1871.930ms

$ node bench-02-async-stream.js
async3*100000: 3295.666ms
async3*100000: 3145.636ms
async3*100000: 2943.988ms
// https://github.com/juliangruber/async-stream
function heap (name, Stream) {
var start = Date.now()
var heap = process.memoryUsage().heapUsed
var a = [], N = 100000
for(var i = 0; i < N; i++)
a.push(Stream())
console.log(name, (process.memoryUsage().heapUsed - heap)/N, (Date.now()-start))
a = null
const writeToRemote = new PassThrough()
reconnect(async con => {
await sendState(con)
writeToRemote.pipe(con)
}).on('disconnect', () => {
writeToRemote.unpipe()
})
theRestOfYourProgram.pipe(writeToRemote)