jbenet

ipfscluster status
runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7fc965b63cc9 m=0

goroutine 0 [idle]:

goroutine 1 [running]:
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_amd64.s:252 fp=0xc420022768 sp=0xc420022760

jbenet

package main

import (
	"encoding/json"
	"errors"
	"flag"
	"fmt"
	"io/ioutil"
	"os"
	"os/user"

jbenet

IPFS Tooling for datasets

Background

We need some tooling for a certain set of use cases around archival and dataset management. This tooling is for fitting how people work with large files and large datasets.

Grounding Assumptions

Basic grounding assumptions here:

• datasets are "large" (From GB to EB in size)

jbenet

Keybase proof

I hereby claim:

• I am jbenet on github.
• I am jbenet (https://keybase.io/jbenet) on keybase.
• I have a public key ASAWhIGxMuoQ0o-F4ZnPOCvXlJsF8U6WHVZXt3juy9H_SAo

To claim this, I am signing this object:

jbenet

Articles discussing cloud services shutting down:

• http://www.archiveteam.org/index.php?title=Deathwatch
  • http://www.archiveteam.org/index.php?title=Deathwatch#Dead_as_a_Doornail
• http://fortune.com/2016/02/12/verizon-closes-two-public-cloud-services/
• http://www.androidpolice.com/2016/02/01/barracuda-is-shutting-down-the-copy-cloud-storage-service-on-may-1st/
• http://lifehacker.com/near-unlimited-cloud-storage-service-copy-com-is-shutti-1756413619
• Even google: http://www.networkworld.com/article/2173255/cloud-computing/cloud-s-worst-case-scenario-what-to-do-if-your-provider-goes-belly-up.html
• http://www.nytimes.com/2016/04/22/technology/apple-no-longer-immune-to-chinas-scrutiny-of-us-tech-firms.html (no longer loads! ironic!)
• http://www.techrepublic.com/blog/the-enterprise-cloud/what-happens-when-your-cloud-provider-goes-out-of-business/

jbenet

Given this Data

{
  "a": "b",
  "c": [
    {},
    {
      "d": 111,

jbenet

What is a speed test?

When picking event spaces and hotels, our group (which includes lots of engineers) requires a good internet connection. For that, we need to check the ping time, the download/upload speeds and capacities, the ISP, and more. This information is required for us to make a good decision, as we cannot get work done if the internet is not of high enough quality.

How to get a speed test?

• Go to http://speedtest.net
• Click "Begin Test"
• Once the test is done, click "Share This Result"
• Copy the link, and paste it to an email back to us.

jbenet

'use strict'

const crypto = require('crypto')
const multihashing = require('multihashing')
// do not use cryptographic hash functions with pht
// use non-cryptographic fast hashes, like murmur3 or xxhash

const nextPowerOf2For = (n) => {
  return Math.ceil(Math.log2(n))
}

jbenet

#
#
# what!?!? this is bizarre!!!
# this has not happenned at all on tests. 
# 
# this reinforces that what we may be encountering is not too many 
# dials, but rather a bug merged in somewhere
#
#

jbenet

Keys have lifetimes. After encrypting a certain number of bytes, keys should be replaced.

What if instead of encrypting large ammounts of our data with keys, we used a method similar to DHE?

protocol

• given symmetric key (or public key) k
• given message m
• generate a one-time symmetric key k2
• encrypt(k1, k2) -> k2c