Skip to content

Instantly share code, notes, and snippets.

View calmh's full-sized avatar
:bowtie:
I may be slow to respond.

Jakob Borg calmh

:bowtie:
I may be slow to respond.
View GitHub Profile
[root@anto ~]# vmadm get 24355762-5896-4056-9ba6-cde7530bf1de
{
"zonename": "24355762-5896-4056-9ba6-cde7530bf1de",
"autoboot": true,
"brand": "joyent",
"limit_priv": "default",
"v": 1,
"create_timestamp": "2015-03-12T13:49:04.788Z",
"image_uuid": "c02a2044-c1bd-11e4-bd8c-dfc1db8b0182",
"cpu_shares": 100,
@calmh
calmh / db-benchmarks.md
Last active August 29, 2015 14:21
bolt benchmarking

old (leveldb) vs new (bolt):

jb@syno:~/src/github.com/syncthing/syncthing/internal/db (leveldb) $ go test -bench . -benchmem > ~/a
jb@syno:~/src/github.com/syncthing/syncthing/internal/db (leveldb) $ git checkout boltv2
Switched to branch 'boltv2'
jb@syno:~/src/github.com/syncthing/syncthing/internal/db (boltv2) $ go test -bench . -benchmem > ~/b
jb@syno:~/src/github.com/syncthing/syncthing/internal/db (boltv2) $ benchcmp ~/a ~/b
benchmark                    old ns/op     new ns/op     delta
Benchmark10kReplace          68852446      24119329      -64.97%

Goals

Syncthing devices that store data in encrypted format. Such a device can participate fully in a cluster, but the information on the device is not directly usable.

Assumptions and Definitions

  • There exists a secret key per folder, known only by devices that can access the unencrypted data. We call these "secure devices".

  • Devices that do not know the secret key can only access encrypted data. We call these "insecure devices".

package db
import (
"os"
"testing"
"github.com/boltdb/bolt"
)
func TestBoltTransactions(t *testing.T) {

Problem

We do two pass scanning to figure out our local contents;

  1. Walk the folder. For each file check if it's already in the database and is a match, otherwise hash it and add to the database. Database is indexes by file name.

  2. Iterate over the database, issuing an os.Lstat() for each item in it. If we get back an error, note the file as deleted.

This breaks when a file is renamed case-only on a case insensitive file system. In step one we find it as a new file, since the database is case

jb@syno:/usr/local/go1.5/src ((detached) $ ./all.bash
##### Building Go bootstrap tool.
cmd/dist
##### Building Go toolchain using /usr/local/go1.4.2/.
bootstrap/internal/obj
bootstrap/asm/internal/flags
bootstrap/compile/internal/big
bootstrap/internal/gcprog
bootstrap/internal/obj/arm
// Copyright 2014 The DST Authors. All rights reserved.
// Use of this source code is governed by an MIT-style
// license that can be found in the LICENSE file.
package main
import (
"crypto/rand"
"flag"
"io"
@calmh
calmh / db_test.go
Last active September 1, 2015 10:23
package dbtest
import (
"database/sql"
"fmt"
"os"
"sync"
"testing"
"time"
reporting-disabled = false
[meta]
dir = "/var/opt/influxdb/meta"
hostname = "localhost"
bind-address = ":8088"
retention-autocreate = true
election-timeout = "1s"
heartbeat-timeout = "1s"
leader-lease-timeout = "500ms"
@calmh
calmh / datarace.go
Last active September 21, 2015 09:46
package main
import (
"fmt"
"time"
"github.com/thejerf/suture"
)
func main() {