Skip to content

Instantly share code, notes, and snippets.

@thepaul
thepaul / encode_ulid_from_timestamp.sql
Created January 6, 2024 17:42
Generate a ULID with a specific TIMESTAMPTZ embedded in it
SELECT
overlay(
overlay(
gen_random_ulid()::STRING
PLACING
overlay(
'0' -- caution: prepending '0' is only appropriate until June of 2527
||
to_hex(
(
@thepaul
thepaul / decode_timestamp_from_ulid.sql
Created January 6, 2024 17:41
Decode the timestamp embedded in a ULID into a TIMESTAMPTZ
SELECT
'epoch'::TIMESTAMPTZ
+
(
'x'
||
lpad(
substr(
replace(
the_uuid::STRING,
@thepaul
thepaul / writecache_sizecalc.go
Created June 14, 2022 23:03
Calculating memory usage of reputation writecache
package main
import (
"sync"
"time"
"unsafe"
"golang.org/x/text/language"
"golang.org/x/text/message"
)
data loss lambda dq thresh grace rounds dq fraction mean rounds to dq rounds
0.01 0.950 0.90 30 0.193 473.36 1000
0.02 0.950 0.90 30 0.724 399.98 1000
0.03 0.950 0.90 30 0.975 271.07 1000
0.04 0.950 0.90 30 0.998 153.60 1000
0.05 0.950 0.90 30 1.000 99.79 1000
0.06 0.950 0.90 30 1.000 74.29 1000
0.01 0.980 0.90 30 0.001 50.00 1000
0.02 0.980 0.90 30 0.032 145.78 1000
0.03 0.980 0.90 30 0.202 291.65 1000
Data Lost Nodes Lambda Init-alpha Init-beta Succ-adj Fail-adj DQ thresh Weight Audits % DQ'd Mean audits to DQ Stddev audits to DQ
0.01 3000 0.95 1 0 1 -1 0.8 1 10000 4.27 97 819
0.02 3000 0.95 1 0 1 -1 0.8 1 10000 11.03 944 2276
0.03 3000 0.95 1 0 1 -1 0.8 1 10000 27.73 2563 3248
0.04 3000 0.95 1 0 1 -1 0.8 1 10000 61.62 3098 3139
0.05 3000 0.95 1 0 1 -1 0.8 1 10000 93.24 2328 2565
0.06 3000 0.95 1 0 1 -1 0.8 1 10000 99.77 1192 1544
0.07 3000 0.95 1 0 1 -1 0.8 1 10000 100.00 571 802
0.08 3000 0.95 1 0 1 -1 0.8 1 10000 100.00 311 457
0.09 3000 0.95 1 0 1 -1 0.8 1 10000 100.00 189 281
@thepaul
thepaul / data_loss_vs_dq_chance_sim.py
Created February 4, 2022 19:50
Simulating chance of disqualification for a given % of data loss
#!/usr/bin/env python3
#
# This is meant to act as a double-check of math used elsewhere to calculate the chance
# of disqualification given a level of data loss on a storage node. Instead of calculating
# the chance algorithmically, this performs a simulation using the specified parameters
# and reports how many nodes under the given circumstances would actually be disqualified.
import argparse
import csv
import math
@thepaul
thepaul / tuning reputation further.ipynb
Created February 4, 2022 19:43
Tuning reputation calculation parameters
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
// Copyright (C) 2021 Storj, Inc.
// See LICENSE for copying information.
package main
import (
"context"
"crypto/tls"
"fmt"
"log"
@thepaul
thepaul / git-get-remote-head.sh
Created December 18, 2020 17:22
Get the name of the remote HEAD (the default branch on that remote)
#!/bin/bash -e
REMOTE=${1:-origin}
ref=$(git rev-parse --symbolic-full-name "$REMOTE/HEAD" 2>/dev/null) || {
git remote set-head "$REMOTE" -a >/dev/null
ref=$(git rev-parse --symbolic-full-name "$REMOTE/HEAD")
}
echo "${ref#refs/remotes/$REMOTE/}"
@thepaul
thepaul / pg_bytes.sql
Created April 12, 2018 15:23
postgresql PL/PGSQL functions for encoding 64-bit numbers as 8-byte strings and vice versa
-- transform 8 bytes into an int64
CREATE FUNCTION bytes_to_int64(s BYTEA, OUT result INT8)
AS $$
BEGIN
SELECT bit_or(get_byte(s, ind)::INT8 << ((7 - ind) * 8))
INTO result
FROM generate_series(0, 7) AS ind;
END;
$$ LANGUAGE 'plpgsql';