Skip to content

Instantly share code, notes, and snippets.

View thepaul's full-sized avatar

paul cannon thepaul

49 followers · 15 following
  • Storj Labs
View GitHub Profile
@thepaul
thepaul / encode_ulid_from_timestamp.sql
Created January 6, 2024 17:42
Generate a ULID with a specific TIMESTAMPTZ embedded in it
SELECT
overlay(
overlay(
gen_random_ulid()::STRING
PLACING
overlay(
'0' -- caution: prepending '0' is only appropriate until June of 2527
||
to_hex(
(
@thepaul
thepaul / decode_timestamp_from_ulid.sql
Created January 6, 2024 17:41
Decode the timestamp embedded in a ULID into a TIMESTAMPTZ
SELECT
'epoch'::TIMESTAMPTZ
+
(
'x'
||
lpad(
substr(
replace(
the_uuid::STRING,
@thepaul
thepaul / spawn_pager.py
Created September 9, 2011 17:06
the Right Way(tm) to Popen a curses-using subprocess
# works on Mac OS X and Linux. haven't tried BSD. obviously windows is out.
import os
import subprocess
import signal
def get_tty_fg():
# make a new process group within the same session as the parent. we
# do this so that if the user hits ctrl-c, etc in the curses program
# that's about to be exec'd, the SIGINT and friends won't be sent to
@thepaul
thepaul / tuning reputation further.ipynb
Created February 4, 2022 19:43
Tuning reputation calculation parameters
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@thepaul
thepaul / writecache_sizecalc.go
Created June 14, 2022 23:03
Calculating memory usage of reputation writecache
package main
import (
"sync"
"time"
"unsafe"
"golang.org/x/text/language"
"golang.org/x/text/message"
)
@thepaul
thepaul / runnable-pyc
Created January 2, 2013 20:23
make byte-compiled python code into an executable
#!/usr/bin/env python
#
# runnable-pyc
usage_info = \
"""runnable-pyc - make compiled Python code into an executable file
Usage: runnable-pyc <pycfile> [<outputfile>]
If no outputfile is specified, and pycfile ends in '.pyc', then the default
@thepaul
thepaul / results of data loss versus DQ chance sim.csv
Created February 9, 2022 16:13
data loss lambda dq thresh grace rounds dq fraction mean rounds to dq rounds
0.01 0.950 0.90 30 0.193 473.36 1000
0.02 0.950 0.90 30 0.724 399.98 1000
0.03 0.950 0.90 30 0.975 271.07 1000
0.04 0.950 0.90 30 0.998 153.60 1000
0.05 0.950 0.90 30 1.000 99.79 1000
0.06 0.950 0.90 30 1.000 74.29 1000
0.01 0.980 0.90 30 0.001 50.00 1000
0.02 0.980 0.90 30 0.032 145.78 1000
0.03 0.980 0.90 30 0.202 291.65 1000
@thepaul
thepaul / reputation and dq.csv
Created February 9, 2022 15:50
Data Lost Nodes Lambda Init-alpha Init-beta Succ-adj Fail-adj DQ thresh Weight Audits % DQ'd Mean audits to DQ Stddev audits to DQ
0.01 3000 0.95 1 0 1 -1 0.8 1 10000 4.27 97 819
0.02 3000 0.95 1 0 1 -1 0.8 1 10000 11.03 944 2276
0.03 3000 0.95 1 0 1 -1 0.8 1 10000 27.73 2563 3248
0.04 3000 0.95 1 0 1 -1 0.8 1 10000 61.62 3098 3139
0.05 3000 0.95 1 0 1 -1 0.8 1 10000 93.24 2328 2565
0.06 3000 0.95 1 0 1 -1 0.8 1 10000 99.77 1192 1544
0.07 3000 0.95 1 0 1 -1 0.8 1 10000 100.00 571 802
0.08 3000 0.95 1 0 1 -1 0.8 1 10000 100.00 311 457
0.09 3000 0.95 1 0 1 -1 0.8 1 10000 100.00 189 281
@thepaul
thepaul / data_loss_vs_dq_chance_sim.py
Created February 4, 2022 19:50
Simulating chance of disqualification for a given % of data loss
#!/usr/bin/env python3
#
# This is meant to act as a double-check of math used elsewhere to calculate the chance
# of disqualification given a level of data loss on a storage node. Instead of calculating
# the chance algorithmically, this performs a simulation using the specified parameters
# and reports how many nodes under the given circumstances would actually be disqualified.
import argparse
import csv
import math
@thepaul
thepaul / nodeping.go
Created April 21, 2021 17:17
// Copyright (C) 2021 Storj, Inc.
// See LICENSE for copying information.
package main
import (
"context"
"crypto/tls"
"fmt"
"log"