ryoqun

slow replay experiment

requreiments

• metrics are enabled with the node
• can disclose some basic node setup publicly as a comment below
• can monitor the node closely
• can understand the ramfinication and take the risk of running following ad-hoc commands on your staked production node

objective

ryoqun

#!/bin/bash

# todo: port reverse remapping to nemonic; recognize --dynamic-port-range?;

# run like this: curl -s https://gist.githubusercontent.com/ryoqun/9ad581cdb0d131f12b528faf00eb9652/raw/0000000000..../solana-packet-monitor.sh | nice -n 20 chrt --idle 0 ionice -c 3 bash

set -e
(

if [[ -e /tmp/solana-packet-monitoring.pid ]];

ryoqun

RATE=$1
PORT=$2

dev=$(ip route get 8.8.8.8 | awk '{print $5}' | head -n 1)
ca() {
  par=$1; class=$2; shift 2
  sudo tc class add dev $dev parent 1:$par classid 1:$class hfsc "$@"
}

sudo tc qdisc del dev $dev root

ryoqun

# setup (only run once)
sudo apt install conntrack dnsutils
sudo modprobe nf_conntrack tstamp=1 acct=1
echo 1 | sudo tee /proc/sys/net/netfilter/nf_conntrack_timestamp
echo 1 | sudo tee /proc/sys/net/netfilter/nf_conntrack_acct
echo $((3600 * 24 * 30)) | sudo tee /proc/sys/net/netfilter/nf_conntrack_udp_timeout
echo $((3600 * 24 * 30)) | sudo tee /proc/sys/net/netfilter/nf_conntrack_udp_timeout_stream
sudo iptables -t mangle -A PREROUTING -p udp -m conntrack --ctstate NEW -j ACCEPT
sudo conntrack -F

ryoqun

$ fping -s -c10 -q $((timeout 20 solana-gossip spy --entrypoint tds.solana.com:8001 2>&1 | grep -vF "  me| " | grep -oE '[0-9]+(\.[0-9]+){3}' || true) | grep -vF 0.0.0.0 | sort -n -t . | uniq)
3.135.231.134   : xmt/rcv/%loss = 10/0/100%
3.135.65.138    : xmt/rcv/%loss = 10/0/100%
3.14.7.80       : xmt/rcv/%loss = 10/0/100%
3.15.0.77       : xmt/rcv/%loss = 10/0/100%
3.16.10.151     : xmt/rcv/%loss = 10/0/100%
3.19.123.2      : xmt/rcv/%loss = 10/0/100%
3.86.104.17     : xmt/rcv/%loss = 10/10/0%, min/avg/max = 0.44/0.45/0.49
5.189.150.170   : xmt/rcv/%loss = 10/10/0%, min/avg/max = 89.1/89.1/89.4
5.58.235.16     : xmt/rcv/%loss = 10/10/0%, min/avg/max = 114/115/116

ryoqun

    field blockhash_queue &std::sync::rwlock::RwLock<solana_runtime::blockhash_queue::BlockhashQueue>
            field hash_height &u64
                    primitive u64
            field last_hash &core::option::Option<solana_sdk::hash::Hash>
                        newtype Hash struct [u8; 32]
                            tuple 32
                            element &u8
                                    primitive u8
                            element &u8
                                    primitive u8

ryoqun

[2020-01-09T05:14:36.179764874Z WARN  solana_vote_program::vote_state] GVthNzmh8UUJjnVqZNDKEjn43hPmHm7WRFw32GBoX6Qy dropped vote Vote { slots: [274775, 274776, 274777, 274779, 274780, 274781, 274782, 274783, 274784, 274785, 274786, 274787, 274788, 274789, 274791, 274792, 274793, 274794, 274795, 274796, 274797, 274798, 274799, 274800], hash: 7UwgoSmE82cPrZXPtimG5YsjFE9Qt4SLt9VhRb1PtXdg, timestamp: None } failed to match hash 7UwgoSmE82cPrZXPtimG5YsjFE9Qt4SLt9VhRb1PtXdg 4JkTCNTb9wYV9ZHUfPon8V8qwBMMboGm2uAMw52CDW1g
[2020-01-09T05:14:37.367954236Z WARN  solana_vote_program::vote_state] GVthNzmh8UUJjnVqZNDKEjn43hPmHm7WRFw32GBoX6Qy dropped vote Vote { slots: [274775, 274776, 274777, 274779, 274780, 274781, 274782, 274783, 274784, 274785, 274786, 274787, 274788, 274789, 274791, 274792, 274793, 274794, 274795, 274796, 274797, 274798, 274799, 274800, 274801], hash: CYZzz1gdhmwwzDnEZCnzaVqvwF5mxox737hy1TqPuBoH, timestamp: None } failed to match hash CYZzz1gdhmwwzDnEZCnzaVqvwF5mxox737hy1TqPuBoH EdrCozsZxEwtKbi6KgRrrHGq9

ryoqun

ryoqun@ubuqun:~$ (sleep 3 && pid=$(pgrep -f ^stopme) && kill -s STOP $pid && sleep 3 && kill -s CONT $pid) & strace -f -e process bash --noprofile --norc -i -c "(exec -a stopme sleep 8) && date; fg"; sleep 10
[1] 16006
execve("/bin/bash", ["bash", "--noprofile", "--norc", "-i", "-c", "(exec -a stopme sleep 8) && date"...], [/* 68 vars */]) = 0
arch_prctl(ARCH_SET_FS, 0x7f587ad0f700) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f587ad0f9d0) = 16011
strace: Process 16011 attached
[pid 16010] wait4(-1,  <unfinished ...>
[pid 16011] execve("/bin/sleep", ["stopme", "8"], [/* 68 vars */]) = 0
[pid 16011] arch_prctl(ARCH_SET_FS, 0x7fb11539d700) = 0
[pid 16011] --- SIGSTOP {si_signo=SIGSTOP, si_code=SI_USER, si_pid=16006, si_uid=1000} ---

ryoqun

Keybase proof

I hereby claim:

• I am ryoqun on github.
• I am ryoqun (https://keybase.io/ryoqun) on keybase.
• I have a public key whose fingerprint is 5CC8 4627 A3A2 651D 4048 F5E9 055C 9BBD 5C48 930F

To claim this, I am signing this object:

ryoqun

cat _X_ANONYMOUS#__block__\$block@1.opt.ll | sed 's/, !dbg.*$//g' | opt-3.3 --dot-cfg -o /dev/null
cat cfg._X_ANONYMOUS#__block__\$block@1.dot |  sed 's/digraph "Call graph" {/digraph "Call Graph" { \n rankdir=LR;/' | sed -r 's/%\\"struct.rubinius::([a-zA-Z]*?)\\"/\1/g' | dot -Tpdf > /tmp/callgraph.pdf