** göerli address employed: 0xB884D62C640BED508EEcA786cF139075C290F3E1 . Decided to try Brave's wallet for this.
** machine: (8-core) Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz, 32GB RAM, dual ISP's + maybe 1-hour battery back-up on all machines.
** OS: Ubuntu server 22.04.01 LTS
✓ Verified 2 göerli ETH + 50k göerli T received, here: https://goerli.etherscan.io/address/0x6bb853467a0901a8Fe4f974ac1B8071F2FD07D5b#tokentxnsPulled up testnet docs here:
https://docs.threshold.network/app-development/ropsten-testnet-staking/testnet-tbtc-v2-node-setup .
Noticed this banner:
"While it is possible to run the client on a local machine, this is not recommended."
Maybe a short explanation as to why such is not recommended might prove helpful.
Under section "Ethereum API", reference is made to "A Keep node..." which adds yet another label to these operations.
The doc is entitled "Testnet tBTC v2 node Setup" and we are setting up a Threshold tbtcv2+randomBeacon node, recommend settling on terms.
Under that same section, a bit more re: the UTC private key file might save some users a bit of "why am I doing this?" confusion:
The tbtcv2 client requires access to a special file containing the private key of your Ethereum "operator" account.
You will generate this key file in a subsequent step using the GoEthereum (Geth) client and using a password you must provide.
The resulting private key file will be named generally as follows:
UTC--<DATE_TIME>--<operator_account>
EXAMPLE:
UTC--2022-08-06T16-60-17.033517626Z--32d59cecaf8d3e35474fe03756282885661ead4f
In this case, the operator Ethereum address is 0x32d59cecaf8d3e35474fe03756282885661ead4f
The password you used must be provided to the tbtcv2 client, normally via the KEEP_ETHEREUM_PASSWORD environment variable.
Next, the notices around "Application Authorization" seem to be associated with "Announced Addresses". Screenshot.
I spend 7 minutes trying to figure out where "To get started, visit the Threshold Dashboard and connect your wallet." can be found, and how to attach the Brave Browser wallet to it. Feeling irritated, but such is normal. Thinking a link to the dashboard would be useful here:
To get started, visit the Threshold Dashboard '
Worked through creating a 50000T stake, authorizing 100% to the 2 applications and designating the operator via 6 transactions. Only my familiarity with the situation keeps me sane- I already knew how to accomplish these steps yet still had to click around quite a bit to find the elements I sought, in order.
I can't imagine not being completely upset and turned off by that experience, were I a newbie.
However, I currently lack the patience to elaborate. Moved on.
Re: "The operator account is the Ethereum account created on your node."
I suggest pointing out that "The operator account is the Ethereum account for which you possess the UTC-- file mentioned above, the one you *should have used in the prior "Register your Operator" step".
(I am editing this days later. I caused myself problems by just blowing through the operator account setup and assigning it to the same address as the stakers. ugh. )
Ran into "Create Folder Structure" and knew I wanted a user specific to this situation, in this case user 'tbtc'.
so I ran a
sudo adduser tbtc
Then I became user "tbtc" via 'ssh', rather than via 'su', which is a required step for using "rootless" docker:
ssh tbtc@localhost
I then created these directories, without the "keep" parent:
mkdir {bin,storage,config}
Just realized I created the göerli address with the Brave Wallet since my göerli rig has expired and I didn't have luck spinning it backup and that I need a UTC file for the operator. Exported the priv key from Brave and created a json UTC file, but don't have a password. I forget if I can get around that, will test tomorrow via geth, too tired to deal with it now. Worst case is I need another 50K goerli T to an account I create via geth. argh... way more missteps for this testnet than for the real one. I realize I'm always way more stressed about using browsers to interact with dashboards than anything else - I managed to convince myself using a browser.
I'm tired, mixed up. I could have created the UTC and just sent göerli ETH to it and been fine. I'm afraid this hasn't been the best week for me to try to do this.
Noting that I created a password-less operator file, I checked to see if geth could unlock the resulting UTC file:personal.unlockAccount(eth.accounts[3]) Unlock account 0x6bb853467a0901a8fe4f974ac1b8071f2fd07d5b Passphrase: true
So I decide to forge ahead, intending to set the password to "" when the time comes. Switched machines to an 8-core i7-4790 CPU @ 3.60GHz with 16GB RAM, the same as is running the mainnet client. I decided to run these tests as rootless docker as well, supposedly more secure, matches my mainnet client.
tbtc@changlas:~$ dockerd-rootless-setuptool.sh install
[INFO] Creating /home/tbtc/.config/systemd/user/docker.service
+ systemctl --user --no-pager --full status docker.service
● docker.service - Docker Application Container Engine (Rootless)
Loaded: loaded (/home/tbtc/.config/systemd/user/docker.service; disabled; vendor preset: enabled)
Active: active (running) since Mon 2022-10-24 22:35:08 BRT; 3min 0s ago
Docs: https://docs.docker.com/go/rootless/
Main PID: 4181273 (rootlesskit)
Tasks: 63
Memory: 52.7M
CPU: 557ms
CGroup: /user.slice/user-1012.slice/user@1012.service/app.slice/docker.service
├─4181273 rootlesskit --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave /usr/bin/dockerd-rootless.sh
├─4181284 /proc/self/exe --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave /usr/bin/dockerd-rootless.sh
├─4181302 slirp4netns --mtu 65520 -r 3 --disable-host-loopback --enable-sandbox --enable-seccomp 4181284 tap0
├─4181309 dockerd
└─4181332 containerd --config /run/user/1012/docker/containerd/containerd.toml --log-level info
Oct 24 22:35:08 changlas dockerd-rootless.sh[4181309]: time="2022-10-24T22:35:08.476024325-07:00" level=warning msg="Unable to find cpuset controller"
Oct 24 22:35:08 changlas dockerd-rootless.sh[4181309]: time="2022-10-24T22:35:08.476160947-07:00" level=info msg="Loading containers: start."
Oct 24 22:35:08 changlas dockerd-rootless.sh[4181309]: time="2022-10-24T22:35:08.561009464-07:00" level=warning msg="Running modprobe bridge br_netfilter failed with message: modprobe: ERROR: could not insert 'br_netfilter': Operation not permitted\ninsmod /lib/modules/5.15.0-48-generic/kernel/net/bridge/br_netfilter.ko \n, error: exit status 1"
Oct 24 22:35:08 changlas dockerd-rootless.sh[4181309]: time="2022-10-24T22:35:08.561910695-07:00" level=info msg="skipping firewalld management for rootless mode"
Oct 24 22:35:08 changlas dockerd-rootless.sh[4181309]: time="2022-10-24T22:35:08.696346966-07:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Oct 24 22:35:08 changlas dockerd-rootless.sh[4181309]: time="2022-10-24T22:35:08.805717854-07:00" level=info msg="Loading containers: done."
Oct 24 22:35:08 changlas dockerd-rootless.sh[4181309]: time="2022-10-24T22:35:08.811291524-07:00" level=warning msg="Not using native diff for overlay2, this may cause degraded performance for building images: running in a user namespace" storage-driver=overlay2
Oct 24 22:35:08 changlas dockerd-rootless.sh[4181309]: time="2022-10-24T22:35:08.811536092-07:00" level=info msg="Docker daemon" commit=e42327a graphdriver(s)=overlay2 version=20.10.18
Oct 24 22:35:08 changlas dockerd-rootless.sh[4181309]: time="2022-10-24T22:35:08.811790567-07:00" level=info msg="Daemon has completed initialization"
Oct 24 22:35:08 changlas dockerd-rootless.sh[4181309]: time="2022-10-24T22:35:08.837095604-07:00" level=info msg="API listen on /run/user/1012/docker.sock"
+ DOCKER_HOST=unix:///run/user/1012/docker.sock /usr/bin/docker version
Client: Docker Engine - Community
Version: 20.10.18
API version: 1.41
Go version: go1.18.6
Git commit: b40c2f6
Built: Thu Sep 8 23:11:43 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.18
API version: 1.41 (minimum version 1.12)
Go version: go1.18.6
Git commit: e42327a
Built: Thu Sep 8 23:09:30 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.8
GitCommit: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
runc:
Version: 1.1.4
GitCommit: v1.1.4-0-g5fd4c4d
docker-init:
Version: 0.19.0
GitCommit: de40ad0
+ systemctl --user enable docker.service
Created symlink /home/tbtc/.config/systemd/user/default.target.wants/docker.service → /home/tbtc/.config/systemd/user/docker.service.
[INFO] Installed docker.service successfully.
[INFO] To control docker.service, run: `systemctl --user (start|stop|restart) docker.service`
[INFO] To run docker.service on system startup, run: `sudo loginctl enable-linger tbtc`
[INFO] Creating CLI context "rootless"
Successfully created context "rootless"
[INFO] Make sure the following environment variables are set (or add them to ~/.bashrc):
export PATH=/usr/bin:$PATH
export DOCKER_HOST=unix:///run/user/1012/docker.sock
-------------------------------------------------------
tbtc@changlas:~$ echo $UID
1012
tbtc@changlas:~$ echo 'export PATH=/usr/bin:$PATH' >> ~/.bashrc
tbtc@changlas:~$ echo 'export DOCKER_HOST=unix:///run/user/1012/docker.sock' >> ~/.bashrc
tbtc@changlas:~$ echo 'export XDG_RUNTIME_DIR=/run/user/1012/' >> ~/.bashrc
-------------------------------------------------------
tbtc@changlas:~$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
tbtc@changlas:~$ systemctl --user enable docker
tbtc@changlas:~$ loginctl enable-linger $(whoami)
tbtc@changlas:~$ loginctl show-user $(whoami)
UID=1012
GID=1012
Name=tbtc
Timestamp=Sun 2022-10-09 15:40:28 BRT
TimestampMonotonic=4030392
RuntimePath=/run/user/1012
Service=user@1012.service
Slice=user-1012.slice
Display=682
State=active
Sessions=682
IdleHint=no
IdleSinceHint=1666676646314408
IdleSinceHintMonotonic=1321422202858
Linger=yes
-------------------------------------------------------
At this point I take a look at "The Docker Launch Script" section. Note that this:
# copied to home/keep/config earlier
should include the leading slash as so:
# copied to /home/keep/config earlier
I removed the following lines from my version because systemd will be managing this.
--detach \
--restart on-failure \
The following lines were altered to allow running the additional instance within my previously arranged machine + firewall:
-p 3920:3919 \
-p 9602:9601 \
and I forwarded 3920 and 9602 to the appropriate system from my firewall.
Then I attempted to start and got this error:
Unable to find image 'us-docker.pkg.dev/keep-test-f3e0/public/keep-client:latest' locally
further investigation showed I could not run the docker hello-world either - a problem with my setup.
Life has settled down somewhat, went back and tried to read what I'd written above, made edits.Discovered tbtc docker client will not accept a zero-length password, though geth will unlock such a UTC file. Asked Sasha for more goerli T, my mistakes have stopped me. Corrected the "Unable to find image.." issue above by simply restarting the rootless docker daemon, recalled I should have tested the situation prior to attempting to launch tbtc daemon by running the hello-world test.
As tbtc user:
tbtc@changlas:~$ systemctl --user stop docker
tbtc@changlas:~$ docker ps
Cannot connect to the Docker daemon at unix:///run/user/1012/docker.sock. Is the docker daemon running?
tbtc@changlas:~$ systemctl --user start docker
tbtc@changlas:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
## make certain the 'hello-world' docker image functions correctly before proceeding
tbtc@changlas:~$ docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
Cleaned up, removed the hello-world container:
tbtc@changlas:~$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f47a7482e40b hello-world "/hello" 4 seconds ago Exited (0) 2 seconds ago beautiful_varahamihira
tbtc@changlas:~$ docker rm f47a7482e40b
f47a7482e40b
I thought I would go see what happens when I try to Unstake on this network, not knowing if there is also a 1.5 month cool-down period. Attempted to UNSTAKE 50k T via the testnet dashboard and when the tx was submitted to my wallet I see the following error:
cannot estimate gas; transaction may fail or may require manual gas limit [ See: https://links.ethers.org/v5-errors-UNPREDICTABLE_GAS_LIMIT ] (reason="execution reverted: Too much to unstake", method="estimateGas", transaction={"from":"0x6bb853467a0901a8Fe4f974ac1B8071F2FD07D5b","to":"0x1da5d88C26EA4f87b5e09C3452eE2384Ee20DC75","data":"0xd3ecb6cd0000000000000000000000006bb853467a0901a8fe4f974ac1b8071f2fd07d5b000000000000000000000000000000000000000000000a968163f0a57b400000","accessList":null}, error={"code":3,"data":"0x08c379a000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000013546f6f206d75636820746f20756e7374616b6500000000000000000000000000","message":"execution reverted: Too much to unstake"}, code=UNPREDICTABLE_GAS_LIMIT, version=providers/5.6.8)
Tried 40k T, same error:
Used Brave Wallet to generate a new göerli staker account. Transferred 50k additional göerli-T tokens to 0xB884D62C640BED508EEcA786cF139075C290F3E1 ( and updated account info at the top of this doc ). Attempted to re-engage Brave Browser with
https://dashboard.test.threshold.network/staking
and was unable to figure out how to switch accounts to the 0xB884 within that interface. Discovered, with much relief, the "dark mode" button on the Threshold interface as I tried to switch to the new account. Clicking in the usual place, upper right button displaying currently connected account proved unfruitful, only "disconnet" choice. Made certain Brave was using 0xB884, reloaded pages, no joy. Cleared 7 days worth of Brave cache, allowed unlock to expire on Brave wallet, unlocked Brave wallet, noted that history showed no auto-fill for dashboard.test.threshold.network ( correct behaviour ), retried, still unable to engage new account.
Bailed on use of Brave, setup Tally on chromium, could not figure out foreign key import, checked Tally Discord, foreign key import not yet implemented. As mentioned, the UX part of crypto is still so problematic, conundrum. I did like the 24 + 1 length of Tally seed.
Bailed on Tally, switched to MetaMask on chromium, imported 0xB884D62C640BED508EEcA786cF139075C290F3E1 priv key and hooked up to https://dashboard.test.threshold.network/staking, generated the 4 tx's to Approve, Stake and Increase Authorizations.
Generated a new operator account 0x7edd65b8314bd489fc319925a288c59c3ee92a1e via geth node, with a known password, copied the UTC to the appropriate machine+path, chmod 400, updated launching script.
Could not find where to map operator account. Carefully looked around this time, my head is now on straight. Never found it, clicked allover the place. Note that this search occurred maybe 5-10 mins after the last Increase Auth tx. So one might expect that "map operator account" to have appeared. I managed to trigger that dialog by reloading the dashboard and then submitted the final 2 tx's.
Invoked launch script, appeared to start, checked /metrics:
curl http://localhost:9602/metrics
client_info{version="v2.0.0-m1-9-g4049dc015"} 1
# TYPE connected_bootstrap_count gauge
connected_bootstrap_count 0 1666735018641
# TYPE connected_peers_count gauge
connected_peers_count 0 1666735018641
# TYPE eth_connectivity gauge
eth_connectivity 1 1666734778640
# TYPE tbtc_pre_params_count gauge
tbtc_pre_params_count 4 1666735018956
Noted lack of seednodes, investigated a bit and found loads are high as storage/work/tbtc/preparams/pp_xyz files are generated, approx one file per minute generated on this system ( which is also running tbtc mainnet as a different user ) - likely it is not yet ready to participate. Suspect tbtc_pre_params_count must reach 1k before p2p participation begins. If so, gonna be a while. Seednodes contacted after a bit, 8 peers showing.
A few hours later, up to 170 tbtc_pre_params generated, realized I'd forgotten to fund operator, sent 1 göerli Eth.
all 1000 params have been generated, node continues to function.The following are root/sudo prep for running docker rootless, but required prior to setting up the user.
adduser tbtc
apt-get remove docker docker-engine docker.io containerd runc
apt autoremove
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
cat /etc/apt/sources.list.d/docker.list
apt update
apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin
systemctl status docker.service docker.socket
systemctl disable --now docker.service docker.socket
updatedb
apt-get install -y dbus-user-session uidmap docker-ce-rootless-extras systemd-container
Review this: https://blog.jitendrapatro.me/migrating-rootful-docker-containers-to-rootless-docker/
Note the section "Login to rootless user account" pertaining to login via pam_systemd and NOT via 'su'.
Required to manage running tbtc client rootless, via systemd, as user 'tbtc'. Edit this file, call it what you will, normally just 'tbtc.service' -
> vi .config/systemd/user/goerli-tbtc.service
and insert something like the following:
[Unit]
Description=göerli tBTCv2 client
After=network.target
Wants=network.target
[Service]
## general form: Environment="VARNAME=value"
Environment="ETHEREUM_WS_URL=wss://goerli.infura.io/ws/v3/<apikey>"
Environment="OPERATOR_KEY_FILE_NAME=UTC--2022-10-25T21-27-55.865009089Z--7edd65b8314bd489fc319925a288c59c3ee92a1e"
Environment="OPERATOR_KEY_FILE_PASSWORD=password"
Environment="PUBLIC_IP=/dns4/<dynDnsName>/tcp/3920"
Environment="CONFIG_DIR=/home/tbtc/config"
Environment="STORAGE_DIR=/home/tbtc/storage"
Environment="XDG_RUNTIME_DIR=/run/user/1012/"
Environment="DOCKER_HOST=unix:///run/user/1012/docker.sock"
Type=simple
WorkingDirectory=/home/tbtc
ExecStart=/usr/bin/docker run \
--volume ${CONFIG_DIR}:/mnt/keep/config \
--volume ${STORAGE_DIR}:/mnt/keep/storage \
--env KEEP_ETHEREUM_PASSWORD=${OPERATOR_KEY_FILE_PASSWORD} \
--env LOG_LEVEL=info \
--log-opt max-size=100m \
--log-opt max-file=3 \
-p 3920:3919 \
-p 9602:9601 \
us-docker.pkg.dev/keep-test-f3e0/public/keep-client \
start \
--goerli \
--ethereum.url ${ETHEREUM_WS_URL} \
--ethereum.keyFile /mnt/keep/config/${OPERATOR_KEY_FILE_NAME} \
--storage.dir /mnt/keep/storage \
--network.announcedAddresses /ip4/$PUBLIC_IP/tcp/3920
ExecReload=/bin/kill -s HUP $MAINPID
Restart=always
RestartSec=15s
[Install]
WantedBy=default.target
Alias=göerli-tbtc
Once the systemd service file is in place, make certain it runs. Kill the running client if you launched via your shell. As user 'tbtc', inform systemd of the new service file:
systemctl --user daemon-reload
...check that it is not currently running:
○ goerli-tbtc.service - göerli tBTCv2 client
Loaded: loaded (/home/tbtcv2/.config/systemd/user/goerli-tbtc.service; disabled; vendor preset: enabled)
Active: inactive (dead)
and attempt to start:
systemctl --user start goerli-tbtc.service
and check that it is running:
systemctl --user status goerli-tbtc.service
● goerli-tbtc.service - göerli tBTCv2 client
Loaded: loaded (/home/tbtc/.config/systemd/user/goerli-tbtc.service; disabled; vendor preset: enabled)
Active: active (running) since Wed 2022-10-26 22:11:35 BIT; 3s ago
Main PID: 502295 (docker)
Tasks: 12 (limit: 19047)
Memory: 10.8M
CPU: 28ms
CGroup: /user.slice/user-1012.slice/user@1012.service/app.slice/goerli-tbtc.service
└─502295 /usr/bin/docker run --volume /home/tbtc/config:/mnt/keep/config --volume /home/tbtc/storage>
Note this line:
Loaded: loaded (/home/tbtc/.config/systemd/user/goerli-tbtc.service; disabled; vendor preset: enabled)
mentions "disabled", meaning the service will not restart automatically on boot. Correct this:
systemctl --user enable goerli-tbtc.service
Created symlink /home/tbtc/.config/systemd/user/default.target.wants/goerli-tbtc.service → /home/tbtc/.config/systemd/user/goerli-tbtc.service.
and re-check that 'enabled' is showing:
systemctl --user status goerli-tbtc.service
● goerli-tbtc.service - göerli tBTCv2 client
Loaded: loaded (/home/tbtc/.config/systemd/user/goerli-tbtc.service; enabled; vendor preset: enabled)
Be certain to check that the user-run dockerd is enabled the as well. If not, enable that too:
systemctl --user status docker.service
● docker.service - Docker Application Container Engine (Rootless)
Loaded: loaded (/home/tbtc/.config/systemd/user/docker.service; enabled; vendor preset: enabled)
Reboot and make certain both dockerd and göerli tBTCv2 client are running after reboot.
systemctl --user status docker goerli-tbtc
● docker.service - Docker Application Container Engine (Rootless)
Loaded: loaded (/home/tbtc/.config/systemd/user/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2022-10-26 19:21:35 BIT; 14min ago
.
.
.
.
.
● goerli-tbtc.service - göerli tBTCv2 client
Loaded: loaded (/home/tbtc/.config/systemd/user/goerli-tbtc.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2022-10-26 19:21:35 BIT; 14min ago
.
.
.
.
.
the "journalctl" command works as expected, also with the '--user' flag:
journalctl --user -fu goerli-tbtc
Oct 26 16:18:38 changlas docker[502295]: 2022-10-26T16:15:38.998Z INFO keep-libp2p libp2p/libp2p.go:241 number of connected peers: [10]
At this point, pam_systemd access is no longer required for this user. As root/sudo,
vi /etc/ssh/sshd_config
and add
DenyUsers tbtc
and restart ssh daemon:
systemctl restart sshd
Check that the tbtc user can no longer successfully login via ssh:
ssh tbtc@localhost
Noticed 1002 pp_ files exist in storage, 1000 reported via metrics, started monitoring them for changes and/or increase in number. Likely to just trigger a tar+rsync off-system in event of directory alterations, storage is 4.1M currently on both test/mainnet. Made rudimentary python gizmo to eventually monitor node(s) are up, metrics within reason etc. and that operator ETH balances are sufficient.