Skip to content

Instantly share code, notes, and snippets.

@brenzi
Created April 29, 2020 11:21
Show Gist options
  • Save brenzi/7be1c6a1b991269870bb4daecfffc4c4 to your computer and use it in GitHub Desktop.
Save brenzi/7be1c6a1b991269870bb4daecfffc4c4 to your computer and use it in GitHub Desktop.
gantree-failure
abrenzikofer@devsgx02:~$ docker run -v ~/gantree-work/:/gantree --env-file ~/gantree-env/envfile.substratee -e HOST_USER=$(id -u) --rm -ti gantree-cli-docker --version
gantree-cli 0.8.4
⮡ gantree-lib 0.6.6
abrenzikofer@devsgx02:~$ docker run -v ~/gantree-work/:/gantree --env-file ~/gantree-env/envfile.substratee -e HOST_USER=$(id -u) --rm -ti gantree-cli-docker sync
██████╗ █████╗ ███╗ ██╗████████╗██████╗ ███████╗███████╗
██╔════╝ ██╔══██╗████╗ ██║╚══██╔══╝██╔══██╗██╔════╝██╔════╝
██║ ███╗███████║██╔██╗ ██║ ██║ ██████╔╝█████╗ █████╗
██║ ██║██╔══██║██║╚██╗██║ ██║ ██╔══██╗██╔══╝ ██╔══╝
╚██████╔╝██║ ██║██║ ╚████║ ██║ ██║ ██║███████╗███████╗
╚═════╝ ╚═╝ ╚═╝╚═╝ ╚═══╝ ╚═╝ ╚═╝ ╚═╝╚══════╝╚══════╝
[!] Using override for Gantree config path
[!] Using override for inventory path
2020-04-29T08:07:02.056Z [Gantree] (lib/gantree) info: reading gantree configuration
2020-04-29T08:07:02.084Z [Gantree] (lib/config/validate) info: Gantree configuration validated successfully
2020-04-29T08:07:02.084Z [Gantree] (lib/config/preprocessors/boolToString) info: converting booleans in config to strings
2020-04-29T08:07:02.085Z [Gantree] (lib/config/preprocessors/injectEnvVars) info: injecting environment variables into config
2020-04-29T08:07:02.085Z [Gantree] (lib/config/inject) info: injecting defaults
2020-04-29T08:07:02.086Z [Gantree] (lib/ansible/inventory) info: creating namespace
2020-04-29T08:07:02.086Z [Gantree] (lib/ansible/inventory) info: namespace created
2020-04-29T08:07:02.086Z [Gantree] (lib/ansible/inventory) info: creating Gantree inventory
2020-04-29T08:07:02.164Z [Gantree] (lib/ansible/inventory) info: Gantree config hash written (490d0441ff1e6958a59170e7d28870043272e85bfabbdd5f3d41d3547f790b27)
2020-04-29T08:07:02.164Z [Gantree] (lib/ansible/inventory) info: Gantree inventory created
2020-04-29T08:07:02.164Z [Gantree] (lib/ansible/commands) info: running playbook: /usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/ansible/infra.yml
2020-04-29T08:07:02.164Z [Gantree] (cmd) info: Executing: ansible-playbook -i /gantree/inventory/substratee-test/gantree -i /gantree/inventory/substratee-test/active /usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/ansible/infra.yml, {"verbose":true}
PLAY [localhost] ***************************************************************
TASK [Gathering Facts] *********************************************************
ok: [localhost]
TASK [add_host] ****************************************************************
ok: [localhost] => (item=None)
ok: [localhost] => (item=None)
ok: [localhost]
PLAY [base_infra] **************************************************************
TASK [gcp selector] ************************************************************
skipping: [gantree-infra-substratee-test-0]
skipping: [gantree-infra-substratee-test-1]
included: /usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/ansible/roles/infra-do-node/tasks/1_present.yml for gantree-infra-substratee-test-0
included: /usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/ansible/roles/infra-do-node/tasks/1_present.yml for gantree-infra-substratee-test-1
TASK [infra-do-node : do sshkey part1] *****************************************
changed: [gantree-infra-substratee-test-1 -> localhost]
TASK [infra-do-node : do sshkey part1] *****************************************
ok: [gantree-infra-substratee-test-0 -> localhost]
TASK [infra-do-node : do sshkey part2] *****************************************
ok: [gantree-infra-substratee-test-1 -> localhost]
TASK [infra-do-node : do sshkey part2] *****************************************
ok: [gantree-infra-substratee-test-0 -> localhost]
TASK [infra-do-node : do get all sshkeys] **************************************
ok: [gantree-infra-substratee-test-1 -> localhost]
TASK [infra-do-node : do filter sshkey] ****************************************
ok: [gantree-infra-substratee-test-1 -> localhost] => (item=None)
ok: [gantree-infra-substratee-test-1]
TASK [infra-do-node : do get all sshkeys] **************************************
ok: [gantree-infra-substratee-test-0 -> localhost]
TASK [infra-do-node : do filter sshkey] ****************************************
ok: [gantree-infra-substratee-test-0 -> localhost] => (item=None)
ok: [gantree-infra-substratee-test-0]
......DONE
TASK [infra-do-node : do droplet] **********************************************
changed: [gantree-infra-substratee-test-1 -> localhost]
TASK [infra-do-node : droplet tags] ********************************************
..DONE
TASK [infra-do-node : do droplet] **********************************************
changed: [gantree-infra-substratee-test-0 -> localhost]
TASK [infra-do-node : droplet tags] ********************************************
.DONE
TASK [infra-do-node : Wait for SSH to come up] *********************************
ok: [gantree-infra-substratee-test-1 -> localhost]
TASK [infra-do-node : include_tasks] *******************************************
skipping: [gantree-infra-substratee-test-1]
TASK [aws selector] ************************************************************
skipping: [gantree-infra-substratee-test-1]
TASK [infra-do-node : Wait for SSH to come up] *********************************
ok: [gantree-infra-substratee-test-0 -> localhost]
TASK [infra-do-node : include_tasks] *******************************************
skipping: [gantree-infra-substratee-test-0]
TASK [aws selector] ************************************************************
skipping: [gantree-infra-substratee-test-0]
PLAY RECAP *********************************************************************
gantree-infra-substratee-test-0 : ok=7 changed=1 unreachable=0 failed=0 skipped=4 rescued=0 ignored=0
gantree-infra-substratee-test-1 : ok=7 changed=2 unreachable=0 failed=0 skipped=4 rescued=0 ignored=0
localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
2020-04-29T08:08:09.439Z [Gantree] (lib/ansible/commands) info: playbook finished: /usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/ansible/infra.yml
!!!NODE_IP_ADDRESSES ==> [{"IP":"104.248.35.51","hostName":"substratee_test_0"},{"IP":"68.183.216.242","hostName":"substratee_test_1"}]
2020-04-29T08:08:11.188Z [Gantree] (lib/ansible/commands) info: running playbook: /usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/ansible/operation.yml
2020-04-29T08:08:11.188Z [Gantree] (cmd) info: Executing: ansible-playbook -i /gantree/inventory/substratee-test/gantree -i /gantree/inventory/substratee-test/active /usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/ansible/operation.yml, {"verbose":true}
PLAY [validator:builder_bin:builder_spec] **************************************
TASK [wait for machine and ssh] ************************************************
ok: [104.248.35.51]
ok: [68.183.216.242]
PLAY [validator:builder_bin:builder_spec] **************************************
TASK [Gathering Facts] *********************************************************
ok: [68.183.216.242]
ok: [104.248.35.51]
TASK [gantree-debug : gantree debug] *******************************************
ok: [104.248.35.51] => {
"msg": [
"Gantree Debug:",
" gantree_root: ../",
" gantree_control_working: /gantree/control/substratee-test/operation",
" ansible_ssh_common_args: -o StrictHostKeyChecking=no -o ControlMaster=no -o UserKnownHostsFile=/dev/null -o ServerAliveInterval=30 -o ControlPersist=60s",
"",
" substrate_network_id: local_testnet",
" project_name: substratee-test",
"",
" repository_version: -NO_KEY-",
" substrate_bin_name: substratee-node-0.6.0",
" substrate_binary_sha256: 6b48d692bfb6ecec60dba537ab79e0cbb50ac0ebd8076505222fe92980928e63",
" substrate_binary_url: https://github.com/scs/substraTEE-node/releases/download/v0.6.0/substratee-node-0.6.0",
" substrate_use_default_spec: false",
" substrate_chain_argument: false",
" substrate_binary_path: false",
" substrate_repository: false",
" substrate_local_compile: false",
" substrate_repository_version: HEAD",
" substrate_bootnode_argument: []",
" substrate_purge_chain: true",
" substrate_options: ['--telemetry-url=\"ws://telemetry.polkadot.io:1024 0\"']",
"",
" telemetry: {'repository': 'https://github.com/flex-dapps/substrate-telemetry.git', 'binary_url': 'https://nyc3.digitaloceanspaces.com/gantree-rozifus-00/flexdapps-telemetry-0.1.0', 'binary_name': 'telemetry', 'src_folder': 'telemetry_src', 'src_subfolder': 'backend', 'operation': 'fetch'}",
" substrate_telemetry_argument: ws://127.0.0.1:8000/submit",
""
]
}
ok: [68.183.216.242] => {
"msg": [
"Gantree Debug:",
" gantree_root: ../",
" gantree_control_working: /gantree/control/substratee-test/operation",
" ansible_ssh_common_args: -o StrictHostKeyChecking=no -o ControlMaster=no -o UserKnownHostsFile=/dev/null -o ServerAliveInterval=30 -o ControlPersist=60s",
"",
" substrate_network_id: local_testnet",
" project_name: substratee-test",
"",
" repository_version: -NO_KEY-",
" substrate_bin_name: substratee-node-0.6.0",
" substrate_binary_sha256: 6b48d692bfb6ecec60dba537ab79e0cbb50ac0ebd8076505222fe92980928e63",
" substrate_binary_url: https://github.com/scs/substraTEE-node/releases/download/v0.6.0/substratee-node-0.6.0",
" substrate_use_default_spec: false",
" substrate_chain_argument: false",
" substrate_binary_path: false",
" substrate_repository: false",
" substrate_local_compile: false",
" substrate_repository_version: HEAD",
" substrate_bootnode_argument: []",
" substrate_purge_chain: true",
" substrate_options: ['--telemetry-url=\"ws://telemetry.polkadot.io:1024 0\"']",
"",
" telemetry: {'repository': 'https://github.com/flex-dapps/substrate-telemetry.git', 'binary_url': 'https://nyc3.digitaloceanspaces.com/gantree-rozifus-00/flexdapps-telemetry-0.1.0', 'binary_name': 'telemetry', 'src_folder': 'telemetry_src', 'src_subfolder': 'backend', 'operation': 'fetch'}",
" substrate_telemetry_argument: ws://127.0.0.1:8000/submit",
""
]
}
PLAY [localhost] ***************************************************************
TASK [Gathering Facts] *********************************************************
ok: [localhost]
TASK [control-prepare : remove session dir from host] **************************
changed: [localhost]
PLAY [validator] ***************************************************************
TASK [Gathering Facts] *********************************************************
ok: [104.248.35.51]
ok: [68.183.216.242]
TASK [validator-env-gantree : create group for gantree] ************************
changed: [104.248.35.51]
changed: [68.183.216.242]
TASK [validator-env-gantree : add user for gantree] ****************************
changed: [104.248.35.51]
changed: [68.183.216.242]
PLAY [builder_telemetry] *******************************************************
TASK [Gathering Facts] *********************************************************
ok: [104.248.35.51]
TASK [binary-rust-compile : select operation] **********************************
included: /usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/ansible/roles/binary-rust-compile/tasks/0_select.yml for 104.248.35.51
TASK [binary-rust-compile : build binary remote] *******************************
skipping: [104.248.35.51]
TASK [binary-rust-compile : build binary local] ********************************
skipping: [104.248.35.51]
TASK [binary-rust-compile : fetch binary] **************************************
included: /usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/ansible/roles/binary-rust-compile/tasks/1_fetch_binary.yml for 104.248.35.51
TASK [binary-rust-compile : create host bin directory] *************************
ok: [104.248.35.51 -> localhost]
TASK [binary-rust-compile : download node binary] ******************************
ok: [104.248.35.51 -> localhost]
PLAY [validator] ***************************************************************
TASK [binary-push : copy binary to validator] **********************************
changed: [104.248.35.51]
changed: [68.183.216.242]
PLAY [validator] ***************************************************************
TASK [binary-service : install systemd] ****************************************
2020-04-29T08:08:29.240Z [Gantree] (cmd) error: Error occured during command execution 'ansible-playbook -i /gantree/inventory/substratee-test/gantree -i /gantree/inventory/substratee-test/active /usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/ansible/operation.yml':
[WARNING]: Could not find aptitude. Using apt-get instead
ok: [68.183.216.242]
ok: [104.248.35.51]
TASK [binary-service : create telemetry service file] **************************
changed: [104.248.35.51]
changed: [68.183.216.242]
TASK [binary-service : configure and stop telemetry service in systemd] ********
changed: [104.248.35.51]
changed: [68.183.216.242]
TASK [binary-service : start telemetry service] ********************************
changed: [104.248.35.51]
changed: [68.183.216.242]
PLAY [builder_bin] *************************************************************
TASK [Gathering Facts] *********************************************************
ok: [104.248.35.51]
TASK [builder-build-bin : get build prerequisites] *****************************
skipping: [104.248.35.51]
TASK [builder-build-bin : build binary remotely] *******************************
skipping: [104.248.35.51]
TASK [builder-build-bin : build binary locally] ********************************
skipping: [104.248.35.51]
TASK [builder-build-bin : fetch binary] ****************************************
included: /usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/ansible/roles/builder-build-bin/tasks/fetch_binary.yml for 104.248.35.51
TASK [builder-build-bin : create host bin directory] ***************************
ok: [104.248.35.51 -> localhost]
TASK [builder-build-bin : download node binary] ********************************
ok: [104.248.35.51 -> localhost]
PLAY [validator:builder_spec] **************************************************
TASK [validator-push-bin : copy node binary to validator] **********************
changed: [104.248.35.51]
changed: [68.183.216.242]
PLAY [validator] ***************************************************************
TASK [validator-gen-keys : ensure home dir] ************************************
ok: [104.248.35.51]
ok: [68.183.216.242]
TASK [validator-gen-keys : create mnemonic from config] ************************
skipping: [104.248.35.51]
skipping: [68.183.216.242]
TASK [validator-gen-keys : check for mnemonic] *********************************
ok: [104.248.35.51]
ok: [68.183.216.242]
TASK [validator-gen-keys : download subkey binary] *****************************
changed: [104.248.35.51]
changed: [68.183.216.242]
TASK [validator-gen-keys : generate mnemonic] **********************************
changed: [104.248.35.51]
changed: [68.183.216.242]
TASK [validator-gen-keys : write mnemonic to disk on remote host] **************
changed: [104.248.35.51]
changed: [68.183.216.242]
TASK [validator-gen-keys : save session info] **********************************
ok: [104.248.35.51] => (item=ed25519)
ok: [68.183.216.242] => (item=ed25519)
ok: [104.248.35.51] => (item=sr25519)
ok: [68.183.216.242] => (item=sr25519)
TASK [validator-gen-keys : get node key] ***************************************
changed: [104.248.35.51]
changed: [68.183.216.242]
TASK [validator-gen-keys : get bootnode public ip] *****************************
ok: [104.248.35.51]
ok: [68.183.216.242]
TASK [validator-gen-keys : write session info to disk on remote host] **********
changed: [104.248.35.51]
changed: [68.183.216.242]
TASK [validator-gen-keys : ensure host session directory] **********************
changed: [104.248.35.51 -> localhost]
ok: [68.183.216.242 -> localhost]
TASK [validator-gen-keys : change owner of substrate user keystore] ************
changed: [104.248.35.51]
changed: [68.183.216.242]
PLAY [validator] ***************************************************************
TASK [validator-gen-keys-retreive : retrieve session info] *********************
changed: [104.248.35.51]
PLAY [validator] ***************************************************************
TASK [validator-gen-keys-retreive : retrieve session info] *********************
changed: [68.183.216.242]
PLAY [builder_spec] ************************************************************
TASK [Gathering Facts] *********************************************************
ok: [104.248.35.51]
TASK [builder-build-spec : create remote chainspec directory] ******************
changed: [104.248.35.51]
TASK [builder-build-spec : generate chain spec] ********************************
ok: [104.248.35.51]
TASK [builder-build-spec : change owner of substrate user keystore] ************
ok: [104.248.35.51]
TASK [builder-build-spec : create local chainspec directory] *******************
ok: [104.248.35.51 -> localhost]
TASK [builder-build-spec : download chain spec] ********************************
changed: [104.248.35.51]
TASK [builder-build-spec : do combine] *****************************************
ok: [104.248.35.51 -> localhost]
TASK [builder-build-spec : ensure host spec directory] *************************
ok: [104.248.35.51 -> localhost]
TASK [builder-build-spec : write combine] **************************************
changed: [104.248.35.51 -> localhost]
TASK [builder-build-spec : do inject] ******************************************
ok: [104.248.35.51 -> localhost]
TASK [builder-build-spec : write inject] ***************************************
changed: [104.248.35.51 -> localhost]
TASK [builder-build-spec : copy chainspec back to generator] *******************
changed: [104.248.35.51]
TASK [builder-build-spec : generate raw chain spec] ****************************
ok: [104.248.35.51]
TASK [builder-build-spec : download raw chain spec] ****************************
changed: [104.248.35.51]
PLAY [builder_spec] ************************************************************
TASK [Gathering Facts] *********************************************************
ok: [104.248.35.51]
TASK [builder-build-edgeware-spec : create host spec directory] ****************
skipping: [104.248.35.51]
TASK [builder-build-edgeware-spec : download edgeware raw spec] ****************
skipping: [104.248.35.51]
PLAY [validator] ***************************************************************
TASK [validator-push-spec : ensure remote spec directory exists] ***************
ok: [104.248.35.51]
changed: [68.183.216.242]
TASK [validator-push-spec : copy raw-chainspec to validator] *******************
ok: [104.248.35.51]
changed: [68.183.216.242]
PLAY [validator] ***************************************************************
TASK [validator-service : install systemd] *************************************
ok: [68.183.216.242]
ok: [104.248.35.51]
TASK [validator-service : create substrate service file] ***********************
changed: [104.248.35.51]
changed: [68.183.216.242]
TASK [validator-service : configure and stop substrate service in systemd] *****
changed: [104.248.35.51]
changed: [68.183.216.242]
TASK [validator-service : start substrate] *************************************
changed: [104.248.35.51]
changed: [68.183.216.242]
PLAY [validator] ***************************************************************
TASK [validator-key-insert : ensure curl] **************************************
ok: [104.248.35.51]
ok: [68.183.216.242]
TASK [validator-key-insert : grab keys] ****************************************
changed: [104.248.35.51] => (item=aura)
failed: [68.183.216.242] (item=aura) => {"ansible_loop_var": "item", "changed": true, "cmd": "if [ aura = gran ]; then\n crypto=\"ed25519\"\nelse\n crypto=\"sr25519\"\nfi\ninspect_result=$(cat /home/subuser/mnemonic | xargs --null /usr/local/bin/subkey --${crypto} inspect)\npublic_key=$(echo -n \"${inspect_result}\" | grep \"Public key\" | cut -d':' -f2 | tr -d '[:space:]')\nmnemonic=$(cat /home/subuser/mnemonic)\n\nprintf \"${public_key}\"\nprintf \"${mnemonic}\"\n\ncurl http://localhost:9933 -H \"Content-Type:application/json\" -d \"{ \\\n \\\"jsonrpc\\\":\\\"2.0\\\", \\\n \\\"id\\\":1, \\\n \\\"method\\\":\\\"author_insertKey\\\", \\\n \\\"params\\\": [ \\\n \\\"aura\\\", \\\n \\\"${mnemonic}\\\", \\\n \\\"${public_key}\\\" \\\n ] \\\n }\"\n", "delta": "0:00:00.036437", "end": "2020-04-29 08:09:13.557565", "item": "aura", "msg": "non-zero return code", "rc": 7, "start": "2020-04-29 08:09:13.521128", "stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 9933: Connection refused", "stderr_lines": [" % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 9933: Connection refused"], "stdout": "0x2c935883949b7cb5503c73e1ee7b765177d4c19d9af9a0977d35c081a5a5ee62ethics response glare rural theme photo small decade poem witness orange window", "stdout_lines": ["0x2c935883949b7cb5503c73e1ee7b765177d4c19d9af9a0977d35c081a5a5ee62ethics response glare rural theme photo small decade poem witness orange window"]}
changed: [104.248.35.51] => (item=gran)
changed: [68.183.216.242] => (item=gran)
changed: [104.248.35.51] => (item=babe)
changed: [68.183.216.242] => (item=babe)
changed: [104.248.35.51] => (item=imol)
changed: [68.183.216.242] => (item=imol)
changed: [104.248.35.51] => (item=para)
changed: [68.183.216.242] => (item=para)
changed: [104.248.35.51] => (item=audi)
changed: [68.183.216.242] => (item=audi)
PLAY [validator] ***************************************************************
TASK [validator-service-restart : stop substrate] ******************************
changed: [104.248.35.51]
TASK [validator-service-restart : clean substrate db] **************************
changed: [104.248.35.51]
TASK [validator-service-restart : start substrate] *****************************
changed: [104.248.35.51]
PLAY RECAP *********************************************************************
104.248.35.51 : ok=59 changed=28 unreachable=0 failed=0 skipped=8 rescued=0 ignored=0
68.183.216.242 : ok=31 changed=19 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0
localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
2020-04-29T08:09:18.023Z [Gantree] (cmd) error: Execution failed with code 2: ansible-playbook -i /gantree/inventory/substratee-test/gantree -i /gantree/inventory/substratee-test/active /usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/ansible/operation.yml
(node:12) UnhandledPromiseRejectionWarning: Error: 2
at childCloseCallback (/usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/src/lib/cmd.js:117:18)
at ChildProcess.<anonymous> (/usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/src/lib/cmd.js:55:5)
at ChildProcess.emit (events.js:323:22)
at maybeClose (internal/child_process.js:1021:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)
(node:12) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:12) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment